AiComing logoAiComing

AI ENCYCLOPEDIA

AI concepts tagged “tokens”

Explore clear explanations of AI concepts tagged tokens.

Browse by tag

2 concepts

Context Window: What an AI Model Can Use at Once concept diagram

Foundations

Context Window: What an AI Model Can Use at Once

A context window is the bounded sequence of tokens a model can process as its current input and, for a generative model, the growing output history it can condition on while generating.

#tokens#LLM limits#prompting
Read entry
Tokenization: How AI Models Break Text into Tokens concept diagram

Foundations

Tokenization: How AI Models Break Text into Tokens

Tokenization is the process of converting text or another input into a sequence of discrete units called tokens, then mapping those tokens to numeric identifiers that a model can process.

#tokens#NLP#text processing
Read entry