AI concepts tagged “LLM fundamentals”

Foundations

Tokenization: How AI Models Break Text into Tokens

Tokenization is the process of converting text or another input into a sequence of discrete units called tokens, then mapping those tokens to numeric identifiers that a model can process.