AI concepts tagged “text processing”

Foundations

Tokenization: How AI Models Break Text into Tokens

Tokenization is the process of converting text or another input into a sequence of discrete units called tokens, then mapping those tokens to numeric identifiers that a model can process.