What is it?
Tokenization is the process of converting text into tokens.
A token can be a whole word, part of a word, punctuation, or another text unit, depending on the tokenizer.
A token is not always the same thing as a word.
05 · Tokens, Context & Inference · Topic 02
What is Tokenization?
Tokenization converts text into smaller units that a language model can process.
Think of it like...
Puzzle pieces
Think of cutting a sentence into puzzle pieces before giving it to the model. The model works with those pieces, then uses them to understand the input and produce an output.
How it works
1 · TextReceive textA user enters a sentence or document.
2 · TokenizerSplit the textThe tokenizer divides text into token pieces.
3 · Token piecesRepresent unitsThe pieces become the model-readable units.
4 · Token IDsMap to numbersEach token is represented by an ID from the tokenizer vocabulary.
5 · Model inputProcess the sequenceThe model receives the token IDs as input.
Text → Tokenizer → Token Pieces → Token IDs → Model Input
Real-world examples
Everyday example · A message
A chat message is split into token units before the language model processes it.
AI product example · Chat assistant
Input: “unbelievable”
System: A tokenizer may represent it as one token or several pieces, depending on the tokenizer.
Output: Token IDs sent into the model.
What it is NOT
Tokenization ≠ Splitting by Spaces
Tokenization: Uses a tokenizer’s rules to create model-readable units.
Splitting by spaces: Only separates text at spaces.
Token ≠ Word
Token: May be a whole word, part of a word, punctuation, or another unit.
Word: A human language unit.
Tokenizer ≠ Language Model
Tokenizer: Converts text into token pieces and IDs.
Language model: Processes those inputs to predict or generate text.
Related concepts
Text → Tokenization → Tokens → Context Window → Inference
Explore next
Remember this
Tokenization turns raw text into the token units a model can process.
Independent explainer
Video
A Tokenization explainer video is not available yet.