A token is a small unit of text that an AI model reads and processes.
A token is a small unit of text processed by a language model.
A token may be a whole word, part of a word, punctuation mark, number, or another text unit depending on the tokenizer.
AI models process tokens rather than reading text exactly the way humans see words and sentences.
Think of a sentence like a LEGO model.
Before the model can work with it, the sentence is broken into smaller LEGO pieces.
Those pieces are tokens.
Token ≠ Word. One word may be one token, or it may become multiple tokens.
Text: “Artificial intelligence”
It may be split into one or several tokens depending on the tokenizer.
A long or uncommon word may be split into several tokens.
Chinese text is also tokenized, but token boundaries do not necessarily match English-style words.
Token: A unit chosen by a tokenizer.
Word: A human language unit.
One word may become one or several tokens.
Token: May contain one or more characters.
Character: A single written symbol.
Token: Input/output unit processed by the model.
Parameter: A learned numerical value inside the model.
Explore next
AI models process tokens, not text exactly the way humans see words.
A short visual explanation of tokens.