AI MaxxingBeginner60 sec

Token

A token is the basic unit of text a language model reads and generates, usually a word, part of a word, or punctuation mark, not always a full word.

Mark as completed

What matters

Because models operate on tokens rather than raw characters or whole words, a word's cost and behavior in the model depends on how many tokens it breaks into, which varies by language and spelling.

Example

The word 'unbelievable' might be split into the tokens 'un', 'believ' and 'able', while a short common word like 'cat' is usually a single token.

Common misconception

A token is not always a whole word; many tokens represent word fragments, and this varies across languages.

Recall question

Why might one word be broken into several tokens instead of one?

Keywords and related concepts

tokentokenizationlanguage modellarge language model