Tokenizer
A tokenizer is the component that cuts text into the tokens a model reads and bills, which is why the same text can come to a different number of tokens on two different models.
Most modern tokenizers are built from data. Frequent sequences, such as common words and familiar code patterns, become single tokens, while rare identifiers, unusual formatting and many non-Latin scripts are split into several. The vocabulary is fixed when the model is trained, so a token count only means something alongside the model it was measured on.
Tokenizers change between generations. Per Anthropic, as of September 2026, Claude 4.7 and later models use a newer tokenizer that produces about 30% more tokens for the same text than Claude Sonnet 4.6 and earlier models, the exact increase depending on the content. Claude Opus 4.6 and Opus 4.7 have the same list price per token, so the same prompt costs more on the newer model through tokenization alone.
Three consequences follow. Counts measured on one model do not transfer to another, so budgets, context-window estimates and cost projections have to be recounted after a switch. Prices per token cannot be compared across tokenizers without adjusting for how much text a token covers. And rules of thumb such as four characters per token are averages for English prose, not a way to count code.
The exact count comes from the provider. Anthropic's token counting endpoint returns the input tokens of a request under the tokenizer of the model you name, free of charge but rate limited, and its documentation describes the result as an estimate that can differ slightly from what is billed.