Tokens Explained: How AI Reads and Counts Text
When you type a sentence to a language model, it does not see letters or even words. It sees tokens. Tokens are the small units a model actually reads, counts, and bills by, and once you understand them, several puzzling things about AI start to make sense: why it struggles with some spellings, why prices are quoted per million tokens, and why a context window is measured the way it is.
What a token is
A token is a chunk of text, usually a word or a piece of a word, sometimes a single character or a bit of punctuation. Common short words are often one token each. Longer or rarer words are broken into pieces. The word "hotel" is probably one token. The word "unbelievably" might be three: "un", "believ", "ably". A space usually attaches to the word that follows it.
The process that splits text into tokens is called tokenization, and the list of possible tokens a model knows is its vocabulary. A typical vocabulary has somewhere around a hundred thousand entries, which is enough to represent any text by combining pieces.
Why not just use words
Words are a surprisingly bad unit for a machine. There are too many of them, new ones appear constantly, and misspellings and names would each need their own entry. Breaking text into reusable pieces solves this. Any word, including one the model has never seen, can be assembled from fragments it has. That is also why models handle many languages: the same pieces recombine differently.
The cost is that the model's view of text is slightly alien. It has no direct sense of letters, which is why asking a model to count the letters in a word or reverse a string can produce odd mistakes. It is working with chunks, not characters.
A useful rule of thumb
For ordinary English, one token is roughly three quarters of a word, so a thousand tokens is about seven hundred and fifty words. A page of text is a few hundred tokens. A long article is a few thousand. Other languages and anything dense with numbers, code, or unusual spelling tend to use more tokens for the same amount of meaning.
Where token counts show up
Tokens are the currency of language models. Three places you will meet them.
Pricing. API usage is billed per token, usually quoted per million, with separate rates for the text you send and the text the model writes back.
Limits. The context window, the amount a model can consider at once, is a token count. So is the maximum length of a single reply.
Speed. Models produce output one token at a time. A longer answer is more tokens, and more tokens take longer.
What this means for your content
Every system that reads your website converts it into tokens before doing anything else. That has two quiet consequences.
Tokens are a budget. When an AI gathers material to answer a question, it can afford only so many. Content that states a fact in a short, direct sentence spends few tokens and leaves room for other facts. Content that takes three paragraphs to say the same thing spends many, and may be trimmed or skipped.
Clean text tokenizes cleanly. Words hidden in images, decorative fonts rendered as pictures, or menus delivered as scanned flipbooks produce no tokens at all, because there is no text to split. To a model, that information does not exist.
The practical lesson is simple. Say important things plainly, in real text, and say them once. You are not writing for a reader with unlimited patience. You are writing for a system with a token budget.