Tokens: The Unit AI Thinks (and Bills) In

A token is the chunk of text an AI model processes as one unit. What tokens are, how many a word has, how they are counted and how they affect cost and limits.

info Quick Definition
A token is the smallest unit of text a language model reads and writes. It can be a whole word, a fragment of a word or a punctuation mark. AI providers measure the size of each request, and its cost, in tokens.

What is a token?

A model does not see letters or words the way we do. Before processing a text, a component called the tokenizer splits it into pieces from a fixed vocabulary. A frequent word is usually a single token; a rare, long or misspelled word is split into several. Numbers, spaces and emojis also take up tokens.

A widely used method for building that vocabulary is byte-pair encoding (BPE). It starts with small units and repeatedly merges the pair of adjacent pieces that appears most often in a corpus, until it reaches a set vocabulary size. According to Wikipedia, a slightly modified version of the algorithm is used in language model tokenizers, and GPT-3.5 and GPT-4 use a vocabulary of 100,258 tokens. The result is neither whole words nor single letters but subword units, which let a model represent any text with a manageable vocabulary.

A token is not a semantic concept. It does not mean “one idea”. That is why two texts with the same meaning can have different token counts.

How many tokens is a word?

There is no fixed figure, but there are orders of magnitude that help you estimate. OpenAI’s documentation gives a rough rule for English text: one token is about 4 characters or 0.75 words. The repository of its tokenizer, tiktoken, says something similar: on average, one token corresponds to about 4 bytes of text.

That rule breaks down in other languages. A study from Meta FAIR and Sorbonne Université (ParaEval, June 2026) measured the Llama 3.2 tokenizer: on English it gets 4.9 characters per token and 1.24 tokens per word; on Portuguese, French and Spanish it drops to 3.6 to 3.8 characters per token and rises to 1.60 to 1.71 tokens per word. Put another way, a 1,000-word text would take about 1,240 tokens in English and between 1,600 and 1,710 in those three languages, roughly a third more. The study reports that figure for the three together, not for Spanish alone.

Two warnings. First, those figures come from one specific tokenizer, not all of them. Second, I could not confirm a published measurement of Claude’s tokenizer on Spanish, so for that case it is better to measure with your own text (see below).

Why do tokens matter when designing?

Tokens affect four things that users or the team can feel:

  • The limit. Each model has a context window measured in tokens. Everything that goes in (instructions, history, documents and the reply) has to fit. According to Anthropic’s documentation, the latest Claude models reach 1 million tokens of context, and a single request can generate up to 128,000 output tokens.
  • The cost. An API bill is calculated from input and output tokens. A 20-page prompt repeated on every message costs far more than a three-line instruction. See cost per token.
  • The speed. The model generates the reply token by token, so long answers take longer. See latency and tokens per second.
  • The language. The same message in English or Spanish can have a different token count, as the study above shows. A multilingual product should measure cost per language rather than assume it is equal.

How are tokens counted?

Each provider has its own way. Anthropic offers a counting endpoint that accepts the same format as message creation (with system prompts, tools, images and PDFs) and returns the total number of input tokens. Its documentation makes four points:

  • The result is an estimate; the actual number can differ slightly.
  • The count may include tokens Anthropic adds for system optimizations, and those are not billed.
  • Using it is free, though it has requests-per-minute limits depending on your usage tier.
  • Claude 4.7 and later models use a newer tokenizer that produces roughly 30 percent more tokens for the same text than earlier models. The official advice is to recount prompts with the model you plan to use rather than reuse counts from an older model.

A model’s reasoning (“thinking”) tokens also count: the documentation includes them in the context window and bills them as output tokens. For a quick estimate with OpenAI models you can use the tiktoken library; for real figures, always use the official count for the model you plan to use.

When should you NOT think only in tokens?

Cutting tokens at all costs can make the product worse. Trimming context to save money can leave the model without the information it needs, and the answers get worse. The useful question is not “how do I spend fewer tokens?” but “how many tokens does this need to produce a good answer?” Measuring quality and cost together prevents optimizing the wrong metric. To cut the cost of repeated context without trimming it, look at prompt caching.

Example

Illustrative example: a design team builds an assistant that answers questions about its 60-page style guide, which we will assume is about 40,000 tokens. Use as a reference the prices of Claude Sonnet 5.5 in Anthropic’s table as of October 17, 2026: 2 dollars per million input tokens and 10 per million output tokens. Each answer is about 500 tokens.

  • Sending the whole guide with every question: 40,000 input tokens cost 0.08 dollars and 500 output tokens cost 0.005. Total: about 0.085 dollars per question, or about 85 dollars per 1,000 questions.
  • Sending only the relevant sections (about 3,000 tokens), retrieved by search: 0.006 dollars of input plus 0.005 of output. Total: about 0.011 dollars per question, or about 11 dollars per 1,000.

The second option costs about eight times less in this example, but it only pays off if the answers stay correct, so you need to measure both. Token counts shape the architecture: more context is not always better, and neither is less. These prices change; go back to the current table before budgeting.

Common mistakes

  • Estimating costs with words instead of tokens. A word is not a token, and the ratio changes with the language.
  • Ignoring output tokens, which cost five times as much as input tokens in Anthropic’s table today.
  • Forgetting that conversation history re-enters as input on every turn.
  • Assuming one provider’s tokenizer gives the same count as another’s, or that two versions from the same provider count alike.
  • Comparing costs across models without recounting the same text on each.
  • Forgetting reasoning tokens, which are billed as output.

Sources

Frequently asked questions

What is a token in AI?
A token is the unit of text a model processes. It can be a whole word, part of a word, a punctuation mark or a space, depending on the model’s tokenizer.
How many tokens is a word?
It depends on the language and the tokenizer. OpenAI gives a rough rule for English text: 1 token is about 4 characters or 0.75 words. Spanish usually takes more tokens per word. For real figures, count with the provider’s own tool.
Why do AI APIs charge by token?
Because the model’s work grows with how many tokens it reads and writes. Providers publish a price per million tokens, with separate rates for input and output.
Do input and output tokens cost the same?
No. In Anthropic’s pricing table as of October 17, 2026, output tokens cost five times as much as input tokens on every current model. Check the current table before you calculate.