This is the first essay in the Foundations pillar. If you only read one piece on this site, it should be this one — every later argument about cost, quality, and capability eventually circles back to tokens.

Models do not see words

When you type “unbelievable” into a chat, the model does not receive the word “unbelievable.” It receives a short sequence of integers — for example [403, 6892, 0] — that index into a fixed vocabulary the model learned during training. Those integers are tokens.

Most production LLMs use a family of algorithms called byte pair encoding, or BPE, to build that vocabulary. BPE was originally a compression technique; it was adapted for neural machine translation in 2016 and has been the default ever since.1 OpenAI’s open-source tiktoken library is a fast BPE implementation that mirrors what their hosted models use.2

The procedure, briefly:

  1. Start with the raw bytes of your training text.
  2. Find the most common adjacent pair of symbols and merge it into a new symbol.
  3. Repeat tens of thousands of times.

The result is a vocabulary of roughly 50,000 to 200,000 entries, mixing whole common words (" the"), word fragments ("ation"), and individual bytes for anything rare.3

Why this matters in practice

Three consequences fall out of the fact that the unit is a token, not a word:

1. The bill is in tokens. Every major API prices input and output by the token. A rule of thumb that holds across English text is roughly 0.75 words per token, but it varies wildly: code, JSON, and non-Latin scripts often use two to four times more tokens than the equivalent English prose.4

2. The context window is in tokens. When a model advertises a “200,000 token context window,” that is not 200,000 words. For dense prose it is closer to 150,000 words; for minified JavaScript it might be 50,000.

3. Spelling-style tasks are unnaturally hard. Asking a model “how many r’s are in strawberry?” is asking it to introspect on characters inside a token it only ever saw as a single integer. It is the LLM equivalent of asking you how many photons hit your retina when you read this sentence.

What to remember

Whenever you read a claim about an LLM — its speed, its cost, its “reasoning length,” its memory — quietly translate the unit to tokens first. Most surprises stop being surprising once you do.

Next in this pillar: Attention, in plain English.

Footnotes

  1. Sennrich et al., Neural Machine Translation of Rare Words with Subword Units, 2016. ↩

  2. OpenAI, tiktoken repository. ↩

  3. Hugging Face, Summary of the tokenizers. ↩

  4. Brown et al., Language Models are Few-Shot Learners, 2020 — appendix on BPE token counts across languages. ↩

Citations

  1. tiktoken — OpenAI's BPE tokenizer · accessed May 17, 2026
  2. Summary of the tokenizers — Hugging Face Transformers docs · accessed May 17, 2026
  3. Byte pair encoding — Wikipedia · accessed May 17, 2026
  4. Neural Machine Translation of Rare Words with Subword Units (Sennrich et al., 2016) · accessed May 17, 2026
  5. Language Models are Few-Shot Learners (GPT-3) — Brown et al., 2020 · accessed May 17, 2026

More in Foundations

  • Attention, in plain English

    The one trick that makes transformers work, explained without matrices — and what changes once you understand it.