This is the second essay in the Foundations pillar. It assumes you have read What is a token, really?; the rest of this piece is about what happens to those tokens once the model has them.

The problem attention solves

Imagine you are reading the sentence:

The trophy didn’t fit in the suitcase because it was too big.

To resolve “it,” you glance back at “trophy.” When you read:

The trophy didn’t fit in the suitcase because it was too small.

…you glance at “suitcase” instead. Same five-letter pronoun, different referent, and you decided which one to look at based on the rest of the sentence.

Before transformers, the dominant approach to language was to read text left to right and squeeze everything into a single hidden state. That forced the model to decide what was important before it knew what came next. Attention is the operation that lets the model decide later, once it has seen the whole sentence.

What attention actually does

For every token in the input, the model produces three vectors: a query, a key, and a value. The query asks “what am I looking for?”; the keys advertise “this is what I am”; the values carry “here is my content.” Each token then takes a weighted average of every other token’s value, weighted by how well its query matches their keys.1

That is the whole trick. Jay Alammar’s Illustrated Transformer walks through it with diagrams that are worth more than another thousand words here.2

A transformer stacks dozens of these attention layers, interleaved with small feed-forward networks, and that is essentially the architecture behind every major LLM in production today.

Two consequences worth knowing

Attention is quadratic. Naively, comparing every token to every other token costs work that grows with the square of the sequence length. Doubling your context length roughly quadruples the compute for the attention layers. This is the reason long context is expensive, and the reason a small industry exists around making attention cheaper — FlashAttention rewrote the kernels to use GPU memory more carefully and became standard within a year of its 2022 release.3 A broader survey from 2020 catalogues dozens of approximate variants that trade exactness for sub-quadratic cost.4

Attention is the thing that gets long. When people talk about a model having a “200K context window,” what they mean technically is that the attention layers can address 200,000 token positions. The fact that this is now common — it used to be 512 — is almost entirely the story of engineering around attention’s cost.

What to remember

Attention is a learned lookup over the sequence. That is what makes transformers different from what came before, and it is the single most useful mental model to carry into every other essay on this site.

Next in this pillar: embeddings, training vs. inference, and context windows — coming soon.

Footnotes

  1. Vaswani et al., Attention Is All You Need, 2017. ↩

  2. Alammar, The Illustrated Transformer. ↩

  3. Dao et al., FlashAttention, 2022. ↩

  4. Tay et al., Efficient Transformers: A Survey, 2020. ↩

Citations

  1. Attention Is All You Need (Vaswani et al., 2017) · accessed May 17, 2026
  2. The Illustrated Transformer — Jay Alammar · accessed May 17, 2026
  3. FlashAttention: Fast and Memory-Efficient Exact Attention (Dao et al., 2022) · accessed May 17, 2026
  4. Efficient Transformers: A Survey (Tay et al., 2020) · accessed May 17, 2026

More in Foundations

  • What is a token, really?

    Models don't see words. They see tokens — chunks of bytes carved up by a learned vocabulary. Here is what that means in practice.