Context rot is the way a model's effective use of the material in its context window degrades as that window fills up. The token count keeps climbing right up to the advertised limit; the model's grip on the content in the middle quietly loosens long before then.

Definition

The classic demonstration is the lost-in-the-middle effect described by Liu et al. in 2023. They put a single relevant fact inside a long document at varying positions and asked the model to use it. Performance was high when the fact sat near the beginning, high again when it sat near the end, and dropped sharply when it sat somewhere in between. The shape of the curve is broadly the same across model families.

The same effect shows up under different names. Needle-in-a-haystack benchmarks measure whether a model can retrieve a single planted fact from a long context — those are the easy version of the problem. Once you need the model to combine several facts from different positions, accuracy falls further and faster than the synthetic benchmarks suggest.

Why it matters

Most product decisions about LLMs start with a context budget — "we have 200k tokens, what should we put in them?" Context rot turns that into the wrong question. The right question is how much material the model can usefully attend to at the precision your task requires, which is almost always less than the advertised window and often dramatically so. A system that pastes the entire knowledge base into every prompt frequently performs worse than one that retrieves a few hundred well-chosen tokens.

It also explains a class of bugs that look mysterious. The model ignores an instruction you clearly gave it. The new system prompt seems to "stop working" once the conversation grows. Long agent runs slowly forget what they were doing. None of this is a memory leak. It is attention being spread too thin across material the model can no longer weight properly.

Common misconceptions

A bigger context window does not fix context rot. It just postpones the cliff and makes it more expensive.

Long-context benchmark scores in marketing material are typically needle tests. They do not predict performance on real, multi-fact work.

"We hit the limit" is usually wrong. The model lost it because the relevant tokens were buried, not because they fell off.

The treatment is a mix of architecture and discipline. Retrieve less, not more. Put the most important instructions and facts at the start or end of the prompt, not in the middle. Summarize aggressively when a conversation grows. And measure your own application's degradation curve rather than trusting the vendor's headline number.

Short reading list