Cost defragmentation: where AI money is actually wasted

The most common way to talk about reducing AI cost is to talk about price — switch to a cheaper model, batch your requests, wait for the next round of provider discounts. All of that is real, and all of it is downstream of a more interesting question: how much of what you are already paying for is being wasted?

This essay is about that question. The framing is the one this publication is named after — defragmentation. Most AI bills are not high because the underlying compute is expensive. They are high because the compute is being held idle, the cache is reporting full while it is half empty, and the context being shipped to the model is not being attended to once it arrives. Each of those is a defragmentation problem with a known fix.

Waste #1 — Memory that is “full” but empty

The cleanest example is the KV cache: the stored attention tensors a transformer keeps for every token it has already processed, so it does not have to recompute attention from scratch on every next token. The vLLM paper opens by identifying KV-cache management as the dominant memory bottleneck in transformer serving, and the way most systems allocate that cache as the textbook case of fragmentation.¹

The same paper measured the waste directly. In vanilla serving systems without paged attention, only 20–38% of the allocated KV-cache memory is actually used. The other 60–80% is wasted — partly through internal fragmentation (blocks over-allocated for the maximum-length request that never arrives) and partly through external fragmentation (unusable gaps between live allocations that are too small for any incoming request).¹

The fix is not “more GPUs.” The fix is PagedAttention: manage the KV cache the way an operating system manages virtual memory, in fixed-size blocks allocated on demand, with a virtual-to-physical mapping that lets blocks be packed without contiguous runs. After paging, the same paper reports waste falling to under 4% and throughput rising 2–4× at the same latency versus the prior state-of-the-art systems it was benchmarked against.¹ The 2025 PagedEviction paper extends the technique with block-wise eviction of low-importance cache pages — a further win for workloads with long sessions.²

That is what “defragmenting” the inference layer looks like in practice: same hardware, same model, materially higher throughput per GPU-second, with the magnitude of the gain set by how much of your fleet was already running paged attention and how varied your request lengths are.

Waste #2 — Tokens that are shipped and ignored

The second waste is more subtle. You can pay for tokens the model never really uses.

The mechanism is what this publication calls context rot — the degradation of a model’s effective use of its input as the input grows longer, even well below the advertised window size. We treat it at length in the Context Rot field-guide entry, which cites the primary research; what matters for the cost story is the downstream consequence. Standard token-metered APIs bill input tokens at a posted per-million-token rate, so a longer prompt maps to a proportionally larger line item — that part is just how the bills are written. The model’s use of that input does not scale the same way, by the context-rot research summarized in the field-guide entry.

The cost-defragmentation move here is easy to name and harder to do: stop paying for context the model will not read. The standard techniques — retrieval over dump-and-pray, summarization of earlier turns, just-in-time tool results instead of pre-loaded ones — are well established in the RAG literature; the only editorial claim we make here is that, in our experience, the same moves that lower the input bill tend to lower the quality issues as well, because they remove the distractor content the context-rot research identifies as harmful. Trimming inputs also interacts with the KV-cache fix in a directionally favorable way: smaller per-request caches leave more room for concurrent sessions per node.

Waste #3 — Agents that re-do work because they forgot

The third waste shows up at the agent layer. When an agent’s memory is misdesigned — for example, it stuffs every prior turn into the next prompt, or it has no recall mechanism so the user repeats themselves — the agent pays the context-rot tax and the per-turn token tax and often runs an extra tool call or two to reconstruct state the previous turn already had.

The cost surface here is rarely measured well. Most teams report agent cost as model spend; the wasted spend hides inside the model spend. We have not yet seen a published figure for the recoverable fraction that we would cite with confidence, so we will not invent one. The qualitative claim — that counting redundant tokens and redundant tool calls per session is worth doing — is an editorial observation from our engagements, not a measured industry figure.

Waste #4 — Tools that overlap

The fourth waste sits one layer up from the model bill: the AI sprawl layer. The pattern is familiar — multiple tools adopted by different teams that solve the same problem, each one looking obviously better than the last, none ever decommissioned. The waste here is not a percentage of a bill; it is the bill, in duplicate.

Consolidation is usually the highest-ROI move on the list, and the hardest one politically. The diagnostic exists for exactly this — a fixed-scope inventory that names the overlaps and the consolidation opportunities, before the next budget cycle bakes them in.

How to actually run this

The four wastes compound. They also resolve in a specific order.

Start at the inference layer. Audit the KV-cache utilization on your inference fleet. Under the workload and allocator conditions the vLLM paper benchmarked, vanilla serving systems wasted 60–80% of allocated KV-cache memory; if your setup resembles those conditions and you are not running paged attention, that is the figure to measure against your own fleet.¹
Then audit the inputs. Run a one-week sample of representative sessions. Measure how much of the supplied context the model actually uses. (One blunt proxy: ablate sections of the input and measure the answer-quality delta. If the delta is zero, the context was paid for and ignored.)
Then audit the agents. Per-session, count redundant tokens and redundant tool calls. We do not have a published recoverable-fraction figure to point to; the audit is worth running because the cost surface is otherwise invisible, not because we can promise a number.
Finally, audit the portfolio. Name the overlapping tools. Decommission, do not just demote. A tool that still has a login page still has a bill.

We run a version of this as a fixed-scope engagement; the writeup is on the diagnostic page. The four-step sequence above works whether you run it with us or on your own.

The point of the framing is to stop treating “AI is expensive” as a pricing problem. The pricing is the pricing. The cost story is, almost entirely, a fragmentation story — and the same word that diagnoses it names the fix.

Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023. ↩ ↩² ↩³ ↩⁴
PagedEviction: Structured Block-wise KV Cache Pruning, 2025. ↩

Citations

Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023) · accessed May 17, 2026
PagedEviction: Structured Block-wise KV Cache Pruning (2025) · accessed May 17, 2026