The vocabulary.
Short definitions for the recurring terms in the field guide. Each entry links to the canonical essay where the term is defended in full. For longer-form answers — what something costs, why it matters, how to tell two things apart — see the FAQ.
- Agent memory
-
The set of mechanisms an AI agent uses to carry information from one turn, session, or task to the next. The phrase covers at least three distinct mechanisms — short-term working memory inside the context window, episodic recall of past sessions, and long-term factual stores — and most product complaints about an agent that 'forgot' are really complaints about which of the three is missing.
- Agent washing
-
The marketing practice of rebranding chatbots, RPA scripts, decision trees, and workflow tools as 'autonomous AI agents.' A real agent pursues goals across multiple steps without re-prompting, chooses between actions based on observed state, and writes to external systems with idempotency. Most products marketed as agents fail the first test.
- AI defragmentation
-
The deliberate work of reorganizing a fragmented AI stack — across hardware memory, model inference, agent state, and tool sprawl — so the pieces compose into a system instead of accumulating as overlapping line items. The opposite of buying another tool to fix the problem the last tool created. This site is the field guide for that work.
- AI fragmentation
-
The accumulation of overlapping, uncoordinated, and unmeasured AI tools, models, and agents across an organization. It appears at four layers: hardware (GPU memory and KV-cache waste), inference (model serving inefficiency), agent (context loss and memory failures), and enterprise (vendor sprawl, shadow AI, governance gaps). The four layers compound — fixing one without naming the others rarely sticks.
- AI sprawl
-
The quiet accumulation of overlapping AI tools, copilots, assistants, and bots across an organization, without any one team owning the resulting system. The AI-era version of SaaS sprawl, with the same root cause — anyone with a credit card can adopt a tool — and a sharper edge, because the tools touch data and write to systems.
- Context rot
-
The measurable degradation of an LLM's accuracy as input length grows, even well below the model's stated context window. Three mechanisms compound: attention dropping in the middle of long inputs, attention dilution as the token count grows, and distractor interference from semantically similar but irrelevant content. A larger context window delays the problem; it does not fix it.
- Context window
-
The maximum number of tokens a model can read in a single request — prompt, supplied documents, and the model's own running response combined. Advertised window size is an upper bound on what fits, not a guarantee of what the model will actually attend to. See context rot for why.
- Fractional AI defrag officer
-
A monthly retainer engagement that provides org-level AI architecture oversight without a full-time hire. Scope typically includes quarterly fragmentation re-scoring, monthly architecture review, vendor-evaluation memos for new AI procurement, and a quarterly board-style update. The role exists because most companies cannot yet justify a full-time CAIO but already need the function.
- KV cache
-
Short for key-value cache. The stored attention tensors a transformer keeps for every token it has already processed, so it does not have to recompute attention from scratch on the next token. KV caches are the single largest consumer of GPU memory during LLM inference, which is why fragmentation in this cache has outsized cost consequences.
- KV cache fragmentation
-
The inference-time version of the memory-fragmentation problem operating systems solved decades ago. Variable-length requests are placed in GPU memory in ways that leave unusable gaps between them; the cache reports 'full' while as much as 60–80% of the allocated memory is empty. PagedAttention is the canonical fix.
- Lost in the middle
-
The empirical finding that an LLM attends most reliably to the start and end of its context window and least reliably to the middle, with accuracy on mid-context information dropping by tens of percentage points. One of the three compounding mechanisms behind context rot.
- PagedAttention
-
An attention algorithm introduced in the vLLM paper that manages an LLM's key-value cache the way an operating system manages virtual memory — in fixed-size blocks allocated on demand. Reduces KV-cache waste from 60–80% down to under 4% and lifts throughput by 2–4× at equivalent latency. The reason 'defragmentation' is the literal right word for this part of the stack.
- Shadow AI
-
The subset of AI sprawl that exists outside IT governance: personal ChatGPT accounts, unsanctioned vendor trials, free tools adopted by individual teams without procurement review. AI sprawl is the inventory problem; shadow AI is the accountability problem. Both are fragmentation, and they require different remediation steps.
- vLLM
-
An open-source LLM inference server built around PagedAttention. The reference implementation for memory-efficient transformer serving and the easiest place to see the effect of defragmenting the KV cache in practice.