Agent memory is the set of mechanisms an AI agent uses to carry information from one turn, session, or task to the next. The phrase is doing a lot of work. In practice it covers at least three distinct systems that get collapsed into one word.
Definition
A language model, on its own, is stateless. Every call rebuilds the entire context window from scratch from whatever the surrounding system decided to put in front of it. "Memory" is whatever that surrounding system chose to remember on the model's behalf — and there are three layers worth keeping separate.
Working memory is the current context window: the prompt, the conversation so far, and any tool outputs already streamed back. It vanishes the moment the request ends.
Session memory is a transcript or running summary the application keeps for the length of a single conversation and replays into working memory on each turn. When a chatbot seems to remember what you said five messages ago, that is almost always session memory being re-pasted into the prompt.
Long-term memory is anything the application chooses to persist past the session — a profile, a vector store of past interactions, a notes file the agent can read and write. These are storage systems with an LLM bolted on, not a property of the model.
Why it matters
The cost, latency, privacy posture, and failure modes of an agent depend almost entirely on which layer it is leaning on. A system that re-summarizes the whole transcript every turn pays a token tax on every call and quietly loses detail. One that uses a vector store inherits retrieval bugs — irrelevant snippets, stale facts, the model trusting whatever it was handed. One that simply stuffs everything into the window pays for it linearly and runs straight into context rot at the far end.
It matters for trust, too. "The agent remembers me" can mean your data was stored on a server, vectorized, indexed, and surfaced into future sessions you did not consent to. The user-facing word is the same; the system underneath is very different.
Common misconceptions
The model itself remembers nothing. Between API calls there is no hidden state surviving on a server somewhere unless an application explicitly put it there.
Bigger context windows do not remove the need for memory design. They just change which problems bite you first. Recall, cost, and retrieval quality all keep mattering.
A vector database is not memory. It is a retrieval index over text some other system decided to keep. The choice of what to keep is the actual memory policy.
Short reading list
- Anthropic, Building effective agents — the clearest overview of which memory shape fits which problem.
- Packer et al., MemGPT: Towards LLMs as Operating Systems (2023) — the canonical paper on an OS-style memory hierarchy on top of a stateless model.
- In this guide: Context Rot explains why piling everything into working memory degrades quality long before you hit the advertised window limit.