An agent that "forgets" usually isn't broken. It has usually just been given a context window instead of a memory system, and those are genuinely different things. A context window is what the model can see in one call. Memory is what persists across calls, sessions, and days. Confusing the two is the most common reason a long-running agent starts repeating mistakes or losing track of decisions it made an hour ago.
Why re-stuffing the full history stops working
The naive approach to agent memory is re-sending the entire conversation history with every new call. This works fine for a short interaction and breaks down for anything long-running. The context window fills up, cost per call climbs as the history grows, and, critically, the model's attention gets diluted across an increasingly long history. That can make it worse at reasoning about what actually matters right now, not better.
The two-layer split that actually works
Current best practice separates memory into two distinct concerns, each solving a different problem.
The memory layer: what to recall
This is a storage and retrieval problem, closer to a database than a prompt. Facts, decisions, and relevant history get stored (often as embeddings in a vector store) and retrieved based on relevance to the current task, not dumped in wholesale. This is the same retrieval discipline behind RAG, applied to an agent's own history instead of external documents.
The context layer: what actually goes in the prompt
This governs meaning: given everything potentially recallable, what's the minimum, most relevant set of information the model actually needs for this specific step? A well-designed context layer pulls only what's relevant from the memory layer, rather than assuming more context is always better.
A practical three-tier model
For agents that genuinely need to operate over long horizons, a tiered memory architecture (mirroring how operating systems handle memory hierarchy) tends to hold up better than a single flat store:
- In-memory / working context: the immediate task state, fast to access, small, cleared or summarized frequently
- Vector database / mid-term memory: retrievable facts and decisions from the current session or recent history, searched by relevance
- Cold archival storage: full historical record, rarely accessed directly but available for audit or deep retrieval when genuinely needed
This maps directly to how different memory actually gets used: you need the working context on every call, mid-term memory occasionally, and the cold archive almost never, except when something specific needs to be looked up.
The failure mode nobody budgets for: stale memory
A subtler problem than "the agent forgot" is "the agent remembered something that's no longer true." If a user's preference, an account status, or a business rule changes, an agent with naive memory retrieval can confidently act on outdated information: it retrieved the fact correctly, but from a memory store that was itself out of date. This is a genuinely hard problem. Memory systems need a way to invalidate or timestamp facts, not just store and retrieve them, and testing for this specifically (not just testing recall accuracy) is worth building into your evaluation process.
How we approach this
For agents that need to operate beyond a single session, we design the memory and context layers as distinct systems from the start, with explicit scoping and staleness handling, rather than defaulting to re-sending history and hoping the context window holds.