Agent Memory
A layered architecture for agent memory: context window, session memory, persistent memory, and skills, with explicit write paths, retrieval, and decay.
- Agent memory is a storage hierarchy, not a feature: context window → session memory → persistent memory → skills, with each tier slower to reach, cheaper to hold, and more durable than the one above it.
- The context window is the cache, not the database. Treating the transcript as the system of record guarantees data loss at the first compaction, restart, or new session.
- Design the write path explicitly: decide at write time whether an observation earns storage at all, which tier it belongs in, and whether it updates an existing record or creates a new one.
- Retrieval is the hard half. A memory that is not recalled at the right moment does not exist; score candidates on relevance, recency, and importance, and load them under a token budget.
- Memory needs an eviction policy. Decay, consolidation, and deletion are what keep a growing store useful—an append-only memory converges on noise plus contradictions.
- Recalled memories are data, not instructions: mark their provenance and age, verify stale facts before acting on them, and gate writes from untrusted content against poisoning.
- Measure memory like a cache: recall precision, staleness incidents, token cost per recall, and how often the agent re-derives something it already knew.
What & why
An agent's model is stateless: every turn, it knows exactly what is in its context window and nothing else. Memory is the set of systems around the model that decide what gets written down, where it lives, when it comes back into context, and when it is forgotten. Get those decisions wrong and the agent re-learns the same facts every session, contradicts itself, or drowns its working context in history.
The useful frame is a storage hierarchy, familiar from every other computing system. Registers, cache, RAM, and disk trade speed against capacity and durability; agent memory tiers trade context cost against persistence in exactly the same way. The engineering problems that follow are also familiar: admission policies, eviction policies, cache invalidation, and index design [1] [2].
This matters more as agents run longer. A single bounded task can live entirely in one context window. An agent that works across sessions, projects, and weeks needs facts that survive restarts, preferences that survive projects, and procedures that survive everything—without paying for all of it on every turn [3].
Memory is also where an agent's behavior becomes legible or opaque. A file- or record-based store can be read, corrected, and deleted by its operator; behavior baked into an ever-growing transcript cannot. The architecture below keeps every tier inspectable.
Mental model
Think in four tiers. The context window is the working set: everything the model can attend to right now, paid for in tokens and attention on every single call. Session memory is the state of one run: the transcript, scratch files, and checkpoints that survive compaction but not the session. Persistent memory is the cross-session system of record: durable facts, preferences, and project state. Skills are the top of the hierarchy: procedures so distilled they behave like deployed code rather than data [2].
Movement between tiers mirrors a cache. Promotion is retrieval: a persistent fact is loaded into context because the current task matches it. Demotion is write-back: an observation made in context is distilled into a persistent record before the transcript that contains it disappears. Eviction is decay: records that stop earning recalls are consolidated or deleted.
The key inversion for engineers new to agents: in classic systems the cache is an optimization and the database is the truth, and losing the cache is harmless. In agents, everything is born in the cache—the context window is the only place new information ever appears—so anything not deliberately written down to a lower tier is lost when the window scrolls, compacts, or resets [1].
That makes the write path, not storage, the design center. Storage is trivial; deciding what to keep, in what shape, at which tier, is the architecture.
The four tiers
- 01Context window: the working set
Everything competes here: system instructions, tool schemas, recent observations, and retrieved memories. It is the most expensive real estate in the system—every token is paid for on every model call, and crowding degrades attention on what matters.
Keep it curated, not accumulated: the goal, the smallest useful view of current state, and only the memories the current turn needs [3].
- 02Session memory: the state of one run
The transcript plus task state, checkpoints, and scratch artifacts. It survives context compaction—summaries replace old turns while files and checkpoints keep the details—but it is scoped to the task and disposable afterward [4].
Treat it like process memory: fine for intermediate work, wrong for anything that should influence next month's runs.
- 03Persistent memory: the system of record
Durable facts, user preferences, project state, and decisions, stored outside any conversation—as files, database records, or an indexed store the agent reads and writes through tools [1] [5].
Records here need the disciplines of any database: stable identity so updates hit the same record, timestamps so staleness is visible, and provenance so a bad record can be traced and corrected.
- 04Skills: the procedural tier
Procedures distilled from completed work: not what is true, but how to do things. They are the most compressed and most reusable form of memory, retrieved by task shape rather than by topic.
This tier has its own loop—distillation, review, retrieval, retirement—covered in the Skill Learning Loops recipe.
Design the write path
- 01Decide what earns a write
Admit a record when it is durable, non-obvious, and likely to change future behavior: a hard-won environment quirk, a stated preference, a decision with reasons. Skip what is derivable on demand from the code, the environment, or a search.
Hoarding is the default failure. Every low-value record admitted today is retrieval noise and consolidation work tomorrow.
- 02Choose the tier at write time
Ask how long the information should outlive its origin: this turn (leave it in context), this task (session state), future sessions (persistent record), or future tasks of the same shape (a skill).
Misfiled writes are worse than missing ones—a preference buried in a transcript is lost, and a one-off detail promoted to persistent memory pollutes every future recall.
- 03Pick write-through or write-behind
Write-through records a fact the moment it is observed: nothing is lost to a crash or compaction, at the cost of interrupting the task and admitting unvetted records. Write-behind runs a reflection pass at task end, distilling the whole run with hindsight: better records, but anything before a mid-task failure is lost.
Production systems mix them: write-through for cheap, clearly durable facts; a write-behind reflection pass for judgment-heavy distillation [1].
- 04Normalize and deduplicate on write
Resolve every write against existing records: update the record it contradicts or refines rather than appending a rival version. An append-only fact store accumulates contradictions the retrieval layer cannot adjudicate.
Normalize at the boundary—convert relative dates to absolute, resolve pronouns and aliases to stable names—because the reader of this record will not share the writer's context.
- 05Gate writes from untrusted content
A memory write is a deferred prompt injection opportunity: text an agent read in a webpage or document today can steer every future run if it lands in the store. Scan writes for instruction patterns and exfiltration attempts, and record which run and which sources produced each record [6].
The same write-path defenses covered in Skill Learning Loops apply to fact memory—the skill tier is just the highest-privilege case.
Design the read path
- 01Index for the recall moment
A record is only as good as its retrievability. Index memories by what future tasks will look like—keywords, entities, and embeddings over a one-line summary—rather than by how the writing session happened to phrase them.
Hybrid retrieval beats any single index: exact match catches identifiers and names that embeddings blur, embeddings catch paraphrases that keywords miss.
- 02Score candidates on relevance, recency, and importance
Pure similarity search recalls whatever sounds alike. Weight it with recency, so current facts beat superseded ones, and importance, so consequential records beat trivia—the scoring model established by the generative-agents work [7].
Tune against real traces: the failure to look for is the right memory existing but losing the ranking to noise.
- 03Budget the recall
Load top-k summaries, not full bodies, and let the agent fetch details on demand—the same progressive disclosure that keeps a large skill store affordable. Recall that floods the window defeats the purpose of having tiers.
A recalled memory should displace cheaper context only when it earns the space; measure token cost per recall alongside hit rate.
- 04Mark provenance and verify before acting
Present recalled records as dated background data—written then, by that process, from those sources—not as fresh instructions. The model should treat them the way an engineer treats an old runbook: probably right, worth a quick check.
For any recalled fact that names a file, flag, endpoint, or person, verify it still holds before building on it. Stale memory confidently applied is worse than no memory.
Decay, consolidation, and deletion
- 01Compact the session before it compacts you
Inside a run, summarize resolved history and move durable facts and artifacts out of the transcript before the window forces a lossy compaction at an arbitrary point [3] [4].
Compaction should be a planned write-back to lower tiers, not an emergency that silently deletes whatever happened to be oldest.
- 02Give persistent memory an eviction policy
Track last-recalled time and recall count per record—the LRU and LFU signals—and put explicit expiry on anything time-bound, like 'the migration is running this week.' A record that has never been recalled and no longer matches anything is occupying index space and ranking attention for nothing.
Eviction can be soft first: archive out of the index before deleting, so a mistaken eviction is recoverable.
- 03Run consolidation passes
Periodically merge near-duplicates, resolve contradictions in favor of the newer or better-sourced record, and rewrite sprawling clusters into one clean entry—the same job compaction does in a log-structured store. This is also where write-behind quality problems get repaired.
Consolidation is a reviewable batch job: run it, diff the store, and keep the history so a bad pass can be reverted.
- 04Make deletion a first-class operation
Wrong memories are worse than missing ones, and some records—personal data, credentials captured by accident, anything the user asks to forget—must be genuinely removable, including from indexes and derived summaries.
If your memory design has no delete path, it is not a memory system; it is a liability with an embedding index.
Common failure modes
- Treating the transcript as the database, so every compaction or new session silently loses state.
- Writing everything down, so retrieval drowns high-value records in trivia.
- Append-only stores that accumulate contradictory facts with no resolution rule.
- Records without timestamps or provenance, making staleness and poisoning invisible.
- Embedding-only retrieval that misses exact identifiers, names, and error strings.
- Recall that dumps full record bodies into every turn instead of budgeted summaries.
- Trusting recalled facts as current without verifying the ones the task depends on.
- Filing procedures in the fact store and facts in prompts, so neither tier retrieves well.
- Accepting memory writes derived from untrusted content without scanning or provenance.
- Having no deletion path for wrong records or data the user asks to forget.
Final checklist
Read next
- Skill Learning Loops — The procedural tier in depth: distilling completed tasks into skills and securing the write path.
- Context Engineering — The top tier of the hierarchy: budgeting, placing, and evicting what sits in the window itself.
- Agent Loops — The runtime that produces observations worth remembering and consumes what memory recalls.
References
- [1] MemGPT: Towards LLMs as Operating Systems — The virtual-memory framing: a bounded context window paging records in and out of external storage.
- [2] Cognitive Architectures for Language Agents — The CoALA taxonomy of working, episodic, semantic, and procedural memory for language agents.
- [3] Effective context engineering for AI agents — Anthropic's guidance on curating the context window, compaction, and structured note-taking.
- [4] OpenAI Agents SDK: Sessions — Session-scoped conversation state as a first-class runtime concept, distinct from durable storage.
- [5] Claude memory tool — A file-based persistent memory the model reads and writes through explicit tool calls.
- [6] OWASP Top 10 for LLM Applications — Risk taxonomy covering prompt injection and poisoning classes that apply to memory write paths.
- [7] Generative Agents: Interactive Simulacra of Human Behavior — The memory-stream design scoring retrieval by relevance, recency, and importance, plus reflection.
Subscribe to all Software Cookbook updates
If this recipe was useful, subscribe for the next ones. New pages arrive with the same concise, technical format.