12 min read

Context Engineering

The infra engineer's treatment of the context window: a fixed-capacity cache with per-layer budgets, eviction policy, placement discipline, and observability.

TL;DR
  • Context engineering treats the model's context window as a fixed-capacity cache with explicit per-layer budgets, an eviction policy, and observability—not as a document you keep appending to.
  • Attention is finite and every token competes for it. Past a saturation point, more context means worse recall, higher cost, and higher latency—longer context is not free and is often net-negative.
  • The window has four layers—system prompt and tool schemas, session history, retrieved content, persistent memory and skills—and history usually consumes 60–80% of the tokens. Budget each layer and treat overruns as bugs.
  • Tool definitions are context. A handful of connected servers can cost tens of thousands of tokens before any reasoning happens; load definitions on demand instead of upfront.
  • Compaction is a first-class operation with a known failure mode: summarizing away a constraint that matters forty turns later. Keep decisions, constraints, and unresolved errors raw.
  • Keep failed tool calls and stack traces in context. Cleaning up errors—the instinct of every engineer—measurably increases repeat mistakes.
  • Debug by layer, not by prompt wording: instrument tokens per layer per turn before touching a single sentence of the system prompt.
Quick view
Best for
Long-running, tool-heavy agent systems
Core primitive
Budgets, tiers, eviction, observability
Do first
Instrument tokens per layer per turn
Primary risk
Attention pollution from unbudgeted layers

What & why

The term settled in mid-2025—Shopify's Tobi Lütke and Andrej Karpathy popularized it, and Anthropic's engineering guide gave it the working definition: finding the smallest set of high-signal tokens that maximizes the likelihood of the desired outcome [1]. Prompt engineering did not die; it became a subskill—the wording of one layer. The canonical operations are write, select, compress, and isolate, and that is the last you will hear of them as an organizing scheme.

This recipe is the infrastructure treatment. The context window is a cache, not a document: it has fixed capacity, everything in it competes for a shared resource, and it degrades in predictable ways when polluted. Engineering it means what cache engineering has always meant—budgets, tiers, an eviction policy, placement rules, and instrumentation—plus a failure-mode taxonomy so you can debug the right layer instead of rewording prompts at random.

The discipline matters most where agents live now: long-horizon runs and heavy tool surfaces. Every marginal token costs money and latency on every subsequent call, and past a saturation point it also costs accuracy—the model's effective recall over a bloated window drops before the window is anywhere near full [1] [4].

Mental model: the window is a cache

A transformer pays attention over every token it holds, so the window behaves like a cache with a fixed capacity and a shared bandwidth budget. Admission is cheap—appending is one line of code—which is exactly why pollution is the default failure. Stale tool results, resolved sub-tasks, and speculative just-in-case documents occupy capacity that the current decision needed.

The hierarchy extends below the window the way storage extends below RAM. Content the model does not need this turn belongs in files, databases, or memory stores, retrieved on demand; the window holds the working set, not the corpus. Compaction passes are the GC pauses of this world: deliberate stop-and-collect moments between reasoning steps, paid for occasionally rather than per turn.

Locality matters too. Models retrieve information placed at the start and end of the window far more reliably than the middle—the lost-in-the-middle effect is one of the better-replicated results in the literature [4]. Placement is therefore a real design decision: stable instructions at the top, the live task and freshest observations at the bottom, and nothing load-bearing buried in between.

State the physical constraint plainly and design against it: attention is finite, every token competes, and precision falls as context grows. The question is never 'can we fit it?' but 'does this token earn its slot on this turn?'

The mapping, term by term

  • Cache with fixed capacity → the context window and its attention budget.
  • Cache pollution → tool-schema bloat, stale tool results, resolved history riding along turn after turn.
  • Eviction policy → compaction and summarization rules: what leaves, what stays raw, when.
  • Tiered storage (RAM → disk) → offloading to files and stores, retrieved on demand instead of held resident.
  • GC pauses → periodic compaction passes between major reasoning steps, amortized rather than per-turn.
  • Cache locality → lost-in-the-middle: critical content at the edges of the window, never the middle [4].

Four layers, four budgets

  • System prompt and tool schemas: the resident set. Changes rarely, loads on every call; typically 5–15% of tokens—more if tool surfaces go unmanaged, which is the first place to look.
  • Session history: the growth layer. Accumulates every turn and typically runs 60–80% of the window; this is the layer eviction policy exists for.
  • Retrieved and external content: the burst layer. Tool results, documents, and search output arriving on demand; budgeted top-k with details fetched when needed, never dumped whole.
  • Persistent memory and skills: the recall layer. Smallest share, loaded selectively when relevant—the tiers below the window, covered in Agent Memory and Skill Learning Loops.

Practical moves

  1. 01
    Budget per layer, explicitly

    Assign a token budget to the system prompt, tool schemas, history, and retrieval, and instrument actual usage against it per turn. A layer running past budget is a bug with an owner, not ambient weather.

    Budgets force the useful arguments—whether a tool description earns 400 tokens, whether history needs the last thirty turns raw—to happen at design time instead of as production mysteries.

  2. 02
    Treat tool schemas as paid context

    Every tool definition rides along on every call: a complex schema eats 500+ tokens, and a few connected MCP servers can mean 90+ tools and tens of thousands of tokens before the model has reasoned at all. Degradation in tool selection starts around ten tools; past twenty, you are paying for confusion [5].

    The fix is progressive disclosure: keep names and one-line descriptions resident, load full definitions on demand via tool search. This is how frontier products now ship, and it routinely reclaims most of the schema budget [5]. The Model Context Protocol recipe covers the schema surface itself.

  3. 03
    Run compaction as a first-class operation

    Compact periodically between major reasoning steps, not per turn—the summarization call has real cost, so amortize it. Keep raw: decisions and their reasons, standing constraints, and unresolved errors. Summarize: resolved sub-tasks and exploration that ended in a conclusion.

    The known failure is summarizing away an early detail that becomes load-bearing forty turns later. Compaction rules deserve evaluation like any other lossy compression: run long-horizon tasks with and without, and diff the outcomes.

  4. 04
    Keep errors in context

    Leaving failed tool calls, wrong paths, and stack traces visible measurably reduces repeat mistakes—the model updates on evidence it can see. Manus reported this as one of the clearest wins in production agent building [2].

    This is counterintuitive to engineers trained to clean up after failures. Resist the instinct: an error message is among the highest-signal tokens in the window. Evict it only after it is genuinely resolved.

  5. 05
    Enforce placement discipline

    Stable, global content—identity, rules, constraints—goes at the top of the window. The current task, the freshest observations, and anything the next decision hinges on go at the end. The middle is where recall goes to die [4].

    This falls out of cache-friendly data layout thinking: hot data where access is cheap, and no critical dependency in the region with the worst hit rate.

  6. 06
    Isolate with sub-agents

    Partitioning is the isolation strategy: give a child agent its own window scoped to one slice of the problem, and let the parent pay only for the conclusion. Token-heavy exploration never touches the parent's working set.

    Sub-Agent Orchestration covers the mechanics—task contracts, budgets, synthesis; from the context side, the point is that a window shared across concerns is a window polluted by all of them.

  7. 07
    Let high-value context evolve

    The research edge: ACE—Agentic Context Engineering—treats context as an evolving, curated document rather than a static prompt, running a generate → reflect → curate loop that revises playbook-style context between runs. It reported a 10.6% gain on agentic coding benchmarks over strong baselines [3].

    This is the research validation of the pattern Skill Learning Loops describes operationally: the highest-leverage context is written by the system itself, distilled from its own outcomes, and curated on every pass.

Context as the agent's operating state

The framing worth adopting from the current wave: context engineering is the engineering of agent state—everything the agent knows, sees, and remembers at the moment of action. The window is merely the top tier of that state, the part resident in the most expensive storage. Once you hold that frame, the boundaries of this recipe become obvious: what persists below the window is the Agent Memory recipe, what runs the turns is Agent Loops, and what this recipe owns is the discipline of deciding, on every turn, which fraction of everything knowable actually earns residence.

Anti-patterns, as named smells

  • Kitchen-sink context: dumping every document and tool in 'just in case.' Attention pollution with a hope attached—each speculative token dilutes the ones doing work.
  • Prompt-first debugging: rewording instructions when the real question is which layer is leaking or starving. Diagnose in order: tokens per layer, then tool schemas, then retrieval relevance, then history and compaction—wording last.
  • Schema sprawl: connecting every available MCP server and accepting an unbounded tool list. The integration was free; the 50k resident tokens are not.
  • Over-compaction: an eviction policy so aggressive it deletes constraints. The symptom is an agent that violates a rule it acknowledged early in the run.
  • Static context files: writing CLAUDE.md or AGENTS.md once and never curating. The ACE result is precisely that curated, evolving context beats frozen context—a context file is a living config, not a founding document [3].
  • Trusting the context supply chain: retrieved documents and tool results are untrusted input, and injection and poisoning enter the system through exactly this door. Treat provenance as part of the layer contract; the defenses get their own recipe on agent security.

Measure it

  1. 01
    Instrument tokens per layer per turn

    Tag every token entering the window with its layer and emit the breakdown as a metric. Most teams discover their actual layer shares here for the first time—usually a tool-schema or history number nobody had budgeted.

  2. 02
    Find the degradation point

    Run the same task suite at increasing context sizes and plot success. Quality falls before capacity runs out; the knee of that curve is your real budget, and it is task-specific. Never assume bigger is better—verify where it stops being true.

  3. 03
    Evaluate compaction like lossy compression

    On long-horizon tasks, compare outcomes with compaction on and off, and audit failures for constraints that existed pre-compaction and vanished after. Evaluation for LLM Features covers how to make this a repeatable suite rather than a one-off experiment.

  4. 04
    Make cost and latency first-class SLOs

    Track tokens and dollars per completed task, and latency per turn, alongside quality. Context bloat announces itself in these numbers long before it shows up in output quality—they are the leading indicators.

Final checklist

Each context layer has an explicit token budget with an owner.
Tokens per layer per turn are instrumented and visible as a metric.
Tool definitions load on demand; the resident set stays under the degradation threshold.
Compaction runs periodically, keeps decisions, constraints, and open errors raw, and summarizes only resolved work.
Compaction quality is evaluated on long-horizon tasks, not assumed.
Failed tool calls and errors stay in context until genuinely resolved.
Nothing load-bearing sits in the middle of the window.
Token-heavy exploration is isolated into sub-agents with scoped windows.
Bulk content lives in tiers below the window and is retrieved on demand.
High-value context files are curated on a cadence, not written once.
Retrieved and tool-supplied content is treated as untrusted input with provenance.
The context-size degradation point for key tasks is measured, and cost and latency are tracked as SLOs.

Read next

  • Agent Loops — The runtime that assembles context every turn—the loop this discipline operates inside.
  • Agent Memory — The tiers below the window: write paths, retrieval, and decay for persistent state.
  • Sub-Agent Orchestration — Context partitioning in depth: isolated windows, task contracts, and fan-out economics.
  • Skill Learning Loops — The operational counterpart of ACE: context the system distills and curates from its own runs.
  • Model Context Protocol — The tool layer whose schemas are one of your largest fixed context costs.
  • Evaluation for LLM Features — The measurement machinery for degradation curves and compaction quality.

References

More recipes

Subscribe to all Software Cookbook updates

If this recipe was useful, subscribe for the next ones. New pages arrive with the same concise, technical format.

New cookbook updates, sent when there's something worth reading.