Budgeting the Context Window: Compaction, Retrieval and Memory for Agents

An agent’s context window is the only memory the model has during a call, and it fills in predictable ways. The system prompt and tool definitions take a fixed share. Every tool result adds to it and stays. Retrieved documents get added “just in case”. By step 20, much of the window is old tool output that no longer matters, and the model is paying attention to all of it.

That costs money, because every token is re-sent on every call. It also costs quality. Chroma’s context rot research found performance on simple tasks degrading as input length grows, well before the window is full. Anthropic’s guidance on effective context engineering frames the goal as the smallest set of high-signal tokens that gets the job done.

In practice that means managing the context window the way you’d manage memory in a constrained system.

Give each component a budget

Start by measuring. Log the token count of each part of the context on every call: system prompt, tool definitions, conversation, tool results and retrieved content. Most teams find one part dominating in a way they didn’t expect, often tool definitions or a few huge tool results.

Then set budgets. An example for an agent working with a 200,000-token window:

The numbers are illustrative. The point is that each component has an owner and a limit, and something in code enforces it. Leaving most of the window unused is deliberate. Quality and cost both favour a context that’s much smaller than the maximum.

Compaction in code, not by request

Asking the model to “keep things concise” doesn’t reclaim context. A compaction policy in the harness does. Run it before each model call:

The following pseudocode illustrates the order of work. A production implementation must preserve tool-call/result pairs, protect pinned instructions, validate the final token count and handle a recent turn that alone exceeds the budget.

def compact(messages, budget, keep_recent=6):
    """Shrink history to fit the budget, oldest and bulkiest first."""
    if count_tokens(messages) <= budget:
        return messages

    old, recent = messages[:-keep_recent], messages[-keep_recent:]

    # 1. Replace old tool results with a one-line stub. The call is kept, the payload dropped.
    old = [stub_tool_result(m) if is_tool_result(m) else m for m in old]
    if count_tokens(old + recent) <= budget:
        return old + recent

    # 2. Summarise everything old into a single note.
    summary = summarise(old)   # small, cheap model; output capped at ~1,000 tokens
    return [summary_message(summary)] + recent

Stubbing old tool results can be cheaper than model summarization, but may remove evidence needed later. The model still sees that it called search_invoices and roughly what came back (“[312 rows; 3 matched dispute; details dropped]”), but not the full payload. Summarisation loses more, so it comes second. Keep the most recent turns intact, because they’re what the next step depends on.

Two rules keep compaction from causing new bugs:

• Never compact away the task’s constraints. The goal, the limits and anything the user explicitly required belong in pinned task state, which compaction doesn’t touch.
• Log every compaction, with before and after token counts. When an agent “forgets” something, the compaction log is the first place to look.

Retrieve just in time

Loading everything that might be relevant at the start of a session fills the context with material the agent may never use. The alternative is to give the agent tools to fetch what it needs when it needs it: search_docs(query), read_file(path, lines), get_record(id). Keep only lightweight references (IDs, file paths, titles) in context until then.

This is how coding agents work well in large repositories. They don’t load the codebase. They list, search and read specific files. It works for other agents too, provided the retrieval tools return concise, well-labelled results (the MCP tool design post covers this).

Upfront retrieval still makes sense when the relevant material is small and always needed, such as a customer’s account summary in a support agent. Budget it like everything else.

Memory outside the window

For work that spans many sessions or outlasts compaction, the agent needs somewhere to write things down: a notes file, a task list, or a structured store it reads at the start of each session. The long-running agents post describes the progress-file pattern.

Treat that memory as data, not as trusted instructions. An agent that writes notes based on untrusted content (a web page, an email) can end up storing injected instructions, which then get read back with the authority of “my own notes”. Record where each memory came from, and let people inspect and delete it.

Sub-agents for clean contexts

When a sub-task needs a lot of reading but produces a short answer (“find which of these 40 log files contains the first error”), give it to a sub-agent with its own fresh context. The sub-agent reads the 40 files, and the parent gets back two paragraphs. The parent’s context grows by the answer, not the investigation.

This costs more tokens in total, because the sub-agent has its own system prompt and reading. Anthropic’s research system write-up reported multi-agent runs using about 15× the tokens of a chat. Use sub-agents where the context isolation is worth that. Don’t use them by default.

Where this stops

Budgets and compaction keep the context manageable. They can’t tell you which tokens are high-signal for a given task. That judgment lives in the design of your tools, retrieval and prompts, and the only way to check it is to run the agent on real tasks and compare outcomes as the budgets change.