Context Engineering as an Architecture Discipline: Memory, Retrieval, and Compaction

Most teams treat context engineering as a prompt-tuning exercise: shuffle instructions, add a few examples, adjust the system message until the agent stops doing the annoying thing. That framing undersells the problem badly. Once an agent runs for more than a few turns, or across sessions, what you’re actually managing is a memory system with a hard capacity limit and a real cost per byte. That’s an architecture problem, not a wording problem, and it deserves the same rigor we’d apply to cache design or data modeling.

Context Is a Scarce, Expensive Resource

Every token in the context window is doing one of three things: informing the next decision, informing a future decision, or doing nothing. The first category is what you want. The second is memory you haven’t organized yet. The third is waste you’re paying to carry around and waste that pushes signal out of the model’s effective attention. Treating context as an undifferentiated blob of “everything that happened” is the equivalent of never evicting anything from a cache — it works until it very much doesn’t, usually right when a task gets complex enough to matter.

Tiered Memory: Working, Episodic, Semantic

The fix is to stop thinking of context as one thing and split it into tiers with different lifetimes and different fidelity requirements:

• Working memory — the current task’s live state: the immediate conversation, recent tool outputs, and anything the agent needs verbatim to finish the step it’s on. High fidelity, short-lived, expensive to keep large.
• Episodic memory — a record of what happened during a run: decisions made, tools called, outcomes observed. This is what lets you resume an interrupted task or audit why an agent did something. It doesn’t need to be verbatim; it needs to be reconstructable.
• Semantic memory — durable facts distilled across many episodes: project conventions, user preferences, standing decisions. This persists across sessions and should look more like a knowledge base entry than a transcript.

The practical test for which tier something belongs in is simple: will I need the exact wording again, or just the fact? Error messages, IDs, and code diffs need exact wording — keep them in working or episodic memory verbatim. “The user prefers terse commit messages” is a fact — it belongs in semantic memory, distilled once and reused indefinitely.

Compaction Without Losing the Thread

Compaction is where most context architectures quietly fail. Naive rolling summarization — “summarize the last N turns into a paragraph” — is lossy in ways that are hard to detect until the agent acts on a fact that got smoothed over. The summary reads fine; it’s just missing the one number or flag that mattered.

Two changes make compaction more reliable:

1. Prefer structured extraction over prose summarization. Instead of asking a model to write a paragraph about what happened, extract a fact table: key decisions, open questions, and any values (IDs, paths, flags) that must survive verbatim. Structured formats are far less prone to silent drift than free-text summaries, and they’re diffable, so you can review what a compaction step actually dropped.
2. Maintain a verbatim allowlist. Some things should never be summarized: exact error strings, file paths, credentials references, and anything with a version number attached. Pull these out before compaction runs and reattach them afterward.

{
  "decisions": ["Use Postgres for the events table, not DynamoDB"],
  "open_questions": ["Retention period for raw events — TBD with compliance"],
  "verbatim": {
    "last_error": "ECONNREFUSED 10.0.4.12:5432",
    "pr_url": "https://github.com/org/repo/pull/482"
  }
}

Run compaction as a testable function with fixed inputs and expected outputs, the same way you’d test a cache eviction policy. If you can’t write a test for “does this compaction preserve the facts that mattered,” you don’t actually know that it does.

Measuring Context Quality, Not Just Retrieval Hit Rate

Retrieval hit rate — did the relevant document get pulled into context — is the metric everyone reaches for first, and it’s necessary but not sufficient. An agent can retrieve the right document and still fail the task because the retrieved chunk lacked a dependency, or because ten other irrelevant chunks buried it in noise. Hit rate tells you the memory system found the needle; it doesn’t tell you the agent used it correctly.

Better signals to track alongside hit rate:

• Task completion rate conditioned on retrieval success — did having the right context actually translate into the right outcome?
• Context waste — the fraction of tokens in the final prompt that had no bearing on the response the model produced. High waste means your relevance filtering isn’t working even if hit rate looks good.
• Re-ask rate — how often the agent has to ask a clarifying question or re-fetch something it should have already had. A rising re-ask rate is often the first symptom of a compaction step dropping something it shouldn’t have.

None of these require exotic tooling. They require you to log what went into the prompt, what came out, and whether the outcome was correct — the same instrumentation discipline you’d want for any production system.

Key Takeaways

• Split context into working, episodic, and semantic tiers instead of one undifferentiated buffer — each has a different fidelity and lifetime requirement.
• Compact with structured extraction plus a verbatim allowlist, not free-text summarization alone, and write tests for what compaction is supposed to preserve.
• Track task completion, context waste, and re-ask rate alongside retrieval hit rate — hit rate alone can’t tell you whether the agent actually used what it retrieved.
• Treat your memory schema as a versioned artifact, not an implicit convention that lives only in prompt templates.