6.3 Context Window Management: Budget, Rot, and Curation Techniques

It's tempting to think of the context window as a container with separate compartments — some space reserved for the system prompt, some for tool schemas, some for history. It isn't. The context window is finite, and it is not reserved for any one purpose: the system prompt, tool schemas, conversation history, tool results, and retrieved documents all compete for the exact same budget. Every token spent on one of these categories is a token unavailable to the others.

This reframes a question you might otherwise treat casually — "how much should I retrieve?", "how much of this tool result should I keep?" — into a budgeting decision with real opportunity cost. Track how full the window is rather than assuming raw capacity is effectively unlimited just because the model supports a large context window on paper.

Here's the counterintuitive part: the problem isn't only running out of space. Models suffer **context rot** — attention degrades as the window fills with irrelevant or stale content, even when there is technically still room left in the window. Low-signal tokens crowd out the high-signal ones the model actually needs to attend to, the same way a desk piled with irrelevant paperwork makes it harder to find the one document that matters, even though there's technically still room on the desk.

This is why "just use the biggest context window and stuff everything in" is a trap dressed up as a solution. Bloated context causes context rot and higher cost regardless of how large the window's rated capacity is — curation beats capacity, every time. Don't dump entire documents or full tool outputs into the window when a summary or the relevant slice would do; that's the practical, day-to-day form this principle takes.

Context rot's most visible symptom in a real session is **context drift**: the model gradually loses track of instructions or facts over a long interaction. The telltale sign is a long agent session that starts ignoring earlier instructions and quality drops, even though nothing about the task itself changed — the instructions are still just as valid as they were at the start, but they're now buried under everything that's accumulated since.

When this happens, the instinct to reach for "raise temperature" or "add more prohibitions" is understandable but wrong — neither one addresses a full, low-signal context window. Temperature affects sampling randomness, not what's competing for the model's attention; more prohibitions just add more tokens to an already crowded window. The actual fix is compacting older turns into a summary and pruning stale tool output — reclaiming a clean, high-signal window rather than tweaking an unrelated dial.

With the diagnosis in hand — rot and drift come from a crowded, low-signal window — the fix is a set of active curation techniques, and the first two work on what's already accumulated. **Tool-output pruning**: after a tool returns a large payload, keep only what later steps actually need, and drop raw dumps from the ongoing history. A verbose tool result that was necessary to inspect once does not need to persist in full for the rest of the session.

**Compaction**: periodically summarize older turns into a compact recap and continue from it, reclaiming budget while preserving the thread. This is how long agent sessions avoid drift — rather than carrying the entire transcript forward indefinitely, the session's history is condensed into what still matters. Compaction is explicitly not the same as truncation: compaction summarizes to reclaim budget while preserving meaning; truncation blindly drops tokens and can lose critical facts the task still depends on. They can look similar in effect — context shrinks either way — but they differ sharply in whether meaning survives.

The just-in-time retrieval technique covered next in this lesson assumes some mechanism for finding the right slice of material to pull in. That mechanism has a name and two genuinely different implementations, and the exam expects you to tell them apart by more than just the name: **retrieval-augmented generation (RAG)**. Rather than loading an entire knowledge base into context, RAG stores material outside the context window, finds the parts relevant to the current request, and supplies only that slice to the model.

**Classical RAG** does its hard work upfront, before any question is ever asked. Source documents get split into chunks, each chunk is converted into an embedding — a vector of numbers capturing its meaning mathematically — and those vectors are stored in a searchable index. At query time, the incoming question is embedded the same way, and a similarity search against the index retrieves the chunks whose embeddings are closest to the question's. Picture a librarian who read every book before the library opened and wrote a precise summary card for every chapter — when you arrive with a question, the matching cards are already sitting there, ready to hand over.

Classical RAG concretely fails in three specific places, and each one sounds like a minor implementation detail while actually determining whether retrieval works at all. **Chunking**: too-small chunks lose the surrounding context a fact needs to make sense; too-large chunks dilute the one relevant sentence inside a block of mostly-irrelevant text, wasting context tokens and blurring the embedding. Sentence- or section-based chunking with some overlap between adjacent chunks is a reasonable default, so a boundary doesn't sever the exact fact a query needs. **Embedding match**: similarity search operates on semantic closeness, which is usually the point — it's what lets a query about "refund policy" match a chunk that says "return window" — but that same property can miss a query that hinges on an exact term or identifier (a SKU, an error code, an exact parameter name), where a semantically-similar-but-wrong chunk outranks the one with the literal string that was needed. Pairing embedding search with a lexical/keyword match catches exactly what pure similarity search misses on identifier-heavy queries. **Assembly**: retrieving the right chunks doesn't guarantee the model uses them — if the assembled prompt's structure doesn't match what the instructions lead the model to expect (undelimited chunks dumped in, or placed somewhere the prompt doesn't point at), the model can silently answer from its own training-data memory instead of the retrieved context, with no error or signal that anything went wrong.

**Agentic search** skips the upfront indexing step entirely — there is no pre-built vector database. Instead, the model searches live, at the moment a question arrives, figuring out what it needs and fetching it on the spot: searching sources, reading documents, pulling in results as the task unfolds. Two concrete examples you may already have encountered without the name: Claude Code's own MCP tool-discovery mechanism (rather than loading every connected server's tool definitions up front, it discovers and loads only the tools a given task needs, searching for them live), and Claude.ai Projects over a knowledge base too large to fit in context (it surfaces only the sections most relevant to each question, found at question time rather than pre-indexed).

Strip away the mechanics and both approaches do the same underlying job — surface the relevant piece of material and generate an answer from it. What separates them is purely timing: one matches against an index built ahead of time, the other looks things up live, right when it's needed. Neither wins outright — each trades a different set of costs for a different set of benefits.

Agentic search's flat-cost-at-scale property is genuinely useful, but it comes with a caveat worth remembering: agentic search is only as good as what it can actually find. Because there's no pre-built index doing the matching for it, the model relies on live search and its own judgment about what to look for and where — which means how source material is organized and named matters far more for agentic search than for classical RAG's embedding-based matching. A file named `notes_final_v3.pdf` is much harder for agentic search to correctly retrieve than one named `Q3 refund policy, updated August 2024` — the descriptive name is itself a retrieval signal a live search step can act on, where a vague version-numbered name gives it nothing to go on.

The other two curation techniques control the window from a different angle: instead of cleaning up what's already inside, they limit what enters in the first place. **Context isolation**: delegate a heavy subtask to a subagent with its own context window; it returns a condensed result so the main context stays clean. The verbose intermediate work of exploring, researching, or drafting happens inside the subagent's window and never has to enter the main session's budget at all.

**Just-in-time retrieval**: pull information into context when it's needed rather than front-loading everything up front. If a task might need any of ten documents but will actually reference two, retrieving just those two on demand — instead of loading all ten preemptively — keeps the window's signal-to-noise ratio high by construction, before pruning or compaction ever has to clean anything up.

These four techniques — pruning, compaction, isolation, and retrieval — are complementary, not interchangeable. Pruning and compaction manage what's already in the window; isolation and retrieval control what enters it in the first place. A well-curated agent session typically combines all four rather than leaning on just one.

• •Pruning — trims a large tool result down to what later steps need, after it arrives.
• •Compaction — condenses accumulated history into a recap, preserving the thread.
• •Isolation — delegates heavy work to a subagent with its own window, returning only a condensed result.
• •Just-in-time retrieval — pulls information in only when needed, rather than front-loading everything.