RAG Mechanics: Classical Retrieval vs. Agentic Search

Retrieval Is the Mechanism Behind Just-in-Time Retrieval

The just-in-time retrieval technique in the previous concept assumes some mechanism for finding the right slice of material to pull in. Retrieval-augmented generation (RAG) is the general pattern that does this: rather than loading an entire knowledge base into context, the system stores material outside the context window, finds the parts relevant to the current request, and supplies only that slice to the model. There are two fundamentally different ways to implement that mechanism, and the exam expects you to distinguish them by more than name.

Classical RAG: Index Now, Search Later

Classical RAG does its hard work upfront, before any question is asked:

1. Chunk the source documents into smaller pieces.
2. Embed each chunk -- convert it into a vector of numbers that captures its meaning mathematically.
3. Store those vectors in a searchable vector index.
4. At query time, embed the incoming question the same way, then run a similarity search against the index to retrieve the chunks whose embeddings are closest to the question's embedding.

This is the librarian-with-pre-written-summary-cards model: all the expensive work (reading everything, writing the cards) happens before the library opens, so that when a question arrives, the matching cards are already sitting there ready to hand over.

Where Classical RAG Concretely Fails

Classical RAG breaks down in three specific, nameable places -- these are the exam's favorite trap surface for this concept, because each failure sounds like a minor implementation detail but actually determines whether retrieval works at all:

(a) Chunking. Chunk size is a real design decision with a real failure mode on both ends. Too-small chunks lose the surrounding context a fact needs to be understood correctly -- a sentence pulled with no paragraph around it can be technically retrieved and still be useless or misleading. Too-large chunks dilute the signal: the one relevant sentence gets buried in a large block of mostly-irrelevant surrounding text, which both wastes context-window tokens and can lower the embedding's precision (a big chunk's embedding represents an average of everything in it, not the one fact that matters). A reasonable default is sentence- or section-based chunking with some overlap between adjacent chunks, so a chunk boundary doesn't sever the exact fact a query would need.

(b) Embedding match. Similarity search operates on *semantic* closeness, not exact matching. This is usually the point -- it's what lets a query about "refund policy" match a chunk that says "return window" even though the words differ. But that same property means embedding search can miss a query that hinges on an exact term or identifier: a specific product SKU, an error code, an exact API parameter name. Two chunks can be semantically similar to a query while the one chunk containing the *exact string* the query needed sits just outside the embedding-similarity threshold. Pairing embedding search with a lexical/keyword match (a plain text/substring search running alongside the vector search) catches exactly what pure similarity search misses on identifier-heavy queries.

(c) Assembly. Retrieving the right chunks doesn't automatically mean the model uses them. If the assembled prompt's structure doesn't match what the model's instructions lead it to expect -- retrieved chunks dumped in without clear delimiting, or placed somewhere the prompt doesn't point the model toward -- the model can silently answer from its own training-data memory instead of the content that was just retrieved, ignoring the retrieval step's entire output without any error or signal that anything went wrong. This connects directly to the delimiter and placement techniques from earlier in this domain (Lesson 6.2): retrieved content needs to be clearly delimited and the instruction needs to point at it explicitly, or retrieval quietly does nothing.

Agentic Search: No Index, Search at the Moment of Need

Agentic search skips the upfront indexing step entirely -- there is no pre-built vector database. Instead, the model searches live, at query time, figuring out what it needs the moment the question arrives and fetching it on the spot: querying whichever live sources are available, opening documents as they're needed, weaving new findings in as the work progresses. This is the researcher-who-goes-and-finds-it model, as opposed to the librarian with cards prepared in advance.

Two concrete, real-world examples of agentic search you may already have encountered without the name: Claude Code's own MCP tool-discovery mechanism (when connected to many external tools, Claude doesn't load every tool definition up front -- it discovers and loads only the tools a given task needs, searching for them live rather than pre-indexing them), and Claude.ai Projects over a large knowledge base (when a project's material is too large to fit in context, the system surfaces only the sections most relevant to each question, found at question time rather than pre-indexed).

Classical RAG and Agentic Search Do the Same Job, on Different Timing

Strip away the mechanics and both approaches are doing the same underlying job: surface the relevant piece of material and build the answer from it. What separates them is purely when that surfacing happens. One matches against an index that was already sitting there; the other goes and looks the moment it's needed. Calling either one categorically "better" misses the point -- each is buying a different set of tradeoffs.

The Fetch-Once vs. Iterative-Search Tradeoff

Agentic search's flat-cost-at-scale property is genuinely useful, but it comes with an important caveat: agentic search is only as good as what it can actually find. Because there's no pre-built index doing the matching for it, the model is relying on live search and its own judgment about what to look for and where -- which means how source material is *organized and named* matters far more for agentic search than it does for classical RAG's embedding-based matching. A file named notes_final_v3.pdf is much harder for agentic search to correctly retrieve than one named Q3 refund policy, updated August 2024 -- the descriptive name is itself a retrieval signal that a live search step can act on, where a vague version-numbered name gives it nothing to go on.

Common exam traps

• Treating chunk size as a minor implementation detail rather than a real tradeoff -- too-small loses context, too-large dilutes signal, and "sentence/section-based with overlap" is the reasonable default to reach for.
• Assuming semantic/embedding search alone is sufficient for queries that hinge on an exact term, code, or product identifier -- pairing it with lexical/keyword search is the fix, not a nice-to-have.
• Assuming that retrieving the correct chunks guarantees the model will use them -- a mismatched or undelimited assembly can cause the model to silently answer from its own training data instead of the retrieved content.
• Treating classical RAG and agentic search as differing in *quality* rather than *timing* -- they do the same fundamental job (find a relevant slice, generate from it); the difference is whether the index was built in advance or the search happens live.
• Assuming agentic search works equally well regardless of how source material is organized -- vague or generic file/document names actively hurt its ability to find what's needed, since there's no embedding index doing that work for it.