3.3 RAG Pipeline Design & Matching Retrieval Strategy to Data

Suppose your company has 50,000 support articles, and you want Claude to answer questions grounded in them. The naive idea — paste all 50,000 articles into the system prompt — fails for reasons that go beyond "it won't fit." Even with a generous context window, dumping an entire corpus into context causes context rot: the more irrelevant material surrounds the one relevant paragraph, the harder it is for the model to find and weight that paragraph correctly, and the more you're paying, on every single request, to ship text that answers nothing about THIS particular question.

This is the exam's favorite trap in this task statement, and it's worth stating baldly: a bigger context window is not a substitute for retrieval. Treating context size as the fix for "the model doesn't know about our data" gets the mechanism backwards. What you actually want is retrieval-augmented generation (RAG): given a question, find the small number of chunks of your knowledge base that are actually relevant, and put ONLY those into context. RAG isn't a workaround for a small context window — it's the right architecture even when the window is enormous, because relevance, not raw capacity, is the scarce resource.

Before you can retrieve anything, you need retrievable UNITS — this is chunking, and it's the first design decision in the pipeline, not an afterthought. Chunking splits source documents into pieces sized to be individually retrieved. Get the size wrong in either direction and you pay for it later, at query time, in ways that are hard to debug after the fact.

Chunks that are too LARGE dilute relevance: a 5,000-word chunk that contains one relevant sentence buried among four other topics still gets retrieved as a monolithic block, and the model has to find the needle in a haystack you handed it directly. Chunks that are too SMALL lose the context needed to answer: a chunk containing only "...and that's when the refund window closes" is useless without the sentence before it that says what the refund window even is. There is no universal right size — the point is to MATCH chunk size to the data's natural shape and how users actually ask about it. A FAQ with self-contained Q&A pairs might chunk naturally at one pair per chunk; a long-form policy document might need paragraph- or section-level chunks with some overlap so no boundary silently splits a critical sentence in two.

Once you have chunks, you need a way to find the right ones for a given query. There are two fundamentally different mechanisms, and understanding WHY each works the way it does is more useful than memorizing their names.

Embedding (semantic) search represents each chunk, and each query, as a vector in a high-dimensional space where nearby vectors mean similar MEANING. It shines on paraphrase: a query asking "how do I get my money back" can retrieve a chunk titled "Refund Policy" even though they share almost no words in common, because the embeddings capture that the concepts are close. But that same strength is a weakness for exact matching — embeddings are comparing meaning, not characters, so a query for a specific product code or a rare proper noun can retrieve something semantically similar but factually wrong, or miss the exact match entirely.

Lexical retrieval, most commonly BM25, is the mirror image: it matches on actual TERMS — the literal words and their statistical importance in the corpus. It has no concept of meaning at all, but that's exactly why it's precise on exactly the things embeddings struggle with: product codes, order numbers, names, rare technical terms. A query for "SKU-88123-B" will find a chunk containing that exact string reliably, where a semantic search might not treat that string as meaningfully "close" to anything.

Because real query sets mix both patterns — some paraphrase-heavy, some identifier-heavy — hybrid retrieval (embeddings + BM25, merged) is the stronger DEFAULT rather than committing to embedding-only search and hoping every query happens to be the paraphrase kind. And after hybrid retrieval surfaces a candidate set, a reranking step reorders those candidates by relevance before they reach the model, improving the precision of what the model actually sees, at the cost of an extra processing step (we'll quantify that tradeoff in Lesson 3.5).

Chunking has a structural side effect worth naming directly: once you cut a document into pieces, each piece loses the surrounding context that made it meaningful in the first place. A chunk that reads "the fee is waived in this case" is useless in isolation — WHICH case? Waived compared to what? The sentence was fine inside its document, and became ambiguous the moment it was extracted as a standalone unit for retrieval.

Anthropic's contextual retrieval technique targets exactly this failure mode. Before a chunk is embedded and indexed, you prepend a short, chunk-specific context blurb — a sentence or two, generated from the surrounding document, that restates what this chunk is about and where it sits. "This chunk is from the Enterprise Plan refund policy, section on early-termination fees" turns "the fee is waived in this case" from ambiguous into anchored. Crucially, this contextualization is applied to BOTH retrieval paths at once — contextual embeddings for the vector index, and contextual BM25 for the lexical index — so neither retrieval mechanism is working from an isolated, context-stripped chunk.

Combining contextual embeddings, contextual BM25, and a reranking step on top substantially reduces retrieval-failure rates compared to naive embedding-only RAG. Each layer compounds the previous one: contextualization keeps chunks meaningful in isolation, hybrid retrieval catches both paraphrase and exact-term queries against those now-meaningful chunks, and reranking sharpens precision on the final candidate set before it reaches the model.

The obvious objection: generating a context blurb for every chunk of a large corpus sounds expensive — you're running an extra generation step per chunk, potentially over the entire document each time. This is exactly where prompt caching earns its keep. The surrounding document content used to generate each chunk's context can be cached, so the marginal cost of contextualizing chunk #200 of a document — after chunk #1 already paid to load that document into a cached prefix — stays low. Prompt caching is what makes contextual retrieval economical at the scale of a real corpus rather than a demo.

Put the pieces in order and you get the full RAG pipeline: ingest → chunk → (add context) → embed + index (vector + lexical) → retrieve → rerank → assemble context → generate. Every lesson so far has been one stage of this pipeline — 3.3.2 was chunking, 3.3.3 was embed/index/retrieve/rerank, 3.3.4 was the "(add context)" stage that sits between chunking and indexing.

• 1.Ingest — pull source documents in from wherever they live.
• 2.Chunk — split into retrievable units sized to the data (3.3.2).
• 3.Add context — prepend a chunk-specific context blurb (3.3.4).
• 4.Embed + index — build both the vector index and the lexical (BM25) index (3.3.3).
• 5.Retrieve — run the query against both indexes and merge (hybrid retrieval).
• 6.Rerank — reorder the merged candidates by relevance.
• 7.Assemble context — place the top chunks into the prompt.
• 8.Generate — Claude answers, grounded in only what was actually retrieved.

But here's the crucial twist the exam tests just as heavily: this pipeline is not mandatory for every kind of data. There is no single right retrieval design — the architect matches STRATEGY to the shape of the data and the pattern of the queries, rather than routing everything through one favorite pipeline because it worked well once.

Structured, tabular data with precise lookups — an inventory table, an orders database — should be queried directly at the source of truth (SQL or an API), not embedded. Embedding a database and hoping semantic search reconstructs a precise row lookup is solving an exact-match problem with an approximate-match tool; querying the database directly is both simpler and more accurate. Exact identifiers, codes, and names lean toward lexical/BM25, as covered in 3.3.3. Natural-language, paraphrase-heavy questions lean toward semantic embeddings, ideally with reranking added for precision. Mixed corpora — a knowledge base with both exact SKU lookups and open-ended "how do I..." questions — call for hybrid retrieval, combining both. And a small, stable reference set — a glossary of ten terms, a short list of current promotions — may be cheaper and simpler to place directly in a cached prompt prefix than to stand up a full retrieval pipeline for it at all.

You now know why context size doesn't substitute for retrieval, how to size chunks, the four retrieval mechanisms and their strengths, how contextual retrieval repairs the context chunking destroys, the full pipeline order, and — the piece most questions actually test — how to match strategy to data shape rather than reflexively reaching for one technique.

Once retrieval is designed well, the next decision is HOW Claude physically connects to it and to every other capability — as an MCP server, a direct API call, or a delegated agent. That's Lesson 3.4.