Retrieval-Augmented Generation (RAG)
Context ManagementDefinition
A pattern that dynamically retrieves relevant information from an external knowledge base and injects it into the context window based on the current query. Allows Claude to reason over large document sets without fitting everything in context at once.
Example Usage
Use vector similarity search to retrieve the 5 most relevant documentation chunks for the current question and include them in the context.
Why It Matters for the CCA-F Exam
RAG is tested in Domain 5 as the primary pattern for extending Claude's effective knowledge beyond the context window. The exam expects you to know the pipeline (embed → retrieve → inject → generate), the main failure modes, why retrieval quality is the critical variable, and when RAG is preferred over full-context stuffing or fine-tuning.
In Depth
Retrieval-Augmented Generation (RAG) is an architectural pattern that decouples a model's reasoning capability from its knowledge storage. Rather than attempting to fit an entire knowledge base into the context window — or relying on facts baked into the model's weights — RAG dynamically retrieves only the most relevant chunks of information for the current query and injects them into the prompt.
The core pipeline:
A RAG system has two phases. At *indexing time*, source documents are split into chunks, each chunk is converted into a dense vector embedding, and the embeddings are stored in a vector database alongside the original text. At *query time*, the user's question is embedded using the same model, the vector store is queried for the nearest-neighbor chunks, and the top-K results are inserted into the Claude prompt as context before generation.
Why RAG is a context management strategy:
From Claude's perspective, RAG is a way to work with knowledge bases that are far larger than any context window. A 1M-token window sounds large, but a corporate knowledge base with thousands of documents is orders of magnitude larger. RAG solves this by turning the window space problem into a retrieval quality problem: instead of fitting everything in, you retrieve only what is relevant.
This places RAG squarely in Domain 5 (Context Management & Reliability). The context window remains manageable; the intelligence about *what to include* moves into the retrieval layer. See also Context Budget Management & Upstream Reduction for how RAG fits into broader context strategy.
Failure modes to know for the exam:
- Retrieval miss: The relevant chunk is not returned because the query embedding is too dissimilar from the chunk's embedding (vocabulary mismatch, semantic gap). Mitigation: hybrid search (vector + keyword BM25).
- Chunk boundary problem: The answer to a question spans two chunks; neither chunk alone is sufficient. Mitigation: overlapping chunks or parent-document retrieval.
- Stale index: The knowledge base is updated but the vector index is not re-built. Retrieved content is outdated.
- Context injection position: Retrieved chunks injected in the middle of a long message history suffer from position effects — place them close to the final user turn.
- Over-retrieval: Too many chunks dilute the relevant signal and fill the context with noise.
RAG versus fine-tuning:
Fine-tuning bakes knowledge into model weights — it is not retrievable or auditable after the fact. RAG keeps knowledge external and auditable; source attribution is straightforward because you know exactly which chunks were retrieved. For dynamic, frequently updated, or domain-specific knowledge, RAG is generally preferred over fine-tuning on the CCA-F exam.
RAG versus full-context stuffing:
For small, stable knowledge bases (dozens of documents), stuffing everything into a large context window may be simpler than building a retrieval pipeline. RAG adds latency and infrastructure complexity. The decision threshold depends on knowledge base size, update frequency, and latency requirements.
How It Compares
| Approach | Knowledge storage | Context usage | Update cost | Attribution |
|---|---|---|---|---|
| RAG | External vector store | Only retrieved chunks | Re-index changed docs | Easy — track retrieved chunks |
| Full-context stuffing | In the prompt each call | High — entire knowledge base | None needed | N/A — everything is in context |
| Fine-tuning | Model weights | Minimal — knowledge is implicit | Expensive — re-train or re-tune | Hard — weights are opaque |
| Rolling window + summarization | Conversation history only | Controlled | Per-session | Limited to conversation history |
Example
Minimal RAG skeleton: retrieve chunks, wrap in XML source tags, inject near the final user turn. Replace retrieve_chunks with your actual vector store client.
import anthropic
client = anthropic.Anthropic()
def retrieve_chunks(query: str, top_k: int = 5) -> list[str]:
# Placeholder: replace with actual vector store query
# e.g. Pinecone, pgvector, Weaviate, etc.
return [f"Relevant chunk {i} for: {query}" for i in range(top_k)]
def rag_query(user_question: str) -> str:
chunks = retrieve_chunks(user_question, top_k=5)
context_block = "\n\n".join(
f"<source id=\"{i+1}\">\n{chunk}\n</source>"
for i, chunk in enumerate(chunks)
)
messages = [
{
"role": "user",
"content": (
f"<retrieved_context>\n{context_block}\n</retrieved_context>\n\n"
f"Question: {user_question}"
),
}
]
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1024,
system="Answer using only the retrieved context. Cite source IDs.",
messages=messages,
)
return response.content[0].textFrequently Asked Questions
Where in the context should retrieved chunks be placed?
As close to the final user turn as possible. Due to position effects, content in the middle of a long message history is recalled less reliably. Injecting retrieved chunks immediately before the question — as part of the final user message — gives them the best chance of influencing the response.
What is the main failure mode RAG introduces compared to full-context stuffing?
Retrieval miss: the correct chunk simply is not returned. With full-context stuffing, if the answer is in the documents it is in the context. With RAG, the retrieval step can fail due to embedding distance, query phrasing, or chunk boundaries. This is why retrieval quality (not generation quality) is typically the first thing to debug in a RAG system.