Domain 3: Integration
19% of examEvaluate tool and agent configuration for capability bloat
Key Points
- Capability bloat -- giving an agent more or stronger tools than its role needs -- harms security, reliability, and cost/context simultaneously.
- Security: a hijacked or confused agent can invoke a destructive capability it never needed.
- Reliability: tool selection accuracy drops as the tool set grows and descriptions overlap.
- Cost/context: every tool definition consumes context tokens on every request, called or not.
- Least privilege means removing unneeded capabilities entirely, not guarding them with logging or confirmation prompts.
- Model size or instruction-following ability is unrelated to authorization scope.
Decision Rules
When: An agent's role doesn't require a tool (e.g., a support agent that only drafts replies has refund/delete-account tools)
→Remove the tool entirely -- don't add logging or a confirmation prompt.
When: Asked to fix over-privilege by 'using a smarter model'
→Reject it -- model capability is unrelated to authorization scope.
When: Auditing an agent's tool set
→Check it against the agent's actual role, not against what might someday be useful.
✗ Anti-Patterns to Reject
- Answering 'add logging' or 'add a confirmation prompt' to a least-privilege question instead of removing the capability.
- Assuming a bigger, more instruction-following model fixes over-privilege.
Analyze integrations for authentication and authorization gaps
Key Points
- Authentication proves identity (API keys, OAuth, service identities); authorization scopes what that identity may do.
- Confused deputy: an agent acting with broad service credentials on behalf of a low-privilege user can leak or mutate data the user shouldn't touch.
- Scope tool permissions to the calling user's entitlements, not the service account's broader credentials.
- Authenticate every hop: Claude to tool, and tool to downstream service.
- Secrets belong in environment variables or a secret store -- never hard-coded, committed, or placed in prompts.
Decision Rules
When: An agent acts on behalf of a low-privilege user using broad service credentials
→Scope its permissions to the user's entitlements to avoid the confused-deputy pattern.
When: A credential is needed by a tool
→Load it from an environment variable or secret store, never hard-code or embed it in a prompt.
When: Reviewing an integration before shipping
→Verify every hop (Claude to tool, tool to downstream service) authenticates its caller.
✗ Anti-Patterns to Reject
- Letting an agent use broad service credentials to act on a specific user's behalf without scoping to that user's entitlements.
- Hard-coding, committing, or embedding secrets directly in prompts.
Design RAG pipelines: chunking, indexing, and contextual retrieval
Key Points
- Chunk size must match data shape and query pattern -- too large dilutes relevance, too small loses context.
- Embedding/semantic search captures meaning and paraphrase; BM25/lexical retrieval is precise on exact terms and identifiers.
- Hybrid (embeddings + BM25) plus reranking is the stronger default for robustness across query types.
- RAG pipeline order: ingest, chunk, add context, embed + index (vector + lexical), retrieve, rerank, assemble context, generate.
- Contextual retrieval prepends a short chunk-specific context blurb before embedding/indexing (both contextual embeddings and contextual BM25), substantially reducing retrieval-failure rates.
- Prompt caching makes generating per-chunk contexts economical at scale.
Decision Rules
When: Chunks retrieved in isolation lose the surrounding document meaning
→Apply contextual retrieval -- prepend a chunk-specific context blurb before embedding/indexing.
When: A query mixes exact identifiers and natural-language phrasing
→Use hybrid retrieval (embeddings + BM25) plus reranking rather than embedding-only search.
When: Tempted to use a bigger context window instead of retrieval
→Don't -- dumping whole corpora into context causes context rot, higher cost, and worse relevance.
✗ Anti-Patterns to Reject
- Treating a bigger context window as a replacement for targeted retrieval.
- Assuming embedding-only search is always best, missing exact-term/identifier queries that BM25 would catch.
Match retrieval strategy to data shape and query pattern
Key Points
- Retrieval strategy should be matched to data shape and query pattern, not a favorite technique applied uniformly.
- Structured/tabular data with precise lookups: query the source of truth directly (SQL/API), don't embed it.
- Exact identifiers, codes, names favor lexical/BM25; natural-language, paraphrase-heavy questions favor semantic embeddings plus reranking.
- Mixed corpora call for hybrid retrieval combining embeddings and BM25.
- A small, stable reference set may be cheaper placed in a cached prompt prefix than built into a full retrieval pipeline.
Decision Rules
When: Data is structured/tabular with precise lookups
→Query the source of truth directly (SQL/API) rather than embedding it.
When: Queries target exact identifiers, codes, or names
→Favor lexical/BM25 over pure semantic search.
When: A reference set is small and stable
→Consider a cached prompt prefix instead of building a full retrieval pipeline.
✗ Anti-Patterns to Reject
- Embedding a database for precise lookups the database could just answer directly via SQL/API.
- Forcing every retrieval need through one pipeline instead of mixing direct queries, hybrid retrieval, and cached prefixes by data shape.
Select the appropriate connection protocol: MCP, API/CLI, or agent-to-agent
Key Points
- MCP is an open standard exposing tools, resources, and prompts to any MCP client -- build-once, reuse-everywhere, maintained on its own release cycle.
- Direct API/CLI fits one-off, app-specific integrations where a standard protocol adds no leverage.
- Agent-to-agent delegation fits when the remote capability is itself an autonomous agent, not a single function.
- stdio transport suits a local subprocess; Streamable HTTP/sockets suit remote, multi-client servers.
- Match the protocol to the capability's reuse and maintenance profile, not habit or familiarity.
Decision Rules
When: A capability is reused across apps/clients and maintained independently
→Expose it via MCP.
When: An integration is one-off and app-specific with no reuse expectation
→Use direct API/CLI.
When: The remote capability is itself a reasoning agent
→Use agent-to-agent delegation, handing off a goal rather than invoking an operation.
When: An MCP server runs locally for one client vs. remotely for many clients
→Use stdio transport for the former; use Streamable HTTP/sockets for the latter.
✗ Anti-Patterns to Reject
- Reaching for MCP for a single app-specific call that will never be reused.
- Hand-rolling a direct integration for a capability that many apps will share and that should be a standard MCP server.
Choose progressive discovery over monolithic context for large integrations
Key Points
- Monolithic context front-loads all tools/schemas/docs into the context window up front.
- Progressive discovery exposes a lean surface and lets the agent fetch detail on demand.
- Monolithic context bloats the window, raises cost, and invites context rot -- most loaded content is unused on any given request.
- Progressive discovery scales to large tool/resource sets; monolithic context is acceptable only when the full set is small and stable.
- This mirrors the context-curation discipline applied elsewhere in agent and RAG design.
Decision Rules
When: An integration surface has hundreds of tools or a large resource catalog
→Use progressive discovery (list, then fetch detail on demand).
When: The full tool/resource set is small and stable
→Monolithic front-loading is acceptable.
When: Most loaded context goes unused on a given request
→Treat that as a signal to switch from monolithic to progressive discovery.
✗ Anti-Patterns to Reject
- Front-loading an entire large tool/resource catalog into context 'for completeness,' bloating every request.
- Treating monolithic context as free just because the window technically has room.
Design observability and justify accuracy-latency tradeoffs at scale
Key Points
- Log request/response pairs, tool invocations and arguments, retrieval hits, stop_reason, and per-hop token usage.
- Trace multi-step runs to walk a failure back to its first deviation, not just the visible symptom.
- At scale, use sampling, structured logs, latency/error dashboards, and per-domain quality metrics.
- Confident-but-wrong answers right after a document refresh (model/latency unchanged) point to retrieval/indexing, not the model.
- Justify every knob against the stated accuracy/latency/cost requirement, not a default preference for speed or precision.
Decision Rules
When: A RAG system returns confident but wrong answers right after a document refresh, with model and latency unchanged
→Investigate the retrieval/indexing step first, not the model.
When: An SLA is strict and latency-sensitive
→Weigh accuracy levers (reranking, more chunks) against the latency budget explicitly.
When: A compliance-review integration values accuracy above all
→Accept the added latency from reranking rather than cutting it for speed.
✗ Anti-Patterns to Reject
- Optimizing latency the SLA doesn't require by dropping retrieval quality the task does need.
- Blaming 'the model got worse' when the actual cause is a retrieval or indexing problem.