Domain 1: Solution Design & Architecture
17% of examTranslate a business problem into a Claude-based solution design
Key Points
- Design flow: elicit the business outcome, derive functional/non-functional requirements, then choose the architecture that meets them at acceptable cost.
- Extract six dimensions before picking a pattern: latency, volume/scale, criticality, data sensitivity, cost ceiling, and quality bar.
- These constraints -- not preference -- decide realtime vs. batch, model tier, integration protocol, and human-in-the-loop placement.
- Every architectural decision should be traceable to a stated constraint, not a default preference.
- Production designs add evaluation, observability, security, and lifecycle as first-class concerns from day one -- a POC is not production.
Decision Rules
When: A stakeholder asks for 'an agent' or 'the biggest model' before constraints are established
→Anchor on the six dimensions first; do not commit to autonomy or model tier until requirements justify it.
When: The task describes fixed, known steps with a predictability requirement
→Recommend a workflow, not an agent, regardless of how capable the model is.
When: You are asked to 'design a solution' rather than 'build a demo'
→Explicitly include evaluation, observability, security, and lifecycle as parts of the answer, not afterthoughts.
✗ Anti-Patterns to Reject
- Jumping to 'use an agent' or 'use the biggest model' before the requirements justify it.
- Treating a proof-of-concept design as production-ready, omitting evaluation, observability, security, and lifecycle.
Select among the augmented LLM, workflow, and agentic architectural patterns
Key Points
- Three patterns on a complexity gradient: augmented LLM (base unit), workflow (predefined code paths), agent (LLM-directed loop).
- The augmented LLM -- model + retrieval + tools + memory -- is the atomic unit that workflows and agents compose.
- Workflows fit known, fixed steps where predictability and testability matter; agents fit open-ended tasks whose steps can't be enumerated in advance.
- Agents run plan -> call tool -> observe -> repeat, trading latency/cost/predictability for handling open-ended tasks.
- Five workflow composition patterns: prompt chaining, routing, parallelization (sectioning/voting), orchestrator-workers, evaluator-optimizer.
- Exam trap: the line between orchestrator-workers and parallelization sectioning is who decides the subtasks, and when.
Decision Rules
When: The task is the same fixed sequence every time
→Use prompt chaining.
When: Input falls into distinguishable categories needing different handling
→Use routing.
When: Independent subtasks are known in advance vs. must be decided by a central LLM at runtime
→Use parallelization (sectioning) for the former; use orchestrator-workers for the latter.
When: Output quality benefits from a critique-and-refine cycle
→Use evaluator-optimizer.
✗ Anti-Patterns to Reject
- Confusing orchestrator-workers (subtasks decided dynamically at runtime) with parallelization sectioning (subtasks known in advance).
- Escalating to a workflow or agent before confirming a single augmented LLM call can't satisfy the requirement.
Design end-to-end architecture with input, processing, output, and feedback loops
Key Points
- End-to-end architecture shape: input -> processing -> output -> feedback loops.
- Input stage: ingestion, validation, separating trusted instructions from untrusted data, retrieval of grounding context.
- Processing stage: the chosen pattern (augmented LLM/workflow/agent), model selection, prompt/context assembly.
- Output stage: structured-output contracts, validation, defensive parsing, delivery to downstream systems.
- Feedback loops (evaluation, monitoring, observability) are mandatory, not optional, because LLM output is non-deterministic.
Decision Rules
When: Asked to 'design the architecture' for a system
→Walk through all four stages explicitly -- an answer that only addresses processing is incomplete.
When: A new model version ships
→Rely on the feedback loop (evals/monitoring) to detect regressions rather than assuming behavior is unchanged.
When: Untrusted data enters at the input stage
→Separate it from trusted instructions before it reaches the model.
✗ Anti-Patterns to Reject
- Stopping the design at 'output' with no plan for evaluation or monitoring.
- Treating feedback loops as optional because 'the demo worked.'
Design multi-agent systems and orchestration for genuinely separable work
Key Points
- A manager/orchestrator coordinates specialized subagents, each with its own context window, returning only a condensed result.
- Context isolation is the defining benefit -- a subagent can read a lot and return only a condensed result.
- Specialization lets each subagent get a focused prompt, tool set, and model tier (Haiku for cheap steps, Sonnet/Opus for hard ones).
- Multi-agent hierarchies multiply token usage and add real coordination overhead -- reserve them for genuinely separable, expensive-enough work.
- A single well-scoped augmented LLM often wins on cost and latency over a multi-agent design.
Decision Rules
When: A subtask requires reading far more material than the final answer needs
→Delegate it to a subagent for context isolation.
When: Subtasks are independent
→Run subagents in parallel for the parallelism benefit.
When: Considering a multi-agent design
→First check whether a single augmented LLM or simpler workflow would satisfy the requirement at lower cost and latency.
✗ Anti-Patterns to Reject
- Treating subagents as 'just more prompts' instead of valuing their separate context window.
- Assuming a multi-agent design is automatically better or higher quality than a single augmented LLM.
Apply decomposition techniques to make complex problems tractable
Key Points
- Four decomposition techniques: sequential, parallel, routing, recursive/hierarchical.
- Sequential decomposition (prompt chaining) fits ordered sub-steps that depend on the last.
- Parallel decomposition fits independent subtasks that run concurrently and merge.
- Routing decomposition classifies first, then sends each class down a specialized path.
- Decomposition improves reliability: smaller steps are easier to prompt, evaluate, and debug, and boundaries are natural gate-check points.
Decision Rules
When: Each step needs the previous step's output
→Use sequential decomposition.
When: Subtasks are independent and can run at once
→Use parallel decomposition.
When: Inputs fall into distinguishable categories
→Use routing decomposition.
When: The task needs open-ended delegation that may itself need further breakdown
→Use recursive/hierarchical decomposition.
✗ Anti-Patterns to Reject
- Using one sprawling mega-prompt instead of decomposing into well-scoped steps with clear input/output contracts.
- Skipping gate checks or validation at decomposition boundaries.
Align architectural decisions to business value pillars
Key Points
- Five business value pillars: efficiency, transformation, productivity, cost, and performance/SLAs.
- Every architectural tradeoff should be traceable to one of the five pillars.
- 'Smaller model + prompt caching' serves cost/performance; 'human-in-the-loop review' serves risk/quality.
- Optimizing an unrequested metric (e.g., latency) at the expense of a requested one (e.g., cost) is wrong even if it looks like an improvement.
Decision Rules
When: A stakeholder states cost as the goal for a high-volume, latency-tolerant job
→Route to a smaller model tier and/or batch it -- not the largest model 'for reliability.'
When: You cannot name the pillar a proposed change serves
→Treat the change as not yet justified.
When: Choosing a tradeoff lever
→Pick the one that most directly serves the stated pillar (model tier/batching for cost, caching/async for performance, human review for quality/risk).
✗ Anti-Patterns to Reject
- Optimizing a metric the business didn't ask for (e.g., squeezing latency) at the expense of one it did (cost, quality).
- Swapping in the largest, most capable model 'for reliability' when the stated goal is cost reduction.
Domain 2: Claude Models, Prompting & Context Engineering
13% of examSelect Claude models and apply model-choice tradeoffs
Key Points
- Three tiers: Haiku (fastest/cheapest, high-volume well-defined steps), Sonnet (balanced workhorse), Opus (most capable, hardest reasoning/orchestration).
- Resolve quality vs. latency vs. cost against the specific requirement, not in the abstract -- there is no universal 'best' model.
- Mixed-model pipelines route each step to the tier that fits its difficulty and budget.
- Extended/adaptive thinking trades billed output tokens and latency for better performance on hard reasoning tasks -- reserve it, don't default to it.
- Pin model versions in production and re-run evals before adopting a new release; a prompt tuned for one version can regress on the next.
Decision Rules
When: A step is high-volume, well-defined, and latency-sensitive (classification, routing, extraction)
→Route it to Haiku.
When: A step requires genuinely hard reasoning, planning, or orchestration
→Use Opus or extended thinking.
When: A new model release becomes available
→Pin the current version in production and re-run evals before adopting the new one.
✗ Anti-Patterns to Reject
- Defaulting to the most capable model everywhere instead of fitting the tier to the constraint.
- Upgrading to a new model version without re-running evals, assuming newer always means strictly better for a given prompt.
Design system prompts, templates, and guardrails
Key Points
- System prompt holds stable rules/role/tone/constraints; user message holds the specific request and per-request data.
- This separation keeps behavior consistent and keeps the stable prefix cacheable.
- Templates parameterize variable parts while holding the scaffold constant, improving consistency and cache hit rates.
- Prompt-level guardrails (refusals, content policy, escalation) are probabilistic -- the model usually follows them but compliance isn't guaranteed.
- Deterministic controls (hooks, permission scoping, output validation) sit around the model and enforce boundaries structurally; use them for destructive or high-stakes actions.
- Untrusted input must be delimited/sanitized so it isn't read as an instruction.
Decision Rules
When: Content is stable across every request (rules, role, tone)
→Place it in the system prompt, not the user message.
When: An action is destructive or high-stakes
→Pair the prompt guardrail with a deterministic control (hook, permission scoping, output validation) -- don't rely on the prompt sentence alone.
When: A prompt incorporates untrusted input (user text, retrieved docs, tool output)
→Delimit and sanitize it so it can't be read as an instruction.
✗ Anti-Patterns to Reject
- Mixing stable and per-request content in one blob, hurting consistency and breaking the cacheable prefix.
- Relying on a single system-prompt sentence as the only guardrail for a destructive or high-stakes action.
Apply prompt engineering techniques matched to the task
Key Points
- Zero-shot suits simple, unambiguous tasks; few-shot locks in format/edge-case behavior; chain-of-thought suits multi-step reasoning.
- Few-shot examples show the desired behavior rather than describing it in prose.
- Chain-of-thought and extended thinking are the same lever: reasoning depth traded for tokens/latency.
- Specific, clear instructions beat clever or terse ones for consistent output.
- Positive instructions ('respond only in JSON matching this schema') beat long negative 'don't' lists.
- With long inputs, place the key instruction after the material, near the end; iterate one change at a time and measure against an eval set.
Decision Rules
When: The task is simple and well-specified
→Use zero-shot -- examples or chain-of-thought would add cost without benefit.
When: Prose instructions alone can't pin down a specific format or edge-case behavior
→Add few-shot examples.
When: The task requires multi-step reasoning, math, or complex judgment
→Use chain-of-thought or extended thinking.
When: The input is long
→Place the key instruction near the end, after the material.
✗ Anti-Patterns to Reject
- Adding few-shot examples or chain-of-thought reflexively, as if they were always an improvement.
- Iterating on multiple prompt changes at once, making it impossible to attribute a quality change to a specific edit.
Optimize the context window and manage token budgets
Key Points
- The context window is one shared budget across system prompt, history, tools, tool results, retrieved docs, and the response.
- Context rot/drift is quality degradation caused by low-signal tokens crowding the window -- not by running out of room.
- Curate to the smallest set of high-signal tokens rather than relying on raw context capacity; a bigger window does not eliminate the need for curation.
- Pruning drops stale tool output; compaction summarizes older turns; isolation moves heavy subtasks into a subagent's own window; progressive disclosure loads information only as needed.
- Track the usage field (input/output/cache tokens) to model cost and detect bloat before hitting the budget.
Decision Rules
When: Tool output is large or stale and no longer informs the task
→Prune it.
When: Conversation history is long
→Compact older turns into a summary recap.
When: A subtask requires reading far more material than the answer needs
→Isolate it into a subagent's own context window.
When: The usage field shows climbing input tokens with no quality gain
→Investigate curation before assuming a bigger context window is the fix.
✗ Anti-Patterns to Reject
- Assuming a bigger context window removes the need to curate -- it only postpones the same degradation.
- Treating pruning, compaction, isolation, and progressive disclosure as interchangeable rather than complementary.
Design for prompt reuse through caching, modular prompts, and Skills
Key Points
- A stable prefix (system prompt + policy + few-shot block) can be cached: reads are cheap, writes carry a slight premium.
- Order stable content first, dynamic content last to maximize the cacheable prefix length -- this cuts both time-to-first-token and per-request cost.
- Modular prompts compose reusable, independently versioned fragments (role, policy, format spec) instead of duplicating text.
- Agent Skills package reusable instructions/procedures, authored once and reused across apps/teams, without a running service.
- Caching, modular prompts, and Skills are three complementary levers for making prompt designs affordable and maintainable at scale.
Decision Rules
When: A large stable prefix repeats across many requests
→Order it first and cache it rather than reprocessing it every call.
When: A needed policy document is large and there's a temptation to truncate it to save tokens
→Cache it instead -- truncation trades away correctness that caching would have preserved for free.
When: A capability (procedure/know-how) needs to be reused across apps/teams without standing up a service
→Package it as an Agent Skill.
✗ Anti-Patterns to Reject
- Putting dynamic content before the stable prefix, breaking the cacheable prefix and defeating caching.
- Confusing prompt caching (reusing a stable prefix within a call) with retrieval (fetching external documents).
Domain 3: Integration
19% of examEvaluate tool and agent configuration for capability bloat
Key Points
- Capability bloat -- giving an agent more or stronger tools than its role needs -- harms security, reliability, and cost/context simultaneously.
- Security: a hijacked or confused agent can invoke a destructive capability it never needed.
- Reliability: tool selection accuracy drops as the tool set grows and descriptions overlap.
- Cost/context: every tool definition consumes context tokens on every request, called or not.
- Least privilege means removing unneeded capabilities entirely, not guarding them with logging or confirmation prompts.
- Model size or instruction-following ability is unrelated to authorization scope.
Decision Rules
When: An agent's role doesn't require a tool (e.g., a support agent that only drafts replies has refund/delete-account tools)
→Remove the tool entirely -- don't add logging or a confirmation prompt.
When: Asked to fix over-privilege by 'using a smarter model'
→Reject it -- model capability is unrelated to authorization scope.
When: Auditing an agent's tool set
→Check it against the agent's actual role, not against what might someday be useful.
✗ Anti-Patterns to Reject
- Answering 'add logging' or 'add a confirmation prompt' to a least-privilege question instead of removing the capability.
- Assuming a bigger, more instruction-following model fixes over-privilege.
Analyze integrations for authentication and authorization gaps
Key Points
- Authentication proves identity (API keys, OAuth, service identities); authorization scopes what that identity may do.
- Confused deputy: an agent acting with broad service credentials on behalf of a low-privilege user can leak or mutate data the user shouldn't touch.
- Scope tool permissions to the calling user's entitlements, not the service account's broader credentials.
- Authenticate every hop: Claude to tool, and tool to downstream service.
- Secrets belong in environment variables or a secret store -- never hard-coded, committed, or placed in prompts.
Decision Rules
When: An agent acts on behalf of a low-privilege user using broad service credentials
→Scope its permissions to the user's entitlements to avoid the confused-deputy pattern.
When: A credential is needed by a tool
→Load it from an environment variable or secret store, never hard-code or embed it in a prompt.
When: Reviewing an integration before shipping
→Verify every hop (Claude to tool, tool to downstream service) authenticates its caller.
✗ Anti-Patterns to Reject
- Letting an agent use broad service credentials to act on a specific user's behalf without scoping to that user's entitlements.
- Hard-coding, committing, or embedding secrets directly in prompts.
Design RAG pipelines: chunking, indexing, and contextual retrieval
Key Points
- Chunk size must match data shape and query pattern -- too large dilutes relevance, too small loses context.
- Embedding/semantic search captures meaning and paraphrase; BM25/lexical retrieval is precise on exact terms and identifiers.
- Hybrid (embeddings + BM25) plus reranking is the stronger default for robustness across query types.
- RAG pipeline order: ingest, chunk, add context, embed + index (vector + lexical), retrieve, rerank, assemble context, generate.
- Contextual retrieval prepends a short chunk-specific context blurb before embedding/indexing (both contextual embeddings and contextual BM25), substantially reducing retrieval-failure rates.
- Prompt caching makes generating per-chunk contexts economical at scale.
Decision Rules
When: Chunks retrieved in isolation lose the surrounding document meaning
→Apply contextual retrieval -- prepend a chunk-specific context blurb before embedding/indexing.
When: A query mixes exact identifiers and natural-language phrasing
→Use hybrid retrieval (embeddings + BM25) plus reranking rather than embedding-only search.
When: Tempted to use a bigger context window instead of retrieval
→Don't -- dumping whole corpora into context causes context rot, higher cost, and worse relevance.
✗ Anti-Patterns to Reject
- Treating a bigger context window as a replacement for targeted retrieval.
- Assuming embedding-only search is always best, missing exact-term/identifier queries that BM25 would catch.
Match retrieval strategy to data shape and query pattern
Key Points
- Retrieval strategy should be matched to data shape and query pattern, not a favorite technique applied uniformly.
- Structured/tabular data with precise lookups: query the source of truth directly (SQL/API), don't embed it.
- Exact identifiers, codes, names favor lexical/BM25; natural-language, paraphrase-heavy questions favor semantic embeddings plus reranking.
- Mixed corpora call for hybrid retrieval combining embeddings and BM25.
- A small, stable reference set may be cheaper placed in a cached prompt prefix than built into a full retrieval pipeline.
Decision Rules
When: Data is structured/tabular with precise lookups
→Query the source of truth directly (SQL/API) rather than embedding it.
When: Queries target exact identifiers, codes, or names
→Favor lexical/BM25 over pure semantic search.
When: A reference set is small and stable
→Consider a cached prompt prefix instead of building a full retrieval pipeline.
✗ Anti-Patterns to Reject
- Embedding a database for precise lookups the database could just answer directly via SQL/API.
- Forcing every retrieval need through one pipeline instead of mixing direct queries, hybrid retrieval, and cached prefixes by data shape.
Select the appropriate connection protocol: MCP, API/CLI, or agent-to-agent
Key Points
- MCP is an open standard exposing tools, resources, and prompts to any MCP client -- build-once, reuse-everywhere, maintained on its own release cycle.
- Direct API/CLI fits one-off, app-specific integrations where a standard protocol adds no leverage.
- Agent-to-agent delegation fits when the remote capability is itself an autonomous agent, not a single function.
- stdio transport suits a local subprocess; Streamable HTTP/sockets suit remote, multi-client servers.
- Match the protocol to the capability's reuse and maintenance profile, not habit or familiarity.
Decision Rules
When: A capability is reused across apps/clients and maintained independently
→Expose it via MCP.
When: An integration is one-off and app-specific with no reuse expectation
→Use direct API/CLI.
When: The remote capability is itself a reasoning agent
→Use agent-to-agent delegation, handing off a goal rather than invoking an operation.
When: An MCP server runs locally for one client vs. remotely for many clients
→Use stdio transport for the former; use Streamable HTTP/sockets for the latter.
✗ Anti-Patterns to Reject
- Reaching for MCP for a single app-specific call that will never be reused.
- Hand-rolling a direct integration for a capability that many apps will share and that should be a standard MCP server.
Choose progressive discovery over monolithic context for large integrations
Key Points
- Monolithic context front-loads all tools/schemas/docs into the context window up front.
- Progressive discovery exposes a lean surface and lets the agent fetch detail on demand.
- Monolithic context bloats the window, raises cost, and invites context rot -- most loaded content is unused on any given request.
- Progressive discovery scales to large tool/resource sets; monolithic context is acceptable only when the full set is small and stable.
- This mirrors the context-curation discipline applied elsewhere in agent and RAG design.
Decision Rules
When: An integration surface has hundreds of tools or a large resource catalog
→Use progressive discovery (list, then fetch detail on demand).
When: The full tool/resource set is small and stable
→Monolithic front-loading is acceptable.
When: Most loaded context goes unused on a given request
→Treat that as a signal to switch from monolithic to progressive discovery.
✗ Anti-Patterns to Reject
- Front-loading an entire large tool/resource catalog into context 'for completeness,' bloating every request.
- Treating monolithic context as free just because the window technically has room.
Design observability and justify accuracy-latency tradeoffs at scale
Key Points
- Log request/response pairs, tool invocations and arguments, retrieval hits, stop_reason, and per-hop token usage.
- Trace multi-step runs to walk a failure back to its first deviation, not just the visible symptom.
- At scale, use sampling, structured logs, latency/error dashboards, and per-domain quality metrics.
- Confident-but-wrong answers right after a document refresh (model/latency unchanged) point to retrieval/indexing, not the model.
- Justify every knob against the stated accuracy/latency/cost requirement, not a default preference for speed or precision.
Decision Rules
When: A RAG system returns confident but wrong answers right after a document refresh, with model and latency unchanged
→Investigate the retrieval/indexing step first, not the model.
When: An SLA is strict and latency-sensitive
→Weigh accuracy levers (reranking, more chunks) against the latency budget explicitly.
When: A compliance-review integration values accuracy above all
→Accept the added latency from reranking rather than cutting it for speed.
✗ Anti-Patterns to Reject
- Optimizing latency the SLA doesn't require by dropping retrieval quality the task does need.
- Blaming 'the model got worse' when the actual cause is a retrieval or indexing problem.
Domain 4: Evaluation, Testing & Optimization
16% of examDefine evaluation metrics across accuracy, latency, cost, safety, and security
Key Points
- A production eval measures five dimensions: accuracy/quality, latency, cost, safety, and security.
- Good success criteria are specific, measurable, and tied to the use case ('95% of extracted fields match the reference on the held-out set'), not vague aspirations ('the model should be accurate').
- Eval metrics must trace back to the business value pillars defined during solution design.
- An accurate-but-slow, -expensive, or -unsafe design still fails its requirement.
- Define success criteria before building the eval, not after.
Decision Rules
When: Writing success criteria for an eval
→Make them specific and measurable, tied to the use case, not a vague aspiration.
When: Aggregate accuracy looks fine but the system fails elsewhere
→Check latency, cost, safety, and security -- the eval was likely incomplete, not accuracy measurement broken.
When: Choosing which metrics to track
→Trace them back to the business value pillars the solution was designed to serve.
✗ Anti-Patterns to Reject
- Measuring only accuracy and ignoring latency, cost, safety, and security.
- Writing aspirational, unmeasurable success criteria like 'the model should be accurate.'
Build evaluation datasets and test frameworks using mixed scoring methodologies
Key Points
- An eval = a representative test set + a scoring method + a metric, run repeatably.
- A usable dataset is representative (real usage, edge cases, known failure modes), large enough to avoid noise, and held out (not tuned on).
- Code/exact-match grading is fast and unambiguous but only fits structured/verifiable output.
- LLM-as-judge scales subjective grading but must itself be validated against human labels before you rely on it.
- Human evaluation is the gold standard, reserved for nuanced or high-stakes cases.
- Automate the eval to run on every prompt change and every model-version bump -- a single passing example is not an evaluation.
Decision Rules
When: Output is structured/verifiable
→Use code/exact-match grading for a fast, cheap, unambiguous check.
When: Output is open-ended or subjective and needs to scale
→Use LLM-as-judge, but validate the judge against human labels first.
When: A judgment is nuanced or high-stakes
→Reserve human evaluation for it.
When: A prompt changes or a model version bumps
→Re-run the automated eval, not a one-time spot-check.
✗ Anti-Patterns to Reject
- Treating one manual spot-check ('ran it once and read the answer') as a valid evaluation.
- Trusting an LLM-as-judge that hasn't been validated against human labels.
Run A/B tests and apply iterative improvement to drive design decisions
Key Points
- Improvement is empirical: change one variable, measure it against the eval, keep what wins.
- A/B test competing prompts, models, retrieval configs, or parameters on the same dataset against the same defined metrics.
- Ablation mindset: change one thing at a time so you can attribute the effect to that specific variable.
- Contextual retrieval is the canonical example: A/B embeddings-only vs. +BM25 vs. +reranking on retrieval-failure rate.
- Iteration is a loop: hypothesize, change, measure, adopt or revert.
Decision Rules
When: Testing a change to prompts, models, retrieval config, or parameters
→A/B it on the same dataset against defined metrics, changing exactly one variable.
When: Comparing retrieval configurations
→Measure retrieval-failure rate across embeddings-only vs. +BM25 vs. +reranking.
When: Deciding whether to ship a change
→Run the hypothesize-change-measure-adopt/revert loop rather than shipping on intuition.
✗ Anti-Patterns to Reject
- Switching multiple variables at once and eyeballing a few answers, which breaks attribution.
- Deciding on intuition rather than a measured A/B test against defined metrics.
Diagnose production system issues by localizing the failure layer
Key Points
- Localize the cause before fixing -- patching the wrong layer just moves the symptom.
- Confident-but-wrong after a document refresh (model/latency unchanged) points to retrieval/indexing.
- Well-formed but factually invented output is a hallucination, fixed with grounding/constraints/citations.
- A regression right after a model-version change is model mismatch -- re-run evals, pin/roll back.
- Truncated output with stop_reason: max_tokens is a token-limit issue, not a prompt or model problem.
- Trace analysis walks logs back to the earliest deviation, not just the final symptom.
Decision Rules
When: Answers are confident but wrong right after a document refresh, with model and latency unchanged
→Suspect retrieval/indexing, not the model.
When: Output is well-formed but factually invented
→Diagnose hallucination and fix with grounding/citations/constraints, not a version rollback.
When: Quality regresses right after a model-version bump
→Re-run evals and pin/roll back the version.
When: Output is truncated
→Check stop_reason: max_tokens and raise the limit -- don't assume a prompt or model defect.
✗ Anti-Patterns to Reject
- Blaming 'the model got worse' for what is actually a retrieval or context problem.
- Fixing only the final visible symptom instead of tracing back to the earliest deviation in a multi-step pipeline.
Optimize cost, latency, and token usage while verifying against the eval
Key Points
- Optimization is a tradeoff against measured quality -- never applied blind.
- Prompt caching cuts cost/latency on stable repeated prefixes with no accuracy loss -- order stable content first.
- Right-size the model: route each step to the cheapest tier that still passes the eval.
- Trimming context reduces both cost and context rot at the same time.
- The Batches API cuts cost for latency-tolerant bulk jobs but does not reduce per-request latency.
- Re-run the eval after every optimization before adopting it -- a cheaper model or trimmed context that drops accuracy below the bar is a regression, not a win.
Decision Rules
When: A workload is latency-tolerant and bulk
→Use the Batches API to cut cost -- but don't expect it to speed up a single latency-sensitive request.
When: Considering a cheaper model, shorter context, or lower max_tokens
→Apply the change, re-run the eval, and adopt only if the metric bar still holds.
When: A stable prefix repeats across requests
→Cache it -- this is the one lever that costs zero accuracy.
✗ Anti-Patterns to Reject
- Assuming 'cheaper is always better' without re-running the eval after the change.
- Truncating content, dropping retrieval, or under-sizing max_tokens as false economies that trade away quality.
Monitor production systems with logging, dashboards, and alerting
Key Points
- Evaluation doesn't stop at launch -- monitoring turns the offline eval into a continuous quality signal.
- Log request/response pairs, tool calls, retrieval results, stop_reason, and per-hop token usage for every production request.
- Dashboard latency, error rates, cost, and per-domain quality metrics.
- Alert on regressions: retrieval-failure spikes, SLA breaches, cost anomalies, eval-score drops after a change.
- Sample and re-score live traffic periodically to catch drift the offline eval didn't cover.
Decision Rules
When: A system goes to production
→Log request/response, tool calls, retrieval results, stop_reason, and token usage for every request.
When: Setting up alerts
→Alert on regressions (spikes, SLA breaches, cost anomalies, eval-score drops), not just absolute thresholds.
When: Real-world input distributions may shift over time
→Sample and re-score live traffic periodically rather than trusting the static offline eval alone.
✗ Anti-Patterns to Reject
- Treating monitoring as a terminal step instead of feeding it back into the next design/eval iteration.
- Relying solely on the offline eval and never re-scoring live traffic for drift.
Domain 5: Governance, Safety & Risk Management
14% of examDesign layered guardrails and safety controls for production Claude systems
Key Points
- Guardrails are layered: input, permissions, deterministic controls, output, and monitoring -- no single layer is assumed perfect.
- A PreToolUse hook is a deterministic control that blocks a destructive action every time, regardless of what the model 'decides.'
- A prompt guardrail is probabilistic -- a system-prompt sentence the model usually follows, with a non-zero failure rate.
- Hard policy limits and destructive-action prevention belong in deterministic controls, not a system-prompt sentence.
- Logging a violation after the fact is detective, not preventive -- it doesn't replace a gate.
Decision Rules
When: A risk is a hard limit (financial threshold, irreversible action, compliance-mandated behavior)
→Enforce it with a deterministic hook/permission rule, not a prompt sentence.
When: A style or tone preference is soft (occasional deviation tolerable)
→A prompt guardrail alone is acceptable.
When: Designing a safety architecture
→Layer input filtering, least privilege, deterministic controls, output validation, and monitoring rather than relying on one layer.
✗ Anti-Patterns to Reject
- Putting a hard safety rule only in the system prompt.
- Relying on a single guardrail layer instead of layered, independent defenses.
Mitigate prompt injection, jailbreaks, and untrusted input
Key Points
- Prompt injection = malicious instructions hidden in untrusted content (web pages, documents, tool results) that the model reads as commands.
- Unlike SQL injection, there is no strict syntactic boundary between instructions and data in natural language -- the fix is architectural, not linguistic.
- Core defenses: isolate/delimit untrusted content, apply least privilege, filter/monitor for injection patterns, gate high-stakes actions with human review.
- Least privilege contains the blast radius even if isolation fails -- an agent that can't delete data can't be tricked into deleting it.
- Model size and sampling temperature do not defeat prompt injection; neither does a polite 'ignore injected instructions' caveat.
Decision Rules
When: An agent consumes untrusted content (web pages, documents, tool results)
→Delimit/label it clearly so it isn't executed as an instruction.
When: An injection might trigger a high-stakes action
→Require human-in-the-loop approval before execution.
When: Asked whether a bigger model or lower temperature reduces injection risk
→Reject it -- neither isolates untrusted content nor limits what a hijacked instruction can do.
✗ Anti-Patterns to Reject
- Believing a more capable model or temperature=0 defeats prompt injection.
- Adding an in-prompt caveat ('please don't obey injected instructions') as if it were a real defense.
Identify the risks, limitations, and failure modes of LLM systems
Key Points
- Six named failure modes: hallucination, non-determinism, injection/jailbreak, data leakage, model drift, automation bias.
- Each failure mode has a distinct, non-interchangeable mitigation -- grounding fixes hallucination but not leakage; pinning versions fixes drift but not automation bias.
- Confidence is not correctness -- fluent output still requires grounding and verification.
- Hallucination mitigations: retrieval grounding, citations, constrained claims, explicit 'I don't know' allowance.
- Automation bias (humans over-trusting confident AI output) is countered with transparency and mandatory human validation.
Decision Rules
When: A high-stakes claim is stated fluently and confidently
→Still require grounding and verification -- confidence is not correctness.
When: Mitigating hallucination in a high-stakes answer
→Ground with retrieval, cite sources, and allow 'I don't know' -- not raise temperature or trust stated confidence.
When: Humans start rubber-stamping confident AI output
→Add transparency and mandatory human validation to counter automation bias.
✗ Anti-Patterns to Reject
- Treating a fluent, confidently-worded answer as evidence of correctness.
- Applying one failure mode's mitigation (e.g., grounding) and assuming it also covers a different failure mode (e.g., data leakage).
Design human-in-the-loop validation for high-stakes agent actions
Key Points
- Decide HITL placement by stakes and reversibility, not a general instinct to add oversight.
- High-stakes, irreversible actions (financial transactions, account deletion, medical/legal outputs) require pre-execution human approval.
- Lower-stakes or reversible actions can run autonomously but benefit from detective review (sampling/auditing).
- HITL must be paired with deterministic gating so it cannot be bypassed by the model.
- Logging after the fact is detective only -- it is not a substitute for a preventive approval gate.
Decision Rules
When: An action is high-value and hard to reverse (e.g., an automatic refund)
→Require human-in-the-loop approval before execution, gated deterministically.
When: An action is lower-stakes or reversible
→Allow autonomous execution with periodic detective sampling instead of full gating.
When: Confidence is low or the request is out of policy
→Escalate to a human rather than guessing or proceeding.
✗ Anti-Patterns to Reject
- Trusting the model's judgment to skip human review 'to keep things fast' on a high-stakes, hard-to-reverse action.
- Logging high-stakes actions and moving on instead of gating them with pre-execution approval.
Design for regulatory compliance across GDPR, HIPAA, and FedRAMP
Key Points
- GDPR: data minimization, lawful basis, data-subject rights, residency.
- HIPAA: PHI safeguards, HIPAA-eligible services, restrict/redact PHI.
- FedRAMP: authorized environments and controls for government workloads.
- Compliance is a whole-system property -- data handling, access, retention, contracts -- not the model alone.
- Anthropic's enterprise features (HIPAA-eligible options, audit logs, retention controls, SSO/SCIM) support but do not automatically satisfy compliance.
Decision Rules
When: A system processes protected health information
→Use HIPAA-eligible services/agreements and restrict/redact PHI -- 'the model is HIPAA-compliant' is not a valid framing.
When: Handling EU personal data
→Apply data minimization, honor data-subject rights, and respect residency requirements.
When: Serving a government workload
→Use authorized environments and controls appropriate to FedRAMP.
✗ Anti-Patterns to Reject
- Assuming 'the model is compliant' implies the whole system is compliant.
- Treating encryption of the prompt alone as sufficient without addressing access, retention, and contractual controls.
Address ethical AI concerns: bias, fairness, transparency, and accountability
Key Points
- Bias/fairness cases belong in the eval set alongside accuracy and safety tests -- evaluate for disparate treatment across groups.
- Transparency: disclose AI interaction, cite sources for grounded claims, document decision logic.
- AI Fluency accountability model: delegation with description, discernment, and diligence -- verify rather than assume.
- Agentic and MCP use carry additional AUP responsibilities beyond the baseline Usage Policy.
- Ethics is a testable and monitorable engineering property, not a checklist item handled outside the technical design.
Decision Rules
When: Building an eval set
→Include fairness/bias test cases alongside accuracy and safety cases.
When: A human delegates a task to an agent
→Apply discernment and diligence in checking the result before relying on it -- accountability doesn't disappear because it was delegated.
When: A design uses agentic or MCP capabilities
→Verify it meets the additional AUP responsibilities beyond the baseline Usage Policy.
✗ Anti-Patterns to Reject
- Treating bias, fairness, and transparency as a soft, post-hoc checklist separate from engineering.
- Assuming delegating a task to an AI removes human accountability for the outcome.
Domain 6: Stakeholder Communication & Lifecycle Management
14% of examConduct structured discovery and requirement gathering with stakeholders
Key Points
- Elicit the business outcome before constraints -- constraints without an agreed outcome have no anchor.
- Use structured questions to surface non-functional requirements stakeholders won't volunteer unprompted (regulatory context, data residency, peak load, failure tolerance).
- Separate wants (preferred implementation) from needs (the must-have outcome) to preserve tradeoff flexibility.
- Reflect requirements back and confirm before design -- this produces the agreed statement of success.
- Jumping to a solution before discovery is complete is the classic exam trap for this task statement.
Decision Rules
When: A stakeholder proposes a solution before requirements are established
→Pause and close the discovery gap rather than proceeding.
When: A stakeholder states a preferred implementation
→Probe whether it's a want or the underlying need, to preserve design flexibility.
When: Requirements have been gathered
→Reflect them back to stakeholders and get explicit confirmation before starting design.
✗ Anti-Patterns to Reject
- Proposing a design before establishing agreed constraints (especially SLA and data sensitivity).
- Treating a stakeholder's preferred implementation as the requirement itself, losing tradeoff flexibility.
Communicate architectural decisions and tradeoffs to stakeholders
Key Points
- Anchor tradeoff conversations on the constraint that matters most to the specific stakeholder (cost, latency, quality, risk).
- Frame decisions explicitly as 'we gain X, we pay Y' -- Anthropic's own agentic-complexity guidance (latency/cost for capability) is the template.
- Never present a design as the only possibility, and never bury a tradeoff in jargon.
- Use precise technical vocabulary with technical stakeholders; translate to outcome/cost/risk for non-technical audiences.
- Present options with a named recommendation ('here are two designs, here's the tradeoff, here's what I recommend') rather than a single verdict.
Decision Rules
When: A stakeholder asks for a fully autonomous multi-agent system with a sub-second latency SLA
→Explain the latency/cost tradeoff of agentic complexity and align on an achievable SLA -- don't silently build something simpler or refuse outright.
When: Presenting a design to a non-technical executive
→Translate the tradeoff into outcome, cost, and risk with a recommendation, not raw token-cost formulas.
When: Presenting a design decision
→Name what is gained and what is paid, and offer at least one alternative considered.
✗ Anti-Patterns to Reject
- Presenting a single design as the only possibility, removing the stakeholder's ability to weigh in.
- Burying a tradeoff in jargon a stakeholder can't parse, which is functionally the same as not naming it.
Manage feedback loops, expectation alignment, and SLAs
Key Points
- SLAs must be grounded in what the chosen model tier, retrieval steps, and infrastructure can measurably deliver -- not aspiration.
- Be honest that LLM output is probabilistic -- frame quality as evaluated and monitored against a bar, not guaranteed perfect.
- Classic trap: promising sub-second latency on a multi-step agentic pipeline with reranking -- both add latency the SLA must account for.
- Expectation management is continuous, not a one-time sign-off at launch.
- Establish a defined mechanism for stakeholders to report issues that feeds prioritized iteration; proactively re-align expectations when a model version, data change, or scaling need shifts what's feasible.
Decision Rules
When: Setting an SLA
→Ground it in the measured capability of the chosen model tier, retrieval hops, and infrastructure -- adjust the architecture first if a non-negotiable SLA can't be met as designed.
When: A model version, data change, or scaling event shifts what's feasible
→Proactively re-align expectations with stakeholders rather than waiting for them to notice the gap.
When: Stakeholders need a way to report issues
→Establish a defined feedback mechanism that feeds prioritized iteration.
✗ Anti-Patterns to Reject
- Committing to an SLA (e.g., sub-second latency) that a multi-step agentic pipeline with reranking cannot meet.
- Treating expectation alignment as a one-time event at sign-off instead of a continuous activity.
Document architectures and provide implementation guidance
Key Points
- Architecture documentation covers components, data flow (input -> processing -> output -> feedback), patterns, and decision rationale.
- Implementation guidance is the actionable layer beneath documentation: pinned model versions, prompt/Skill structure, integration protocols, retrieval config, guardrails, eval/monitoring setup.
- Record the why behind each decision, not just the decision -- rationale tells future maintainers which constraints a choice served.
- Documentation quality directly determines whether the handoff lifecycle phase succeeds.
Decision Rules
When: Documenting an architecture
→Cover components, the full data flow, chosen patterns, and the rationale for each decision.
When: Handing off to an implementation team
→Provide actionable guidance (pinned versions, prompt/Skill structure, protocols, retrieval config, guardrails, eval/monitoring), not just the high-level document.
When: Recording a decision
→Capture the rationale/constraint it served, not just the decision itself.
✗ Anti-Patterns to Reject
- Logging decisions without their rationale, leaving future maintainers to unwind a choice without knowing what it protected against.
- Treating documentation as complete without the actionable implementation-guidance layer beneath it.
Support the solution across the discovery-to-iteration lifecycle
Key Points
- Five lifecycle phases: discovery, design, handoff, monitoring, iteration.
- The lifecycle is a loop, not a line -- monitoring and iteration feed back into design as requirements, data, and models evolve.
- Handoff requires documentation plus implementation guidance; monitoring requires observability/eval to already be in place from handoff.
- Treating design as 'done' at handoff is the classic exam trap -- non-determinism, model updates, and data drift demand ongoing monitoring and iteration.
- When monitoring shows an SLA breach, feed it into iteration and re-align with stakeholders, don't ignore it or silence the alert.
Decision Rules
When: Handoff occurs
→Ensure observability and evals are already in place so monitoring can function.
When: Monitoring reveals latency creeping past an agreed SLA (e.g., after a data-volume increase)
→Feed the finding back into iteration and re-align expectations with stakeholders.
When: Asked whether design is 'done' once handed off
→No -- non-determinism, model updates, and data drift require ongoing monitoring and iteration.
✗ Anti-Patterns to Reject
- Treating handoff as an end state instead of one phase in a continuing loop.
- Disabling monitoring or ignoring an SLA breach instead of feeding it back into iteration.
Domain 7: Developer Productivity & Operational Enablement
7% of examConfigure Claude tools and environments for teams
Key Points
- CLAUDE.md is memory/instructions; settings.json is deterministic configuration (permissions, hooks, MCP, env vars, model selection).
- Project memory is hierarchical: enterprise/system, user (~/.claude), project (./CLAUDE.md), and subdirectory levels.
- Project-level CLAUDE.md is committed to the repo so the whole team shares conventions and context.
- Tool allow/deny lists, hooks, environment, model selection, and MCP servers all live in settings.json.
- Configure common integrations once as shared MCP servers (build-once, reuse-across-the-team) rather than per-developer.
- Distribute team-wide Skills via repos, plugins, or enterprise managed settings so know-how is shared, not reinvented.
Decision Rules
When: Permissions or tool allow/deny rules are found in CLAUDE.md
→Move them to settings.json -- CLAUDE.md is instructions/memory, not enforcement.
When: Team-wide conventions live only in individual developers' personal ~/.claude config
→Move them to the committed project-level CLAUDE.md so every developer receives them.
When: Multiple developers need the same integration (internal API, data source)
→Configure it once as a shared MCP server rather than each developer configuring it individually.
✗ Anti-Patterns to Reject
- Putting permissions or guardrails in CLAUDE.md instead of settings.json.
- Configuring the same integration per-developer instead of as a shared MCP server, fragmenting tooling and multiplying maintenance.
Improve developer workflows with AI-assisted tooling
Key Points
- Best-practice workflow: gather-context, plan, act, verify -- not a single unsupervised step.
- Keep context lean at each stage to avoid context rot degrading output quality.
- The verify stage (tests, build, review) is what makes AI-assisted speed safe, not optional overhead.
- The Agent SDK enables building custom internal agents for CI/CD checks, codebase modernization, and repetitive engineering tasks.
- Headless/non-interactive modes let Claude Code run inside scripts and CI pipelines without a human at the keyboard.
- Custom slash commands and automation (e.g., GitHub integration) standardize common team tasks.
Decision Rules
When: Structuring an AI-assisted workflow
→Follow gather-context, plan, act, verify -- don't skip the verify step for speed.
When: A workflow needs to run in CI or on a schedule without a human present
→Use headless/non-interactive mode.
When: A team needs custom automation beyond the interactive tool (CI/CD checks, codebase modernization)
→Build it with the Agent SDK.
✗ Anti-Patterns to Reject
- Equating 'AI-assisted' with 'unsupervised' -- running an agent fully unsupervised on production as a 'productivity win.'
- Removing the verify step or over-loading context instead of keeping it lean at each stage.
Support debugging and operational issue resolution
Key Points
- Isolate a failure to transport, integration/parsing code, retrieval, or model output before applying any fix.
- Trace request/response pairs, tool calls, retrieval hits, stop_reason, and token usage back to the earliest deviation, not just the final symptom.
- Truncated output usually means stop_reason: max_tokens -- raise max_tokens, don't assume a quality or security issue.
- Model version mismatches and late-session context bloat are operational root causes distinct from prompt or code bugs.
- Feed confirmed fixes back into evals and monitoring so the same issue is caught automatically next time.
Decision Rules
When: Output is truncated
→Check for stop_reason: max_tokens and raise the limit -- don't assume prompt-injection, a compliance violation, or capability bloat.
When: A response is wrong
→Isolate which layer failed (transport, integration/parsing code, retrieval, model output) before changing any code or prompts.
When: A root cause and fix are confirmed
→Feed them back into the eval suite and monitoring dashboards to prevent recurrence.
✗ Anti-Patterns to Reject
- Patching the prompt when the bug is actually in integration/parsing code (or vice versa), fixing nothing and masking the real defect.
- Assuming a security or quality issue when the actual cause is a simple stop_reason: max_tokens truncation.