Domain 4: Eval, Testing, and Debugging
2.6% of examIdentify the type of a Claude application error
Key Points
- Five failure buckets: transport/HTTP (429/529/5xx/timeouts), request error (400/401), parsing/validation (code throws reading output), model-output error (well-formed but wrong), tool-loop error (wrong tool/malformed args/result not fed back).
- Three buckets live in the integration layer, one lives in the model's output, tool-loop errors straddle both.
- Core discipline: don't "fix" the prompt when the bug is in your code, and don't patch code when the model output is the problem.
- A JSON parse crash on a 200/end_turn response is an integration-layer parsing bug, not evidence the model is wrong.
Decision Rules
When: Code throws while reading a successful (200 status, end_turn) response
→Classify it as an integration-layer parsing bug and add defensive parsing.
When: Content is well-formed but factually or semantically wrong
→Classify it as a model-output error, not a parsing bug.
When: Classifying any failure
→Identify which of the five buckets it lives in before choosing a fix.
✗ Anti-Patterns to Reject
- "Fixing" the prompt when the bug is actually in the integration code.
- Treating a parsing exception as evidence the model hallucinated.
Isolate integration-layer failures from model-output failures
Key Points
- Reproduce-inspect-localize procedure: reproduce with the exact failing request; inspect the raw response (status, stop_reason, usage, content blocks) before any post-processing; localize by status and content quality.
- Non-2xx status localizes to the integration/transport layer; 2xx-but-throws localizes to parsing/validation; 2xx-and-parses-but-wrong localizes to model output.
- stop_reason: max_tokens means the output was truncated -- a common, easily misdiagnosed cause of "invalid JSON".
- Fix truncation by raising max_tokens, not by rewriting the schema or lowering temperature.
Decision Rules
When: Debugging any failure
→Reproduce with the exact request, then inspect the raw response before any post-processing runs.
When: stop_reason is max_tokens and the JSON looks malformed
→Raise max_tokens; don't redesign the schema or lower temperature.
When: Status is 200 with a clean stop_reason but your code still throws
→Treat it as a bug in your consumer, not evidence the model is wrong.
✗ Anti-Patterns to Reject
- Assuming the model is at fault just because your code raised an exception.
- Rewriting the schema or lowering temperature in response to a truncation-caused parse error.
Use trace analysis to find failure modes in multi-step workflows
Key Points
- A bad final answer in a multi-step agent is usually the end of a chain of steps, not an isolated event.
- Trace analysis: log every model call, tool call, arguments, tool result, and intermediate message, then walk the sequence.
- Look for the earliest deviation, not just the most visible or final one.
- Late-session quality degradation exposed in a trace often points to context bloat/drift, not a model defect.
Decision Rules
When: An agent produces a wrong answer after several steps
→Analyze the trace and find the earliest deviating step rather than only rewording the final prompt.
When: Quality degrades late in a long session
→Check the trace for context bloat or drift before assuming a model limitation.
✗ Anti-Patterns to Reject
- Debugging only the final output of a multi-step agent.
- Fixing only the last step, which often just moves the symptom rather than removing the cause.
Select recovery strategies and verify fixes with evals
Key Points
- Recovery strategy follows diagnosis: retry+backoff for 429/529/5xx; fix-and-resend for 400/401; defensive parsing+reprompt/repair for malformed output; raise max_tokens for truncation; ground-and-constrain for hallucination; graceful degradation/human-in-the-loop when automated recovery can't guarantee correctness.
- Evals close the loop: a representative test set, a scoring method (exact match, graded rubric, or LLM-as-judge), and a target metric.
- Non-deterministic output means a single manual spot-check cannot verify a fix -- only a repeatable eval can, and it also catches future regressions.
Decision Rules
When: A fix has been applied to any Claude-application bug
→Verify it with an eval set and metric, not a single manual re-run.
When: Content is hallucinated
→Ground and constrain (add sources, allow "I don't know", tighten instructions) rather than simply retry.
When: Automated recovery can't guarantee correctness
→Escalate to graceful degradation or human-in-the-loop instead of shipping a wrong answer.
✗ Anti-Patterns to Reject
- Treating "it looks fixed on one example" as verification.
- Applying retry-with-backoff to 400/401 errors instead of fixing the payload or credentials.