4.3–4.4 Trace Analysis, Recovery Strategies, and Evals
4.3.1 Why the Final Answer Is Rarely the Whole Story
In an agent or any multi-step workflow, a bad final answer is almost never a self-contained failure — it's the end of a chain. Something several steps earlier set up the eventual bad answer: a wrong tool choice, a tool error the agent silently ignored, or context that quietly got lost along the way. By the time you're looking at step eight's wrong output, the actual bug may have happened at step three.
Trace analysis is the tool that makes finding that earlier step possible. It means logging every model call, every tool call and its arguments, every tool result, and every intermediate message — then walking that full sequence, hop by hop, instead of staring only at the final output.
A bad final answer is the end of a chain. Trace analysis walks the whole chain to find the earliest deviation, not just the last one.
The one idea to hold onto
A bad final answer in a multi-step agent is the end of a chain, not an isolated event. Trace analysis finds where the chain actually broke.
4.3.2 What Trace Analysis Actually Logs — and What It Exposes
Concretely, a usable trace logs request/response pairs, tool invocations and their arguments, tool results, and token usage at every hop of the workflow. The discipline is to look for the earliest deviation, not the most visible one — an agent that ends badly often took its wrong turn several steps before the ending you're staring at.
- •Log every model call and its full response, not a summary of it.
- •Log every tool call, its arguments, and the raw tool result that came back.
- •Log token usage at each hop — unexpected growth is itself a signal.
- •Walk the sequence forward and mark the first point where behavior stops matching what you'd expect, rather than starting from the end and working backward from the symptom alone.
Traces also expose a category of problem that has nothing to do with any single step being wrong: context problems. If quality degrades late in a long session — the agent starts repeating itself, forgetting earlier constraints, or losing track of what it already tried — that pattern in the trace points to context bloat or drift, not a model defect. The model isn't getting worse; the context it's reasoning over is getting worse.
4.3.2 — Exam Trap
Exam trap: debugging only the final output of a multi-step agent. That hides the real cause — trace back to the first faulty step, because fixing only the last step usually just moves the symptom to a different wrong final answer instead of removing the actual bug.
4.4.1 Matching Recovery Strategy to Failure Type
Once a failure is classified into its bucket (Lesson 4.1–4.2) and, for a multi-step workflow, traced back to its earliest deviating step, the recovery strategy follows directly — each strategy addresses a different root cause, and applying the wrong one burns effort without fixing anything.
| Failure | Recovery strategy |
|---|---|
| Transient 429/529/5xx | Retry with exponential backoff + jitter |
| 400/401 (bad payload or auth) | Fix and resend — retrying unchanged is pointless |
| Malformed structured output | Defensive parsing + reprompt/repair: validate, then retry, ask the model to fix its output, or fall back |
| Truncation (stop_reason: max_tokens) | Raise max_tokens |
| Hallucinated content | Ground and constrain: add sources, allow "I don't know," tighten instructions, and add an eval |
| Automated recovery can't guarantee correctness | Graceful degradation / human-in-the-loop — surface the failure instead of shipping a wrong answer |
Strategy follows diagnosis. A 429 is time-dependent, so retrying works; a 400/401 will fail identically forever until the payload or credentials change.
4.4.1 — Key Concept
429/529/5xx get retry-with-backoff. 400/401 get fix-and-resend, never a bare retry. Malformed output gets defensive parsing plus repair. Truncation gets a higher max_tokens. Hallucination gets grounding and constraining, not a retry. When nothing automated can guarantee correctness, escalate to graceful degradation or a human.
4.4.2 Evals: Closing the Loop
Applying a recovery strategy is not the end of debugging. To know whether a fix actually helped — and to catch a regression the next time the underlying model version changes — you need an evaluation, not a glance at the one example that originally failed.
- 1.A representative test set — covering the cases that matter, not just the single input that originally broke.
- 2.A scoring method — exact match, a graded rubric, or LLM-as-judge.
- 3.A target metric — the number that defines "good enough" so the check is repeatable, not a matter of taste.
The reason a single spot-check can't substitute for this: Claude's output is non-deterministic. A fix that looks like it worked on one re-run of the failing example can still fail on a slightly different input, or even on another run of the exact same input. One clean run proves nothing about the general case — only a scored test set against a metric can distinguish "the fix worked" from "this particular run happened to come out fine," and only a repeatable eval will catch a regression a future model version quietly introduces.
4.4.2 — Exam Trap
Exam trap: treating "it looks fixed on one example" as verification. Non-deterministic output requires an eval set and a metric, not a single manual spot-check — and skipping the eval on the assumption that the strategy's logic alone ("we now ground with sources") guarantees the outcome improved is the same mistake wearing a different hat.
4.4.3 Building and Calibrating an LLM-as-Judge
LLM-as-judge is one of the three scoring methods available to an eval, reserved for open-ended quality -- faithfulness, instruction-following, tone -- that no code rule can express. Concretely, it's a second model call: hand a grading model the task and the solution, guided by a rubric, and it returns a score. Building one well means being deliberate about exactly what you ask it to return, in what order, and earning trust in its output before relying on it in production.
def grade_by_model(task, solution):
eval_prompt = f"""
Act as a careful reviewer grading the solution against the task.
Task: {task}
Solution: {solution}
Respond with JSON containing:
"strengths": 1-3 specific things the solution does well
"weaknesses": 1-3 specific things it gets wrong or misses
"reasoning": a short justification, roughly 50 words or fewer
"score": an integer from 1 to 10
"""
messages = [{"role": "user", "content": eval_prompt}]
result = chat(messages) # returns the JSON above
return json.loads(result)4.4.3 -- Key Concept
The ordering is not cosmetic. Asking for strengths, weaknesses, and reasoning before score anchors the number to reasoning the judge already committed to in writing. Without that ordering, judge models measurably drift toward a safe middle number -- around 6 out of 10 -- almost regardless of actual quality. Once the judge has written down two specific weaknesses, giving the same output a 9 would visibly contradict its own stated reasoning.
A judge prompt that returns clean JSON is not the same thing as a judge whose scores mean anything. Before trusting a judge in production, measure its agreement rate against a set of human-labeled examples: take cases a human has already scored, run the judge on the same cases, and check how often its verdict matches. A judge whose calls only line up with the human's about half the time is producing a number that looks authoritative but tells you nothing real. If agreement comes back low, the fix is to iterate on the rubric -- tighten what each score band means, add explicit good/bad examples directly in the judge prompt, and re-measure -- not to discard the judge or, worse, deploy it uncalibrated and hope.
4.4.3 -- Exam Trap
A judge prompt that returns only a bare numeric score, with no strengths/weaknesses/reasoning fields, has nothing anchoring its number -- scores drift to an uninformative safe middle value. And deploying a judge straight from its prompt design, without calibrating against human-labeled examples first, can look rigorous (clean JSON, confident numbers) while actually tracking nothing.
4.3–4.4 Put It Together: From Symptom to Verified Fix
The full arc of this small domain is one continuous procedure: classify the failure into its bucket, isolate which layer owns it, and — if it's a multi-step workflow — trace back to the earliest deviation rather than the visible symptom. Only then pick the recovery strategy that matches the actual cause, never the one that's most convenient to apply. And the arc doesn't end at the fix: build (or extend) an eval so "seems better" becomes a measurable, repeatable check that also protects you against the next model upgrade quietly undoing your work.
Where this shows up on the exam
Domain 4 questions tend to describe a symptom (a wrong final answer, a crash, a truncated response, a hallucinated fact) and ask for the correct next action. Work the chain in order — classify, isolate/trace, then match the recovery strategy — and treat "verify with an eval" as the closing step every time a fix is proposed.
Key Takeaways
- ✓A bad final answer in a multi-step agent is usually the end of a chain — trace analysis finds where the chain actually broke, not just where it ended.
- ✓Log every model call, tool call, tool result, and intermediate message; look for the earliest deviation, not the most visible one.
- ✓Late-session quality degradation exposed in a trace often points to context bloat/drift, not a model defect.
- ✓Recovery strategy follows diagnosis: retry-with-backoff for transient 429/529/5xx, fix-and-resend for 400/401, defensive parsing/repair for malformed output, raise max_tokens for truncation, and ground-and-constrain for hallucination.
- ✓When automated recovery can't guarantee correctness, escalate to graceful degradation or a human in the loop instead of shipping a wrong answer.
- ✓An eval needs a representative test set, a scoring method (exact match, rubric, or LLM-as-judge), and a target metric — this is what verifies a fix and catches future regressions.
- ✓Non-deterministic output means a single manual spot-check cannot verify a fix; only a scored eval set can.
- ✓A judge prompt should return structured JSON -- 1-3 strengths, 1-3 weaknesses, a reasoning field capped near 50 words, then a final score -- rather than a bare number
- ✓Asking for reasoning before the score anchors the score to stated reasoning and measurably reduces drift toward a safe middle number (around 6/10) regardless of actual quality
- ✓Calibrate a judge before trusting it in production: measure its agreement rate against human-labeled examples, and if agreement is low, tighten the rubric and add explicit good/bad examples rather than deploying it uncalibrated
Check Your Understanding
Test what you learned in this lesson.
Q1.An agent produces a wrong final answer after eight steps. What is the best debugging approach?
Q2.Which failure should be handled with exponential backoff and retry rather than a code change?
Q3.How do you confirm a hallucination fix actually worked and won't regress on the next model version?
Q4.A long-running agent session shows steadily degrading answer quality over many turns, with no single tool error or wrong tool choice visible in the trace. What does this pattern most likely indicate?
Q5.A team fixes a malformed-JSON bug and confirms it by re-running the one example that used to fail; it now returns clean JSON. Is this sufficient verification?
Q6.Two judge-prompt designs are proposed: one asks for a bare 1-10 score only; the other asks for strengths, weaknesses, and reasoning first, then the score. Which is preferred, and why?
Q7.A team builds an LLM-as-judge, gets clean structured JSON output on every test case, and deploys it to production without further checks. What is missing?
Practice This Lesson