Trace Analysis to Localize the Earliest Faulty Step
CoreUse trace analysis to find failure modes in multi-step workflows · Difficulty 2/5
Explanation
Why the Final Answer Is Rarely the Whole Story
In an agent or multi-step workflow, a bad final answer is usually the *end* of a chain of steps, not a self-contained failure. Something several steps earlier -- a wrong tool choice, an ignored tool error, or lost context -- often set up the eventual bad answer. Trace Analysis is the tool that makes finding that earlier step possible.
What Trace Analysis Is
Trace Analysis means logging every model call, tool call, its arguments, the tool result, and every intermediate message, then walking that sequence to find the first step that went wrong:
- Log request/response pairs, tool invocations, and token usage at each hop.
- Look for the earliest deviation, not the most visible one -- an agent that ends badly often took a wrong turn several steps earlier.
- Traces also expose context problems: quality that degrades late in a long session points to context bloat/drift, not a model defect.
Common exam traps
- Debugging only the final output of a multi-step agent, which hides the real cause. Trace back to the first faulty step; fixing the last step often just moves the symptom to a different final output rather than eliminating the actual bug.
- Assuming a late-session quality drop must be a model limitation, when the trace shows context bloat or drift accumulating over the session instead.
Key Takeaways
- A bad final answer in a multi-step agent is usually the end of a chain -- trace analysis finds where the chain broke
- Log every model call, tool call, arguments, tool result, and intermediate message
- Look for the earliest deviation, not just the final failing output
- Fixing only the last step often just moves the symptom rather than removing the cause
- Late-session quality degradation exposed in a trace often points to context bloat/drift, not a model defect
Glossary Terms
Attention degradation caused by a context window filling with irrelevant, stale, or low-signal content, even when technically there is still room left in the window. Distinct from running out of space (a hard context-window limit) and from position effects (attention bias by location within the window) -- context rot is specifically about signal-to-noise degrading as low-value tokens accumulate.
A field in the Claude API response indicating why the model stopped generating. Values: 'end_turn' (natural completion), 'max_tokens' (hit limit), 'stop_sequence' (hit custom stop), 'tool_use' (wants to call a tool). The primary signal for controlling agentic loops.
A content block type in the user message that returns the output of a tool execution back to Claude. Must include the 'tool_use_id' matching the original tool_use block. Can be text, images, or error messages. Claude processes the result and continues reasoning.
A content block type in Claude's response indicating the model wants to call a specific tool. Contains 'id', 'name', and 'input' fields. The agent must execute the tool and return results in a tool_result content block for the conversation to continue.
A debugging technique for multi-step agents and workflows: logging every model call, tool call and its arguments, tool result, and intermediate message, then walking that sequence to find the earliest step that deviated -- rather than debugging only the final failing output.
Related Concepts
Error-Type Taxonomy: Transport, Request, Parsing, Model-Output, Tool-Loop
Five recognizable failure buckets: transport/HTTP, request error, parsing/validation, model-output, and tool-loop
Matching Recovery Strategy to Failure Type
Retry with exponential backoff + jitter for transient 429/529/5xx failures