The Reproduce-Inspect-Localize Procedure
CoreIsolate integration-layer failures from model-output failures · Difficulty 2/5
Explanation
A Reliable Isolation Procedure
When a Claude application fails, a repeatable procedure isolates which layer owns the bug:
- Reproduce with the exact request that failed (same messages, params, tools, model).
- Inspect the raw response -- status code, `stop_reason
,usage`, and the full content blocks -- *before* any of your post-processing runs. - Localize:
- Error/non-2xx status → integration layer (request/transport). Check payload, auth, rate limits.
- 2xx but your code throws → parsing/validation layer. The model returned something; your consumer was too strict.
- 2xx, parses fine, but content is wrong → model-output layer. Now it's a prompt/context/model-fit problem.
Why Inspecting Before Post-Processing Matters
The raw response is the only reliable evidence. If you only ever look at your code's already-transformed view of the response, you cannot tell whether a problem originated in what the model actually sent or in how your code touched it afterward. Step 2 exists specifically to preserve that evidence before it's lost.
Common exam traps
- Assuming that because your code raised an exception, the model must be at fault. A
200status with a clean `stop_reason(e.g.,end_turn`) means the model responded successfully -- a parse crash on top of that is your consumer being too strict, not the model being wrong. - Skipping reproduction and jumping straight to a fix based on the symptom alone, which risks fixing a coincidence rather than the actual failing request.
Key Takeaways
- Reproduce with the exact failing request before attempting any fix
- Inspect the raw response -- status, stop_reason, usage, content blocks -- before any post-processing runs
- Non-2xx status localizes to the integration/transport layer; 2xx-but-throws localizes to parsing/validation; 2xx-and-parses-but-wrong localizes to model output
- A 200 status with a clean stop_reason means the model responded; a parsing exception on top of that is on your consumer, not the model
Glossary Terms
The five recognizable buckets a Claude application failure falls into -- transport/HTTP (429/529/5xx), request error (400/401), parsing/validation (a code-level exception reading the response), model-output error (well-formed but wrong content), and tool-loop error (wrong tool, malformed arguments, or a mis-fed tool_result). The bucket a failure falls into dictates the correct fix, and misclassifying it sends the fix to the wrong layer.
API parameter that sets the maximum number of tokens Claude will generate in a single response. If generation would exceed this limit it is truncated and `stop_reason` is set to `"max_tokens"`. This is a required parameter — omitting it returns a 400 error.
A field in the Claude API response indicating why the model stopped generating. Values: 'end_turn' (natural completion), 'max_tokens' (hit limit), 'stop_sequence' (hit custom stop), 'tool_use' (wants to call a tool). The primary signal for controlling agentic loops.
A debugging technique for multi-step agents and workflows: logging every model call, tool call and its arguments, tool result, and intermediate message, then walking that sequence to find the earliest step that deviated -- rather than debugging only the final failing output.
Related Concepts