Issue Isolation: Localizing Failures Across Layers
CoreSupport debugging and operational issue resolution · Difficulty 3/5
Explanation
When a Claude-powered system misbehaves, the architect's role is to localize the failure to the correct layer before touching anything -- reusing the diagnostic discipline from evaluation and monitoring work (Domain 4).
Isolate the Layer
A failure can originate in one of four distinct layers, and the layer determines the fix:
| Layer | What Goes Wrong | Example |
|---|---|---|
| Transport | HTTP/auth errors | Request never reaches the model correctly |
| Integration/parsing code | A successful (200) response mishandled by your code | Response parsed incorrectly downstream |
| Retrieval | Stale or irrelevant chunks | RAG returns the wrong context |
| Model output | Well-formed but wrong | The model answered fluently but incorrectly |
Patching the wrong layer -- for example, rewriting the prompt when the real bug is in integration code that mishandles a valid response -- fixes nothing and can mask the actual defect.
Use Traces and Logs
Walk request/response pairs, tool calls, retrieval hits, `stop_reason, and token usage` back to the earliest deviation, not just the final symptom. The visible failure (a bad answer, an error message) is often several steps downstream of where things actually went wrong; tracing backward from the symptom to the first point of deviation is what correctly identifies the layer.
Check the Operational Basics
Before assuming a deeper defect, rule out common operational causes:
- Truncated output -- check for `stop_reason: max_tokens
; the fix is raisingmax_tokens`, not rewriting the prompt - Model version change -- a model mismatch between what was tested and what is deployed can silently change behavior
- Context bloat late in a session -- an overloaded context window degrades output quality independent of any code or prompt defect
Feed Fixes Back
Once the root cause and fix are confirmed, feed them back into evals and monitoring so the same issue is caught automatically next time, rather than relying on the same manual diagnosis to recur.
Common exam traps
- Patching the prompt when the bug is in the integration code (or vice versa). Isolate the origin before changing anything -- an exam item describing a truncated response should point to `stop_reason: max_tokens`, not to prompt-injection, a compliance violation, or "capability bloat."
Key Takeaways
- Isolate a failure to transport, integration/parsing code, retrieval, or model output before applying any fix
- Trace request/response pairs, tool calls, retrieval hits, stop_reason, and token usage back to the earliest deviation
- Truncated output usually means stop_reason: max_tokens -- raise max_tokens, don't assume a quality or security issue
- Model version mismatches and late-session context bloat are operational root causes distinct from prompt or code bugs
- Feed confirmed fixes back into evals and monitoring so the same issue is caught automatically next time
- Patching the wrong layer (prompt vs. integration code) fixes nothing and can mask the real defect
Glossary Terms
API parameter that sets the maximum number of tokens Claude will generate in a single response. If generation would exceed this limit it is truncated and `stop_reason` is set to `"max_tokens"`. This is a required parameter — omitting it returns a 400 error.
A field in the Claude API response indicating why the model stopped generating. Values: 'end_turn' (natural completion), 'max_tokens' (hit limit), 'stop_sequence' (hit custom stop), 'tool_use' (wants to call a tool). The primary signal for controlling agentic loops.
A pattern that dynamically retrieves relevant information from an external knowledge base and injects it into the context window based on the current query. Allows Claude to reason over large document sets without fitting everything in context at once.
The practice of returning structured, actionable error information from tools rather than generic error strings. Well-designed error responses include: error type, what went wrong, what Claude should try next. Prevents Claude from retrying the same failing approach repeatedly.
Related Concepts
The Gather-Context, Plan, Act, Verify Loop
The best-practice workflow is gather-context, plan, act, verify -- not a single unsupervised step
Programmatic Enablement: Agent SDK & Headless Mode
The Agent SDK enables building custom internal agents for CI/CD checks, codebase modernization, and repetitive engineering tasks