4.1–4.2 Identifying Error Types and Isolating Failures
4.1.1 The Five Failure Buckets
This is a small domain — just 2.6% of the exam — but it tests one skill you'll use every single day building with Claude: when something goes wrong, where did it actually go wrong? A Claude application fails in recognizable, repeatable ways, and those failures sort cleanly into five buckets. The bucket a failure lands in dictates the fix, which means misclassifying the bucket sends you off to fix the wrong thing entirely — tuning a prompt when the bug is in your code, or rewriting your parser when the model genuinely got the answer wrong.
Three of the five buckets live squarely in your integration layer — the code you wrote to build the request and consume the response. One lives in the model's actual output. And one, tool-loop errors, straddles both, because a tool call is a handoff between your code and the model's judgment about what to call and how.
| Failure type | Where it lives | Signal |
|---|---|---|
| Transport/HTTP | Integration layer | 429 rate limit, 529/5xx overload, timeouts |
| Request error | Integration layer | 400 bad request, 401 auth, schema/max_tokens mistakes |
| Parsing/validation | Integration layer | Code throws when reading Claude's output (bad JSON, missing field) |
| Model-output error | Model output | Well-formed but wrong: hallucinated fact, missed instruction, wrong format |
| Tool-loop error | Integration ↔ model | Wrong tool chosen, malformed arguments, tool result not fed back correctly |
Five recognizable failure buckets. Notice how much of the surface area is integration-layer, not model-layer — most "the model is broken" instincts are misdiagnoses.
The one idea to hold onto
Five buckets, one question: is this a transport problem, a request problem, a parsing problem, a model-output problem, or a tool-loop problem? Answer that before you touch anything.
4.1.2 The Core Discipline and a Common Trap
The discipline that matters most in this whole domain fits in one sentence: don't "fix" the prompt when the bug is in your code, and don't patch code when the model output is the problem. It sounds obvious stated flatly, but under time pressure — a production incident, a failing test, a frustrated stakeholder — the reflex is to blame whichever layer is easiest to touch, not whichever layer the evidence actually points to.
The single most common version of this trap: your code raises an exception — a JSON parse error, a KeyError on a missing field — while reading Claude's response. It's tempting to read that crash as proof the model produced garbage. But a crash while reading the response is an integration-layer parsing bug, not a model-quality problem. The model may have returned entirely reasonable text; your parser was simply too strict to tolerate it. The fix is defensive parsing on your side, not a prompt rewrite aimed at a model that may not have done anything wrong.
4.1.2 — Exam Trap
Exam trap: treating a parsing exception as evidence the model hallucinated or misbehaved, and treating a genuinely wrong-but-well-formed answer as if it were a parsing bug. The fix lives wherever the failure signal actually points — not wherever is easiest to blame.
4.1.3 Retriable vs. Terminal: A Complete Two-Set Classification
Every failure a production Claude application hits starts with one question: would waiting and retrying the exact same request plausibly succeed? If yes, the failure is retriable. If not, it's terminal. That single question -- not the status code by itself -- is what actually decides how a failure should be handled.
RETRIABLE = {429, 529, 500, 502, 503, 504} # rate limit, overload, transient server fault
TERMINAL = {400, 401, 403, 404} # bad request, auth, permissions, not found
def is_retriable(status):
return status in RETRIABLE # everything else fails fastFor a status you haven't seen before, apply the same underlying test fresh. If genuinely unsure, default to treating it as terminal -- the asymmetry is deliberate. A request wrongly classified as terminal fails loudly and gets noticed and fixed quickly. A request wrongly classified as retriable does something worse: it silently hammers a service that may already be struggling, burning retry budget without ever surfacing that anything is wrong.
One case falls completely outside this classification and is easy to miss: a refusal. When Claude declines a request on content-policy or safety grounds, that arrives as stop_reason == "refusal" -- riding on an ordinary 200 HTTP status. A classifier that only inspects the status code will see 200 and wave it through as an unremarkable success.
# A refusal rides on an ordinary 200 -- the retriable/terminal
# classifier above never sees it, since it never inspects stop_reason.
if response.stop_reason == "refusal":
raise ApplicationRefusalError("Claude declined this request -- flag for manual review, do not retry.")4.1.3 -- Exam Trap
Defaulting an unfamiliar error to retriable 'to be safe' is backwards -- the safe default is terminal. And a 200 status does not always mean a normal, retry-safe response: stop_reason == "refusal" is a 200 that is neither retriable nor terminal in the usual sense -- it's a content decision that must be raised explicitly and never retried.
4.1.4 The Four Test Levels: Unit, Functional, Integration, End-to-End
The five failure buckets in 4.1.1 classify a failure after it happens. Test levels are a complementary, design-time idea: they classify what a given test is actually capable of catching before anything fails, so a passing test suite doesn't quietly hide a gap nobody wrote a test for. This is a distinct taxonomy from the failure buckets -- and complements, rather than replaces, the reproduce-inspect-localize procedure in 4.2.1, which works backward from a failure that already happened in production.
| Level | What it isolates | What it cannot catch |
|---|---|---|
| Unit | A single function on its own -- a parser, a tool wrapper, isolated from everything around it | Whether the pieces actually cooperate once wired together |
| Functional | A single Claude call, checked for the right shape and type given a particular input | Anything going wrong in the surrounding system that call is embedded in |
| Integration | The join between two components -- for instance, a retrieval result feeding into a model call | Problems that only surface once the entire pipeline runs together |
| End-to-end | The complete path from input to output, run the same way a real user would trigger it | Pinpointing which link in the chain actually failed -- it shows you the outcome, not the cause, and it's the slowest tier to execute |
Know exactly which failure a given test is designed to surface -- and just as importantly, which ones it's blind to.
The integration level is where most silent failures hide, and it's worth dwelling on why: each side of a handoff can pass its own test in complete isolation while the handoff between them is still broken. A retrieval function is unit-tested and returns a list of chunk dictionaries -- it passes. A prompt-builder function is functionally tested with a well-formed plain string -- it also passes. Wire them together, though, and the list of dicts gets inserted where the prompt-builder expected a plain string; the context arrives malformed, and the model answers from its own memory instead of the retrieved content. Neither test could have caught this, because the unit itself works and the call works fine on well-formed input -- the failure lives specifically in the undefined contract between the two components, and only a test that drives them together with real data exercises that seam at all.
Where this shows up on the exam
Test levels are a design-time discipline (which test would catch this class of bug) that works alongside, not instead of, the post-hoc reproduce-inspect-localize procedure (4.2.1). A missing integration test is exactly the kind of gap that procedure eventually surfaces in production -- the fix, once localized, is often adding the integration test that should have existed already.
4.2.1 The Reproduce–Inspect–Localize Procedure
Classifying a failure into one of the five buckets requires evidence, and evidence requires a repeatable procedure — not a guess based on the symptom that happened to be visible. The procedure has three steps, always in this order: reproduce, inspect, localize.
- 1.REPRODUCE — rerun the exact request that failed: same messages, same params, same tools, same model. A fix aimed at a coincidence, rather than the real failing request, will not hold up.
- 2.INSPECT — look at the raw response before any of your post-processing runs: the status code, stop_reason, usage, and the full content blocks. This is the only reliable evidence — your code's already-transformed view of the response can't tell you whether a problem started with what the model sent or with what your code did to it afterward.
- 3.LOCALIZE — map what you observed to a layer: a non-2xx status means the integration/transport layer (check payload, auth, rate limits); a 2xx status that your code still throws on means the parsing/validation layer; a 2xx status that parses cleanly but contains wrong content means the model-output layer, which is now a prompt/context/model-fit problem.
The raw response — inspected before your code transforms it — is the only evidence that reliably separates integration-layer bugs from model-output problems.
4.2.1 — Key Concept
Reproduce with the exact failing request. Inspect the raw response — status, stop_reason, usage, content blocks — before any post-processing runs. Localize: non-2xx → integration/transport; 2xx-but-throws → parsing/validation; 2xx-and-parses-but-wrong → model output.
4.2.2 stop_reason and the Truncation Trap
One field the inspect step calls out by name deserves its own attention, because a single value hiding inside it is a frequent source of misdiagnosis: stop_reason. When stop_reason is max_tokens, the output was cut off before Claude finished — the response is truncated, not malformed on its own terms.
Truncation is a master of disguise. A response cut off mid-object very often looks exactly like a schema bug: JSON that ends without a closing brace, a string that never got its closing quote. Without checking stop_reason first, it's easy to conclude the model produced invalid JSON, when the real story is that the model was still writing valid JSON and simply ran out of the token budget it was given.
The correct fix is to raise max_tokens — not to redesign the schema, simplify the output format, or lower temperature. None of those touch the actual cause. The model didn't fail to produce valid structure; it wasn't given enough room to finish producing it.
4.2.2 — Key Concept
stop_reason: max_tokens means truncation. Truncated output commonly presents as malformed JSON and invites misdiagnosis as a model or schema problem — but the correct fix is raising max_tokens, never rewriting the schema or lowering temperature.
4.1–4.2 Put It Together: Reading a Failure's Signal
Put the taxonomy and the procedure side by side and a single workflow emerges: when something fails, reproduce it, inspect the raw response, and let the observed signal — not your gut instinct about which layer feels more likely at fault — tell you which of the five buckets you're in. A 200 status with a clean stop_reason like end_turn means the model responded successfully; anything that goes wrong after that point is on your side of the boundary.
- •Non-2xx status → transport or request error → integration layer → check payload, auth, or retry logic.
- •2xx status but your code throws → parsing/validation error → integration layer → add defensive parsing.
- •2xx, stop_reason: max_tokens, structure looks cut off → truncation → integration layer → raise max_tokens.
- •2xx, parses cleanly, content is wrong → model-output error → now it's a prompt/context/model-fit problem.
- •Wrong tool chosen or malformed tool arguments → tool-loop error → straddles both layers — inspect the specific tool call and result.
Where this shows up on the exam
Exam questions in this domain almost always hand you a status code, a stop_reason, or a description of where an exception was raised, then ask which layer owns the bug. Match the signal to the bucket first — the correct recovery action (Lesson 4.3–4.4) follows automatically once the layer is right.
Key Takeaways
- ✓Five recognizable failure buckets: transport/HTTP, request error, parsing/validation, model-output error, and tool-loop error.
- ✓Transport, request, and parsing errors live in the integration layer; model-output errors are well-formed but wrong; tool-loop errors straddle both.
- ✓Don't fix the prompt when the bug is in your code, and don't patch code when the model output is the actual problem.
- ✓A JSON parse crash on a 200 response with a clean stop_reason is an integration-layer bug, not evidence the model is wrong.
- ✓The isolation procedure is reproduce (exact failing request) → inspect (raw response before post-processing) → localize (map status/content to a layer).
- ✓stop_reason: max_tokens means truncation, not a schema or model-quality problem — the fix is raising max_tokens, not rewriting the schema.
- ✓RETRIABLE = {429, 529, 500, 502, 503, 504}; TERMINAL = {400, 401, 403, 404} -- the underlying test is 'would waiting and retrying the exact same request plausibly succeed?'; default an unfamiliar status to terminal if unsure
- ✓A refusal (stop_reason == "refusal") arrives with an ordinary 200 status, so a status-code-only classifier will not catch it -- it must be checked explicitly, raised, logged, and never silently retried
- ✓Four test levels -- unit, functional, integration, end-to-end -- classify what a test can catch; silent failures cluster at the integration level, the join between two components, because each side can sail through its own test while the connection between them stays broken
Check Your Understanding
Test what you learned in this lesson.
Q1.Your code raises a JSON parse error on Claude's response, which returned HTTP 200 and stop_reason: "end_turn". Where is the bug most likely?
Q2.A response is truncated mid-JSON and stop_reason is "max_tokens". What is the correct fix?
Q3.A tool-loop error — the model calls the wrong tool with malformed arguments — is best described as living in which layer?
Q4.During the isolation procedure, a request returns HTTP 429. Which step of reproduce–inspect–localize does this observation belong to, and what does it tell you?
Q5.A response returns HTTP 200 with stop_reason: "refusal". A retry system classifies purely by HTTP status code. What happens, and what should happen instead?
Q6.A retrieval function returns [{"content": "..."}] and passes its unit test. A prompt-builder function correctly builds a prompt from a plain string and passes its functional test. Wired together, the agent answers from memory instead of the retrieved content. What test level would have caught this, and why?
Practice This Lesson