Matching Recovery Strategy to Failure Type
CoreSelect recovery strategies and verify fixes with evals · Difficulty 2/5
Explanation
Strategy Follows Diagnosis
Once a failure is classified into its bucket, the recovery strategy follows directly:
| Failure | Recovery strategy |
|---|---|
Transient 429/529/5xx | Retry with exponential backoff + jitter |
400/401 (bad payload or auth) | Fix and resend -- retrying unchanged is pointless |
| Malformed structured output | Defensive parsing + reprompt/repair: validate, and on failure either retry, ask the model to fix its output, or fall back |
Truncation (stop_reason: max_tokens) | Raise max_tokens |
| Hallucinated content | Ground and constrain: add sources, allow "I don't know," tighten instructions, and add an eval |
| Automated recovery can't guarantee correctness | Graceful degradation / human-in-the-loop: surface the failure instead of shipping a wrong answer |
Why the Mapping Matters
Each strategy addresses a different root cause. Retrying a transient overload works because the failure is time-dependent; retrying an unchanged 400/401 request does nothing because the payload or credentials are still wrong. Applying the wrong strategy -- retrying a bad-request error, or fixing code for a hallucination -- burns effort without addressing the actual failure.
Common exam traps
- Applying retry-with-backoff to
400/401errors. These need a fix (corrected payload or credentials), not a retry -- the request will fail identically every time until it's changed. - Treating a hallucinated citation the same as a transient server error. Hallucination is a model-output problem addressed by grounding and constraining, not by backoff-and-retry.
Key Takeaways
- Retry with exponential backoff + jitter for transient 429/529/5xx failures
- Fix and resend for 400/401 -- retrying an unchanged request is pointless
- Defensive parsing + reprompt/repair for malformed structured output; raise max_tokens for truncation
- Ground and constrain for hallucinated content; escalate to graceful degradation/human-in-the-loop when automated recovery can't guarantee correctness
- 429 needs backoff-and-retry, but 400/401 need a fix and hallucination needs grounding -- the strategies are not interchangeable
Glossary Terms
The five recognizable buckets a Claude application failure falls into -- transport/HTTP (429/529/5xx), request error (400/401), parsing/validation (a code-level exception reading the response), model-output error (well-formed but wrong content), and tool-loop error (wrong tool, malformed arguments, or a mis-fed tool_result). The bucket a failure falls into dictates the correct fix, and misclassifying it sends the fix to the wrong layer.
A system design principle where partial failures result in reduced functionality rather than complete failure. In multi-agent systems, if one subagent fails, the coordinator produces a partial result with a clear explanation of what is missing rather than returning an error to the user.
A design pattern that interrupts the agentic loop at defined checkpoints to request human review or approval before proceeding. Used for high-stakes decisions, irreversible actions, or cases where confidence is below threshold. Balances automation with oversight.
A pattern where structured output that fails validation is sent back to Claude with the specific error, requesting a correction. More effective than silent retries because Claude uses the error feedback to understand and fix the problem. Typically capped at 2-3 retries.
Related Concepts
Error-Type Taxonomy: Transport, Request, Parsing, Model-Output, Tool-Loop
Five recognizable failure buckets: transport/HTTP, request error, parsing/validation, model-output, and tool-loop
stop_reason: max_tokens and Truncation
stop_reason: max_tokens means the output was truncated before completion
Evals as Verification: Closing the Loop
An eval needs a representative test set, a scoring method (exact match, rubric, or LLM-as-judge), and a target metric