Diagnosing Quality Regressions: A Decision Table
CoreDiagnose production system issues by localizing the failure layer · Difficulty 3/5
Explanation
When quality drops, localize the cause before fixing -- patching the wrong layer just moves the symptom instead of resolving it.
Symptom -> Cause -> First Move
| Symptom | Likely cause | First move |
|---|---|---|
| Confident but wrong after a document refresh (model/latency unchanged) | Retrieval/indexing returning stale or irrelevant chunks | Inspect the retrieval step and re-index |
| Well-formed output that's factually invented | Hallucination | Ground with retrieval, constrain claims, add citations |
| Quality regressed right after a model version change | Model mismatch | Re-run evals; pin/roll back the version |
| Output ignores or misreads the instruction | Prompt failure | Clarify instruction, fix placement, add few-shot |
| Truncated output | stop_reason: max_tokens | Raise max_tokens (not a prompt or model problem) |
| Quality falls late in a long session | Context bloat/drift | Prune, compact, or isolate context |
Reading the Table
The table works by elimination: hold constant what didn't change (model, latency, version) and vary what did (the data, the session length) to isolate the layer. A document refresh with model and latency unchanged points at retrieval, not the model -- a broken re-index or mismatched embeddings is the likely culprit.
A well-formed JSON output that's factually invented is a hallucination, not a transport error, truncation, or model-mismatch regression -- grounding and constraint are the fixes, not a version rollback.
Common exam traps
- Blaming "the model got worse" for what is actually a retrieval or context problem. If latency and version are unchanged but the underlying data changed, suspect retrieval; if the version just changed, suspect model mismatch and re-run evals to confirm before rolling back.
Key Takeaways
- Localize the failure layer before fixing -- don't patch the wrong layer
- Confident-but-wrong after a document refresh (model/latency unchanged) points to retrieval/indexing
- Well-formed but factually invented output is a hallucination, fixed with grounding/constraints/citations
- A regression right after a model-version change is model mismatch -- re-run evals, pin/roll back
- Truncated output with stop_reason: max_tokens is a token-limit issue, not a prompt or model problem
- Quality falling late in a long session points to context bloat/drift
Glossary Terms
API parameter that sets the maximum number of tokens Claude will generate in a single response. If generation would exceed this limit it is truncated and `stop_reason` is set to `"max_tokens"`. This is a required parameter — omitting it returns a 400 error.
A field in the Claude API response indicating why the model stopped generating. Values: 'end_turn' (natural completion), 'max_tokens' (hit limit), 'stop_sequence' (hit custom stop), 'tool_use' (wants to call a tool). The primary signal for controlling agentic loops.
A pattern that dynamically retrieves relevant information from an external knowledge base and injects it into the context window based on the current query. Allows Claude to reason over large document sets without fitting everything in context at once.
Related Concepts
Trace Analysis for Multi-Step Failures
Trace analysis walks logs of every call, tool invocation, retrieval hit, and result to find the earliest deviation
Evaluation Metrics: Accuracy, Latency, Cost, Safety, Security
A production eval measures five dimensions: accuracy, latency, cost, safety, and security