Eval, Testing, and Debugging

Knowledge of

• The five recognizable failure buckets: transport/HTTP, request error, parsing/validation, model-output error, and tool-loop error
• Which layer each failure bucket lives in -- the integration layer (code, request, parsing) versus the model output
• The signals that identify each bucket: status codes (429, 400, 401, 5xx), a code-level parsing exception, or well-formed-but-wrong content
• The core discipline of not fixing the prompt when the bug is in code, and not patching code when the model output is the actual problem

Skills in

• Reading a failure's signal (status code, exception type, or content quality) to classify it into the correct bucket
• Recognizing that a crash while reading Claude's response is a parsing bug in the consumer, not evidence the model is wrong
• Avoiding the reflex of tuning the prompt for a bug that is actually in the integration code, or vice versa

Knowledge of

• The reproduce-inspect-localize procedure: reproduce with the exact failing request, inspect the raw response before post-processing, then localize by status code and content quality
• The mapping from observation to layer: non-2xx status means integration/transport; 2xx but code throws means parsing/validation; 2xx and parses fine but content is wrong means model-output
• stop_reason as a diagnostic field, and specifically that stop_reason: max_tokens means the output was truncated
• That truncation is a common, easily misdiagnosed cause of 'invalid JSON' errors

Skills in

• Reproducing a failure with the exact same messages, params, tools, and model before attempting any fix
• Inspecting the raw response -- status code, stop_reason, usage, and full content blocks -- before any post-processing runs
• Localizing a failure to the correct layer using status code and content-quality observations
• Recognizing stop_reason: max_tokens as truncation and fixing it by raising max_tokens rather than blaming the schema

Knowledge of

• That a bad final answer in an agent or multi-step workflow is usually the end of a chain of steps, not an isolated event
• Trace analysis: logging every model call, tool call, arguments, tool result, and intermediate message to walk the sequence
• The principle of looking for the earliest deviation rather than debugging only the final output
• That traces can also expose context problems -- quality degrading late in a long session points to context bloat/drift, not a model defect

Skills in

• Logging request/response pairs, tool invocations, and token usage at each hop of a multi-step workflow
• Walking a trace to find the earliest step that deviated rather than fixating on the final failing output
• Distinguishing a genuine model or tool-choice error from a late-session quality drop caused by context bloat

Knowledge of

• Recovery strategies mapped to failure types: retry with backoff, fix-and-resend, defensive parsing/repair, raising max_tokens, grounding/constraining, and graceful degradation/human-in-the-loop
• That retrying an unchanged request is pointless for 400/401 errors, which need a fix, not a retry
• Evaluations as the mechanism that verifies a fix actually worked and catches regressions on the next model version
• The components of an eval: a representative test set, a scoring method (exact match, graded rubric, or LLM-as-judge), and a target metric
• That non-deterministic model output requires an eval set and metric, not a single manual spot-check, to confirm a fix

Skills in

• Matching retry-with-backoff to transient transport errors (429/529/5xx) rather than to errors that need a code or payload fix
• Applying defensive parsing and reprompt/repair strategies to malformed structured output
• Grounding and constraining prompts to address hallucinated content instead of merely retrying
• Choosing graceful degradation or human-in-the-loop escalation when automated recovery can't guarantee correctness
• Building an eval set with a scoring method and target metric to verify a fix and guard against regressions