Defensive Parsing, Response Validation & Skepticism Toward Confident Output
CoreHandle Claude's output defensively · Difficulty 2/5
Explanation
Response Validation
Validate output against the schema -- types, required fields, allowed values -- before trusting it. Well-formed JSON can still be semantically wrong: structure and correctness are two different checks, and passing the first says nothing about the second.
Defensive Parsing
Assume the output may be imperfect. Tolerate extra prose around the structured payload, handle truncation (stop_reason: max_tokens), and on failure retry, ask the model to repair its own output, or fall back to a safe default -- don't let a parse error crash the application.
Skepticism Toward Confident Output
Claude can be confidently wrong; polish and certainty in the phrasing are not evidence of correctness. For high-stakes claims, ground the answer in sources, verify it independently, and keep a human in the loop rather than acting on the model's word alone.
Common exam traps
- Trusting output because it "looks right" or sounds confident -- validate content, not just format.
- Parsing the model's text with brittle string operations and no error handling; use schema validation plus defensive parsing instead of assuming the output will always match expectations.
Key Takeaways
- Validate structure AND semantics -- well-formed JSON can still contain a clearly wrong value
- Defensive parsing tolerates extra prose and truncation, and retries, repairs, or falls back instead of crashing
- Confident, polished output is not evidence of correctness -- stay skeptical of high-stakes claims
- Brittle string parsing with no error handling is a common exam trap; use schema validation instead
Glossary Terms
Confident, plausible-looking content that Claude generates which is false or fabricated -- an invented statistic, citation, source, or quote. Delivered with the same fluent tone as accurate content, which is what makes it hard to detect by tone alone. Concentrates in specific-looking details, at the edge of the model's knowledge, and inside long outputs.
Guaranteed formatted output (typically JSON) from Claude. The most reliable method is to define a schema as a tool and set tool_choice to force its use — Claude's tool_use blocks are always valid JSON. Alternatively, use --output-format json with --json-schema in Claude Code CLI.
A pattern where structured output that fails validation is sent back to Claude with the specific error, requesting a correction. More effective than silent retries because Claude uses the error feedback to understand and fix the problem. Typically capped at 2-3 retries.
Related Concepts