Domain 2: Output Evaluation and Validation
21% of examEvaluate output accuracy and completeness
Key Points
- Decide what 'good' looks like before reading the draft -- explicit, checkable success criteria (word limit, required citation, prohibited content) replace 'does this look okay?'
- Five-part checklist: accuracy, completeness, relevance, consistency, fitness for audience.
- Completeness means every requested part is actually present, not that the answer feels thorough.
- Relevance means answering the actual question asked, not a fluent-sounding nearby one.
- Each dimension can fail independently -- accurate facts don't guarantee completeness, consistency, or audience fit.
- Polish and confident phrasing are never evidence of correctness.
Decision Rules
When: You're about to evaluate an output
→Write the success criteria first, before reading the draft, so the draft can't shape the standard.
When: A multi-part request comes back as a confident, well-organized answer
→Check each requested part against the original ask, not against how complete it feels.
When: An answer reads fluently and on-topic
→Still verify it answers the actual question and check all five checklist dimensions.
✗ Anti-Patterns to Reject
- Treating a confident, well-written answer as correct because it sounds right.
- Checking only that an output sounds complete instead of confirming every requested part is present.
Recognize hallucinations, inconsistencies, and bias
Key Points
- Hallucination: confident, plausible-looking content that is false or fabricated -- invented stats, citations, sources, or quotes.
- Hallucinations concentrate in specific-looking details (numbers, dates, citations, URLs), at the edge of the model's knowledge, and inside long outputs.
- A hallucination reads exactly as confident as correct content -- tone is never a signal.
- Inconsistency is an internal contradiction (a total that doesn't match its line items); bias is skewed framing, unrepresentative examples, or assumptions about people.
- Four reduction techniques: ground in provided source, allow an 'I don't know' exit, request traceable citations/quotes, cross-check against an authoritative source.
- Self-reported model confidence is not a reliable accuracy signal, and grounding/RAG reduces but never eliminates hallucination risk.
Decision Rules
When: An output contains a specific-looking statistic, date, or citation
→Treat it as the highest-risk spot for fabrication and verify it, regardless of tone.
When: You want to reduce hallucination on a source-grounded task
→Ground the answer in the provided material and explicitly allow an 'I don't know' exit.
When: Claude states how confident it is in an answer
→Ignore that self-report as an accuracy signal -- confidence is not evidence.
When: A total doesn't match its line items or a recommendation conflicts with an earlier statement
→Recognize it as an inconsistency, not a hallucination, and fix accordingly.
✗ Anti-Patterns to Reject
- Believing a fabricated citation is a rare edge case rather than exactly where hallucinations concentrate.
- Assuming grounding or RAG eliminates hallucination entirely instead of merely reducing the risk.
Apply fact-checking and validation techniques scaled to stakes
Key Points
- Validation is a deliberate confirmation step, not a byproduct of polish -- reformatting an answer is not validating it.
- Match the technique to the claim: verify names/numbers/dates/citations against an authoritative source; open the actual source text for a cited regulation or policy; recompute numeric totals; confirm a summary adds nothing beyond its source; require human review for customer/legal/compliance-facing output.
- Canonical exam scenario: Claude cites a regulation subsection -- verify it against the official text before sharing, regardless of Claude's confidence.
- 'It's just internal' is not a reason to skip verification -- stakes, not audience size, determine diligence.
Decision Rules
When: Claude cites a specific regulation, statute, or policy subsection
→Open the actual source text and confirm the citation and wording before sharing it -- never send on confidence alone.
When: An output contains numeric analysis
→Recompute or spot-check totals against the source data.
When: An output summarizes a document you provided
→Confirm the summary reflects the source and adds nothing not in it.
When: Output is customer-, legal-, or compliance-facing
→Route it through human review before it leaves the building.
✗ Anti-Patterns to Reject
- Substituting reformatting (making it sound more formal) for actual validation.
- Skipping verification because 'the analysis is just internal.'
Determine when human review is required
Key Points
- Escalate to human review when output is high-stakes (legal/financial/medical/compliance/safety/reputational), will be published or sent externally without further checks, makes unverifiable factual claims, involves regulated/sensitive data or a policy decision, or exceeds Associate scope.
- Human-in-the-loop is a feature of responsible use, not a sign the tool failed.
- The tested skill is calibration: escalating everything wastes the tool's value; escalating nothing ignores risk.
- A team-lunch brainstorm and a customer-facing legal notice do not warrant the same review intensity.
Decision Rules
When: Output is high-stakes, external-facing, or touches regulated data
→Require human review before it's used or shared.
When: A task is technical or exceeds Associate scope (system design, API/agent work)
→Escalate to an Architect/Developer rather than attempting it.
When: Deciding review intensity for a low-stakes internal output
→Match the review level to actual stakes, not a blanket policy of always or never.
✗ Anti-Patterns to Reject
- Thinking automation means removing humans everywhere, even for high-stakes outputs.
- Escalating everything (wastes the tool's value) or nothing (ignores risk).
Edit, adapt, refine, and compare Claude's output
Key Points
- Treat Claude's first response as a draft, never a finished deliverable.
- Four standard moves: edit (correctness), adapt (reframe for a different audience), refine (tone/structure), compare (generate alternatives and pick the strongest).
- Adapting changes presentation for the reader (executive vs. engineer, Slack vs. client email) without changing the underlying facts.
- Requesting two or three variations and picking the best is a standard, expected technique, not an extra step.
- These moves come after evaluation (accuracy/hallucination checks), turning a validated draft into a fit-for-purpose deliverable.
Decision Rules
When: The same content needs to reach both engineers and executives
→Adapt the presentation for each audience rather than sending one version to both.
When: You're unsure which framing works best
→Ask for two or three variations and select the strongest instead of accepting the first draft.
When: An output is correct but reads awkwardly
→Refine tone/structure without changing the underlying facts or re-verifying already-validated content.
✗ Anti-Patterns to Reject
- Treating the first response as a finished deliverable rather than a draft.
- Manually rewriting from scratch instead of asking Claude to adapt or generate a variation.
Organize information and choose the right output format
Key Points
- Three surfaces: inline (short, conversational), Artifact (separate editable window for substantial, iterate-and-share deliverables), structured data/tables/JSON (feeds a spreadsheet, form, or downstream system).
- The right surface matches the deliverable's lifecycle -- one-off read, ongoing revision, or downstream ingestion.
- Structured/JSON output that parses correctly is well-formed, not necessarily correct -- content still needs validation.
- Canonical exam case: a one-page proposal refined over several rounds and shared with a client belongs in an Artifact.
- Forcing every output into one format (always inline, always structured) is the most common format-selection mistake.
Decision Rules
When: The deliverable is a polished document that will be iterated on and shared with a client
→Use an Artifact, not a long inline chat message.
When: Output feeds a spreadsheet, form, or another system
→Use structured data (table/JSON) with a predictable shape -- then still validate its content.
When: The answer is a quick, two-line clarification
→Keep it inline; wrapping it in an Artifact adds friction without benefit.
✗ Anti-Patterns to Reject
- Assuming structured/JSON output is automatically correct because it's well-formed.
- Defaulting to one format (always inline or always structured) regardless of how the deliverable will be used.