Output Evaluation and Validation

Knowledge of

• Defining success criteria up front -- what 'good' looks like -- before judging an output, per Anthropic's success-criteria guidance
• The practical checklist: accuracy, completeness, relevance, consistency, and fitness for audience
• Why polish and confident phrasing are not evidence of correctness
• The difference between an output that merely sounds complete and one that actually addresses every requested part
• Discernment as the AI Fluency competency for judging output against what was asked for, what the sources actually say, and the standards of your field -- and how it differs from Diligence

Skills in

• Writing explicit, checkable success criteria (e.g., word limits, required citations, prohibited recommendations) before evaluating an output
• Checking an output against each practical-checklist dimension rather than forming a single gut impression
• Confirming that every discrete part of a multi-part request was addressed, not just the most prominent one
• Resisting the pull of confident, well-written prose as a substitute for verification
• Recognizing whether an exam scenario is testing the skill of reviewing (Discernment) or the discipline of deciding to review at all (Diligence)

Knowledge of

• Hallucination as confident, plausible-looking, but false or fabricated content -- invented statistics, citations, sources, or quotes
• The three places hallucinations concentrate: specific-looking details, the edge of the model's knowledge, and long outputs where one fabrication hides among correct content
• Inconsistency (internal contradiction) and bias (skewed framing, unrepresentative examples, assumptions about people) as distinct categories of 'unexpected' output
• Capability hallucination -- Claude claiming to have taken an external action (sent an email, saved a file) that it has no tool to actually perform
• Grounding, the 'I don't know' exit, requesting traceable citations/quotes, and cross-checking as the techniques that reduce -- not eliminate -- hallucination risk
• Why self-reported model confidence is not a reliable accuracy signal, and why grounding/RAG reduces but does not eliminate hallucination risk

Skills in

• Spotting specific-looking numbers, dates, citations, and quotes as the highest-risk spots for fabrication
• Distinguishing a hallucination from an internal inconsistency from a bias issue so the correct fix follows
• Recognizing a claimed external action (an email 'sent,' a file 'saved') as unverified until independently confirmed, since claude.ai has no tool to perform it
• Prompting Claude to ground answers in provided source material and to say 'I don't know' rather than guess
• Requesting traceable citations/quotes and cross-checking specific claims against an authoritative source
• Not treating a model's stated confidence, or the presence of grounding/RAG, as proof of accuracy

Knowledge of

• Validation as the deliberate step of confirming an output before it is used, distinct from making it look more polished
• The stakes-scaled validation approaches: verifying names/numbers/dates/citations, opening source text for cited regulation/policy, recomputing numeric totals, confirming a summary adds nothing beyond its source, and human review for customer/legal/compliance-facing output
• Quote-grounding (asking Claude to extract supporting quotes before drawing conclusions) and best-of-N comparison (re-running a request and comparing outputs) as additional grounding and validation techniques
• The exam's canonical case: a cited regulation subsection must be verified against the official text before it is shared

Skills in

• Selecting a validation technique that matches the type of claim (citation, number, summary) and how the output will be used
• Verifying a cited subsection, statute, or policy against the actual source text rather than trusting Claude's citation
• Recomputing or spot-checking numeric analysis against source data
• Asking Claude to quote supporting passages before analyzing a long document, so reasoning and errors stay visible
• Re-running a request and comparing outputs (best-of-N) to build confidence where runs agree and flag soft spots where they diverge
• Refusing to substitute reformatting, or 'it's just internal,' reasoning for genuine verification

Knowledge of

• The conditions that require human review: high-stakes (legal/financial/medical/compliance/safety/reputational) content, external publication without further checks, unverifiable factual claims, regulated or sensitive data or policy decisions, and tasks beyond Associate scope
• Human-in-the-loop as a feature of responsible use, not a failure of the tool or a sign Claude is inadequate
• The judgment skill of matching review level to actual stakes, rather than escalating everything or nothing

Skills in

• Screening an output against the human-review trigger conditions before it is used or shared
• Escalating technical or out-of-scope tasks to an Architect/Developer rather than attempting them beyond Associate scope
• Calibrating review intensity to stakes instead of defaulting to blanket escalation or blanket automation

Knowledge of

• Raw Claude output as a draft, not a finished deliverable
• The four standard moves: editing for correctness, adapting for a different audience, refining tone/structure, and comparing alternative versions
• Requesting multiple variations or an audience-specific rewrite as standard techniques for improving a draft
• Code execution as the mechanism that produces a genuinely verified, computed numeric result -- distinct from a plausible-looking prose figure

Skills in

• Editing an output for correctness before treating it as final
• Reframing the same content for a different audience (executives vs. engineers, a Slack message vs. a formal client email)
• Asking Claude for two or three variations and selecting the strongest rather than accepting the first draft
• Requesting a rewrite for a different reader instead of manually rewriting from scratch
• Asking Claude to compute a number via code execution rather than write it in prose, when the number must be right
• Recognizing that a deterministic, re-runnable calculation still depends on correct code, so the result should be checkable, not blindly trusted

Knowledge of

• Inline responses as the fit for short answers, quick edits, and conversational back-and-forth
• Artifacts as a separate, editable window suited to substantial, self-contained deliverables that will be refined, reused, or shared
• Structured data (tables, JSON) as the fit when output feeds a spreadsheet, form, or downstream system and needs a predictable shape
• Structured/well-formed output still needing content validation -- valid shape is not valid content

Skills in

• Choosing inline, Artifact, or structured output based on how the deliverable will actually be used, not by default
• Recognizing when a substantial, iterate-and-share deliverable belongs in an Artifact rather than buried in chat
• Validating the content of structured/JSON output rather than trusting it because it parses correctly
• Avoiding a one-format-fits-all habit across different output needs