Defining Success Criteria Before Evaluating Output

Why "Does This Look Okay?" Is Not an Evaluation

Claude produces fluent, confident text -- but fluency is not accuracy. Before an output is used, it must be judged against what the task actually required, not against a vague impression of whether it reads well. Anthropic's guidance on defining success criteria applies directly here: decide up front what "good" looks like, then check the output against that definition.

From Vague to Checkable

A vague standard ("does this look okay?") cannot be verified consistently -- two reviewers might disagree, or the same reviewer might miss something on a re-read. An explicit, checkable success criterion removes the ambiguity. For example:

• "Must cite the correct regulation subsection"
• "Must stay under 200 words"
• "Must not recommend a refund"

Each of these can be verified with a yes/no check. That is the point of writing success criteria before generating or reviewing output: it turns evaluation from a subjective impression into a checklist.

Set the Bar Before You Read the Output

The discipline is sequencing: decide the success criteria first, then read Claude's output against them. Deciding what "good" means only after seeing an appealing draft invites the draft to shape the standard, rather than the standard shaping the judgment of the draft.

Common exam traps

• Treating a confident, well-written answer as correct. Polish is not proof -- a fluent answer can still fail an explicit success criterion.
• Skipping the step of writing down success criteria and jumping straight to "does this look okay," which produces an unverifiable, inconsistent evaluation.