Evals as Verification: Closing the Loop
CoreSelect recovery strategies and verify fixes with evals · Difficulty 2/5
Explanation
Why a Fix Needs an Eval, Not Just a Look
Applying a recovery strategy is not the end of the debugging process. To know whether a fix actually helped -- and to catch regressions on the next model version -- build evaluations with explicit success criteria:
- A representative test set covering the cases that matter, not just the one example that originally failed
- A scoring method: exact match, a graded rubric, or LLM-as-judge
- A target metric that defines what "good enough" means
Evals turn "seems better" into a measurable, repeatable check.
Why a Single Spot-Check Fails
Claude's output is non-deterministic. A fix that appears to work on one re-run of the originally failing example can still fail on the next slightly different input, or even on a re-run of the same input. A single manual check cannot distinguish "the fix worked" from "this particular run happened to come out fine." Only a test set scored against a metric provides that distinction, and only a repeatable eval catches a regression introduced by a future model version.
Common exam traps
- Treating "it looks fixed on one example" as verification. Non-deterministic output requires an eval set and a metric, not a single manual spot-check -- one clean run proves nothing about the general case.
- Skipping the eval step after a recovery strategy is applied, on the assumption that the strategy's logic alone (e.g., "we now ground with sources") guarantees the outcome improved -- the eval is what confirms it did.
Key Takeaways
- An eval needs a representative test set, a scoring method (exact match, rubric, or LLM-as-judge), and a target metric
- Evals turn 'seems better' into a measurable, repeatable check
- Non-deterministic output means a single manual spot-check cannot verify a fix
- Evals also catch regressions introduced by a future model version, not just the original bug
Glossary Terms
Comparing competing prompts, models, retrieval configurations, or parameters on the same dataset against the same defined metrics, changing exactly one variable at a time (an ablation mindset) so the effect can be attributed to that variable, then adopting or reverting based on the measured result rather than intuition.
A representative set of test cases -- including edge cases and known failure modes -- that the system design has not been tuned against, used to give an honest read of quality. An eval built only from examples the design was tuned on only measures fit to those examples, not performance on new input; automating the eval to run on every prompt change and model-version bump is what actually catches regressions.
Related Concepts