Held-Out Eval Set
PatternsDefinition
A representative set of test cases -- including edge cases and known failure modes -- that the system design has not been tuned against, used to give an honest read of quality. An eval built only from examples the design was tuned on only measures fit to those examples, not performance on new input; automating the eval to run on every prompt change and model-version bump is what actually catches regressions.
Why It Matters for the Claude Certification Exams
CCAR-P evaluation and CCDV-F eval domains treat a held-out set as the honest read of quality. Tuning prompts against the only test cases you have is the named failure mode.
In Depth
A held-out eval set is a representative collection of test cases — including edge cases and known failure modes — that you did not tune the system against. After you change a prompt, a tool schema, a model tier, or a retrieval blurb, you run the held-out set and read the score. If the only examples you ever look at are the ones you just debugged, you will overfit and ship a regression you cannot see.
What belongs in the set
- Production-like inputs, not toy sentences
- Known failure modes (missing fields, conflicting sources, jailbreak-ish user text, empty tool results)
- Edge cases the last incident actually hit
- Labels or rubrics you can score: expected JSON keys, citation presence, "must not mention X," severity class
Redact personal data. An eval warehouse of raw customer tickets is a GDPR problem and a leak surface; see GDPR.
How Claude teams go wrong
The loop looks productive: fail a case, patch the prompt, the case passes, ship. That is a training set, not a held-out set. Keep a frozen split. When a new failure appears in production, add a copy to the development set for debugging, and periodically add a new held-out slice so the frozen set does not go stale — but do not edit the frozen cases to match the new prompt.
Automate the scorer where you can (schema validation, exact match, LLM-as-judge with a rubric). Humans still review a sample. Developer exam items care about this as SDLC: evals are part of the change, not a demo after the fact. Professional items care about it as a gate: you do not raise an SLA or switch model tiers without a held-out read.
Pair with Validation loop (runtime retry) vs held-out (offline measurement). They are not substitutes.
Frequently Asked Questions
Can I use the same 20 examples I used to write the prompt as my eval set?
Not as the held-out set. Those examples are contaminated. Keep them for development, and measure ship/no-ship on cases you did not just fit.
Should a held-out set include real customer data?
Use production-like shape, not raw PII. Redact or synthesize. The set needs realistic difficulty, not identifiable people.