Held-Out Eval Set

Definition

A representative set of test cases -- including edge cases and known failure modes -- that the system design has not been tuned against, used to give an honest read of quality. An eval built only from examples the design was tuned on only measures fit to those examples, not performance on new input; automating the eval to run on every prompt change and model-version bump is what actually catches regressions.