Model-Version Pinning & Eval-Gated Upgrades
AdvancedSelect among Claude model tiers against quality/latency/cost tradeoffs · Difficulty 2/5
Explanation
Model Releases Can Change Behavior
A newer model release can change behavior in ways that break prompts carefully tuned for an older one -- even when the API contract itself is unchanged. Formatting habits, tone, and edge-case handling can all shift between versions.
Pin, Then Re-Run Evals Before Upgrading
The safe pattern for selecting or upgrading models is:
- Pin the model version used in production rather than floating to "latest" and inheriting behavior changes automatically.
- Re-run evaluations against the pinned prompt/model pair before adopting a new release.
- Promote to the new version deliberately, once evals confirm the prompt still performs as expected.
Switching immediately to a new release to "get the improvements," or assuming newer always means strictly better for a specific prompt, skips the step that actually protects production quality.
Why This Belongs With Model Selection
Version pinning is inseparable from tier selection: choosing Haiku vs. Sonnet vs. Opus for a step is a decision made *against a specific, evaluated version* of that tier. An upgrade is effectively a new model choice and deserves the same eval-gated scrutiny as picking a tier in the first place.
Common exam traps
- "Switch immediately to the new model to get the improvements." The correct action is to re-run evals and pin the version, since behavior can change in ways that regress a specific prompt.
- Assuming only the API key or model-name string needs to change on an upgrade, when the actual output behavior against your prompts is what needs re-validating.
Key Takeaways
- A newer model release can change behavior in ways that break prompts tuned for an older version, even with an unchanged API contract
- Pin the production model version rather than floating to 'latest'
- Re-run evaluations against the pinned prompt/model pair before adopting a new release
- Model upgrades deserve the same eval-gated scrutiny as an initial model choice
Glossary Terms
Comparing competing prompts, models, retrieval configurations, or parameters on the same dataset against the same defined metrics, changing exactly one variable at a time (an ablation mindset) so the effect can be attributed to that variable, then adopting or reverting based on the measured result rather than intuition.
A representative set of test cases -- including edge cases and known failure modes -- that the system design has not been tuned against, used to give an honest read of quality. An eval built only from examples the design was tuned on only measures fit to those examples, not performance on new input; automating the eval to run on every prompt change and model-version bump is what actually catches regressions.
Related Concepts