Verifying Optimizations Against the Eval

Advanced

Optimize cost, latency, and token usage while verifying against the eval · Difficulty 3/5

cost-optimizationregressionevaluationverification

Prerequisites

Cost, Latency & Token Optimization Techniques

Explanation

Every cost or latency change is a hypothesis about the system, not a guaranteed improvement. It must be verified against the eval so a cost win doesn't silently cost accuracy.

The Verification Rule

Before adopting any optimization -- a cheaper model, a shorter context, a lower max_tokens cap -- re-run the eval built in Domain 4b and confirm the metric bar is still met. If it isn't, the change is a regression, not an optimization, regardless of how much cost or latency it saved.

Which Optimizations Are Free, and Which Are Not

Not all five levers carry the same accuracy risk:

Prompt caching with stable content ordered first reduces cost and latency with no accuracy loss -- it reuses the exact same content, just cheaper to reprocess.
Truncating a policy document, dropping retrieval entirely, or lowering max_tokens below the needed length all trade away content the system actually needs -- these are not equivalent to caching, even though they also reduce cost.

Common exam traps

Optimizing cost/latency without re-running the eval. A cheaper model or shorter context that quietly drops accuracy below the bar is a regression, not an optimization. The correct workflow is always: apply the change, re-run the eval, then adopt only if the metric bar still holds -- never "cheaper is always better" and never compensating for a quality drop by adjusting an unrelated parameter like temperature.

Key Takeaways

Re-run the eval after every optimization before adopting it
A cheaper model or trimmed context that drops accuracy below the bar is a regression, not a win
Prompt caching with stable content ordered first is unique in costing zero accuracy
Truncating content, dropping retrieval, or under-sizing max_tokens are false economies that trade away quality
"Cheaper is always better" without re-running the eval is the core exam trap for this task statement

Glossary Terms

Accuracy-Latency-Cost Tradeoff

The recurring architectural tension where an integration decision that improves accuracy (e.g., reranking, retrieving more chunks) typically adds latency and cost, and vice versa. The architect's job is not to eliminate the tradeoff but to make it explicit and justify the chosen configuration against whichever constraint the stated requirement names as dominant. Prompt caching a stable repeated context is the rare exception that improves cost and latency with no accuracy loss.

Related Concepts

Evaluation Metrics: Accuracy, Latency, Cost, Safety, Security

A production eval measures five dimensions: accuracy, latency, cost, safety, and security

A/B Testing & the Iterate Loop

A/B test one variable at a time on the same dataset against defined metrics (ablation mindset)