Domain 4: Evaluation, Testing & Optimization
16% of examDefine evaluation metrics across accuracy, latency, cost, safety, and security
Key Points
- A production eval measures five dimensions: accuracy/quality, latency, cost, safety, and security.
- Good success criteria are specific, measurable, and tied to the use case ('95% of extracted fields match the reference on the held-out set'), not vague aspirations ('the model should be accurate').
- Eval metrics must trace back to the business value pillars defined during solution design.
- An accurate-but-slow, -expensive, or -unsafe design still fails its requirement.
- Define success criteria before building the eval, not after.
Decision Rules
When: Writing success criteria for an eval
→Make them specific and measurable, tied to the use case, not a vague aspiration.
When: Aggregate accuracy looks fine but the system fails elsewhere
→Check latency, cost, safety, and security -- the eval was likely incomplete, not accuracy measurement broken.
When: Choosing which metrics to track
→Trace them back to the business value pillars the solution was designed to serve.
✗ Anti-Patterns to Reject
- Measuring only accuracy and ignoring latency, cost, safety, and security.
- Writing aspirational, unmeasurable success criteria like 'the model should be accurate.'
Build evaluation datasets and test frameworks using mixed scoring methodologies
Key Points
- An eval = a representative test set + a scoring method + a metric, run repeatably.
- A usable dataset is representative (real usage, edge cases, known failure modes), large enough to avoid noise, and held out (not tuned on).
- Code/exact-match grading is fast and unambiguous but only fits structured/verifiable output.
- LLM-as-judge scales subjective grading but must itself be validated against human labels before you rely on it.
- Human evaluation is the gold standard, reserved for nuanced or high-stakes cases.
- Automate the eval to run on every prompt change and every model-version bump -- a single passing example is not an evaluation.
Decision Rules
When: Output is structured/verifiable
→Use code/exact-match grading for a fast, cheap, unambiguous check.
When: Output is open-ended or subjective and needs to scale
→Use LLM-as-judge, but validate the judge against human labels first.
When: A judgment is nuanced or high-stakes
→Reserve human evaluation for it.
When: A prompt changes or a model version bumps
→Re-run the automated eval, not a one-time spot-check.
✗ Anti-Patterns to Reject
- Treating one manual spot-check ('ran it once and read the answer') as a valid evaluation.
- Trusting an LLM-as-judge that hasn't been validated against human labels.
Run A/B tests and apply iterative improvement to drive design decisions
Key Points
- Improvement is empirical: change one variable, measure it against the eval, keep what wins.
- A/B test competing prompts, models, retrieval configs, or parameters on the same dataset against the same defined metrics.
- Ablation mindset: change one thing at a time so you can attribute the effect to that specific variable.
- Contextual retrieval is the canonical example: A/B embeddings-only vs. +BM25 vs. +reranking on retrieval-failure rate.
- Iteration is a loop: hypothesize, change, measure, adopt or revert.
Decision Rules
When: Testing a change to prompts, models, retrieval config, or parameters
→A/B it on the same dataset against defined metrics, changing exactly one variable.
When: Comparing retrieval configurations
→Measure retrieval-failure rate across embeddings-only vs. +BM25 vs. +reranking.
When: Deciding whether to ship a change
→Run the hypothesize-change-measure-adopt/revert loop rather than shipping on intuition.
✗ Anti-Patterns to Reject
- Switching multiple variables at once and eyeballing a few answers, which breaks attribution.
- Deciding on intuition rather than a measured A/B test against defined metrics.
Diagnose production system issues by localizing the failure layer
Key Points
- Localize the cause before fixing -- patching the wrong layer just moves the symptom.
- Confident-but-wrong after a document refresh (model/latency unchanged) points to retrieval/indexing.
- Well-formed but factually invented output is a hallucination, fixed with grounding/constraints/citations.
- A regression right after a model-version change is model mismatch -- re-run evals, pin/roll back.
- Truncated output with stop_reason: max_tokens is a token-limit issue, not a prompt or model problem.
- Trace analysis walks logs back to the earliest deviation, not just the final symptom.
Decision Rules
When: Answers are confident but wrong right after a document refresh, with model and latency unchanged
→Suspect retrieval/indexing, not the model.
When: Output is well-formed but factually invented
→Diagnose hallucination and fix with grounding/citations/constraints, not a version rollback.
When: Quality regresses right after a model-version bump
→Re-run evals and pin/roll back the version.
When: Output is truncated
→Check stop_reason: max_tokens and raise the limit -- don't assume a prompt or model defect.
✗ Anti-Patterns to Reject
- Blaming 'the model got worse' for what is actually a retrieval or context problem.
- Fixing only the final visible symptom instead of tracing back to the earliest deviation in a multi-step pipeline.
Optimize cost, latency, and token usage while verifying against the eval
Key Points
- Optimization is a tradeoff against measured quality -- never applied blind.
- Prompt caching cuts cost/latency on stable repeated prefixes with no accuracy loss -- order stable content first.
- Right-size the model: route each step to the cheapest tier that still passes the eval.
- Trimming context reduces both cost and context rot at the same time.
- The Batches API cuts cost for latency-tolerant bulk jobs but does not reduce per-request latency.
- Re-run the eval after every optimization before adopting it -- a cheaper model or trimmed context that drops accuracy below the bar is a regression, not a win.
Decision Rules
When: A workload is latency-tolerant and bulk
→Use the Batches API to cut cost -- but don't expect it to speed up a single latency-sensitive request.
When: Considering a cheaper model, shorter context, or lower max_tokens
→Apply the change, re-run the eval, and adopt only if the metric bar still holds.
When: A stable prefix repeats across requests
→Cache it -- this is the one lever that costs zero accuracy.
✗ Anti-Patterns to Reject
- Assuming 'cheaper is always better' without re-running the eval after the change.
- Truncating content, dropping retrieval, or under-sizing max_tokens as false economies that trade away quality.
Monitor production systems with logging, dashboards, and alerting
Key Points
- Evaluation doesn't stop at launch -- monitoring turns the offline eval into a continuous quality signal.
- Log request/response pairs, tool calls, retrieval results, stop_reason, and per-hop token usage for every production request.
- Dashboard latency, error rates, cost, and per-domain quality metrics.
- Alert on regressions: retrieval-failure spikes, SLA breaches, cost anomalies, eval-score drops after a change.
- Sample and re-score live traffic periodically to catch drift the offline eval didn't cover.
Decision Rules
When: A system goes to production
→Log request/response, tool calls, retrieval results, stop_reason, and token usage for every request.
When: Setting up alerts
→Alert on regressions (spikes, SLA breaches, cost anomalies, eval-score drops), not just absolute thresholds.
When: Real-world input distributions may shift over time
→Sample and re-score live traffic periodically rather than trusting the static offline eval alone.
✗ Anti-Patterns to Reject
- Treating monitoring as a terminal step instead of feeding it back into the next design/eval iteration.
- Relying solely on the offline eval and never re-scoring live traffic for drift.