4.1 Evaluation Metrics, Datasets & Test Frameworks
4.1.1 You Can't Improve What You Don't Measure
Picture two teams shipping the same customer-support agent. Team A eyeballs a handful of transcripts before every release, nods, and ships. Team B has a scored test set of 200 real support tickets, and every release has to clear a bar on that set before it goes out. Six months later, Team A is fighting fires they can't explain — a prompt tweak that "felt fine" quietly broke a whole category of refund requests, and nobody noticed until customers complained. Team B caught the regression the moment it happened, because their test set flagged it automatically.
That's the entire argument for Domain 4 in one story. LLM systems are non-deterministic — the same prompt can produce a slightly different answer twice in a row — so an architect cannot rely on spot-checks the way you might for a deterministic function. Quality has to be measured against defined metrics, on a representative dataset, with a framework you can run again and again. Evaluation, in this sense, is not a QA afterthought bolted onto the end of a project; it's the feedback loop that makes every other domain honest. Domain 1 lets you build agentic loops and orchestration; Domain 4 is how you prove those loops actually work, and keep proving it every time something changes.
Task Statement 4.1 starts at the very beginning of that feedback loop: before you build an eval, you have to decide what "good" even means for this system. That sounds obvious, but it's the step most teams skip — they jump straight to "let's write some test prompts" without first agreeing on what a passing score looks like, or on which dimensions of quality actually matter for this use case. Skip this step and you'll build an eval that measures something, just maybe not the thing that determines whether the system should ship.
A one-off spot-check tells you almost nothing about a non-deterministic system; a repeatable, scored evaluation is what actually catches regressions before customers do.
The one idea to hold onto
Because LLM output is non-deterministic, quality must be measured against defined metrics on a representative dataset with a repeatable framework — a manual spot-check is not a substitute for an evaluation, no matter how good the answer looked.
4.1.2 The Five Dimensions of a Production Eval
Ask an engineer "is this agent good?" and the first word out of their mouth is almost always "accurate." That instinct isn't wrong, but it's incomplete — and the exam leans on that incompleteness constantly. A production system can be perfectly accurate and still fail its requirement, if it's also too slow, too expensive, unsafe, or insecure. An architect defines success criteria across FIVE dimensions, not one.
| Metric | What it captures |
|---|---|
| Accuracy / quality | Correctness against a reference or rubric (exact match, semantic match, or graded score) |
| Latency | Time-to-first-token and end-to-end response time vs. an SLA |
| Cost | Tokens and dollars per request or per completed task |
| Safety | Rate of harmful, policy-violating, or unsafe outputs |
| Security | Resistance to prompt injection / jailbreak attempts and data-leakage rate |
The five metric categories a production eval should cover. Accuracy is necessary but never sufficient on its own.
Notice how each one maps to a different way a real deployment can fail even when the model "gets the answer right." A support agent that answers every ticket correctly but takes nine seconds to respond violates its latency SLA. A document-summarization pipeline that's accurate but burns $4 of tokens per page will bankrupt the business at scale. A perfectly fluent agent that can be talked into revealing another customer's order history has failed on security even though its prose reads beautifully. None of these are accuracy problems — and an eval that only checks accuracy will wave every one of them through.
These five dimensions aren't arbitrary — they should trace directly back to the business value pillars you established when you designed the solution (Domain 1 territory). If the business case for this agent depended on cutting support cost per ticket, then cost per ticket belongs in the eval as a first-class metric, not an afterthought you check manually once a quarter. Metrics chosen in isolation from what the business actually needs are a common way for an eval to look rigorous while measuring the wrong thing.
Once you've picked your dimensions, the next skill is writing success criteria that are actually usable. "The model should be accurate" gives you nothing to test against — there's no way to fail that criterion, which means there's no way to pass it either. "95% of extracted fields match the reference on the held-out set" is specific, measurable, and tied to the use case: you can point at a number and say whether the bar was cleared.
4.1.2 — Exam Trap
The exam frequently presents a scenario where accuracy looks fine but the system fails on cost, latency, safety, or security — and tests whether you correctly diagnose the eval itself as incomplete, rather than concluding "accuracy measurement failed" or "the model needs to be smarter." Measuring only accuracy is the single most common trap in this task statement.
4.1.2 — Key Concept
Define success criteria across all five dimensions — accuracy, latency, cost, safety, security — BEFORE building the eval. Good criteria are specific, measurable, and tied to the use case ("95% match on the held-out set"), and they trace back to the business value pillars the solution was designed to deliver.
4.1.3 What an Evaluation Actually Is
With the metrics decided, the next question is how you actually measure them in a repeatable way. Here's the compact definition that the rest of this lesson (and much of the exam) builds on: an evaluation is a REPRESENTATIVE TEST SET, scored by a DEFINED METHOD, against a METRIC, run REPEATABLY. Every word in that sentence is load-bearing, and each one rules out a shortcut someone will inevitably try to take.
"Representative" rules out testing only the easy, happy-path cases you thought of first. A usable eval dataset is drawn from real usage and deliberately includes edge cases and known failure modes — the weird ticket, the malformed input, the customer who asks two unrelated questions in one message. It also needs to be large enough that a single lucky or unlucky run doesn't swing the score — results have to be more signal than noise. And critically, it should include a HELD-OUT set: examples the design hasn't been tuned on. Without a held-out set, your eval only tells you how well the system fits the examples you already optimized against — which flatters you, but doesn't predict how the system handles input it hasn't seen.
An evaluation is these three pieces working together, run repeatably: a representative held-out test set, a defined scoring method, and a metric drawn from the five dimensions in 4.1.2.
4.1.3 — Key Concept
An evaluation = a representative, held-out test set + a defined scoring method + a metric, run repeatably. A held-out set the design hasn't been tuned on is required for an honest read of quality — without it you're only measuring fit to your own tuning examples.
4.1.4 Choosing How to Score: Code, Judge, or Human
Once the dataset exists, you need a way to grade each example's output. There are three scoring methodologies, and the skill the exam tests is knowing when to reach for which one — and, just as often, knowing when to combine them.
| Method | Best for | Tradeoff |
|---|---|---|
| Code / exact-match grading | Structured or verifiable output (JSON fields, extracted numbers, classifications) | Fast, cheap, unambiguous — but only works where correctness is deterministically checkable |
| LLM-as-judge | Open-ended, subjective quality at scale (tone, helpfulness, coherence) | Scales well, but the judge itself must be validated against human labels before you trust it |
| Human evaluation | Nuanced or high-stakes judgments automated methods can't reliably make | The gold standard for quality — but expensive, so reserve it for what the other two can't judge |
Three scoring methodologies. Most production evals mix all three: deterministic checks where possible, an LLM judge for scale, and periodic human review to validate the judge and catch what it misses.
Code grading is the easiest to trust and the cheapest to run, but it only applies where "correct" has a crisp, checkable definition — did the extracted date match the reference date, exactly? For genuinely open-ended output — is this customer email empathetic and on-brand? — there's no exact string to compare against, so teams turn to LLM-as-judge: a model reads the output against a rubric and assigns a score, the same way a human reviewer would, but at a scale a human panel never could.
Here is the trap the exam sets, and it's a subtle one because LLM-as-judge sounds so convenient it's tempting to treat it as a free pass: the judge is itself a model, and models can be wrong, biased toward certain phrasing, or simply miscalibrated. Before you rely on an LLM judge's scores, you validate the judge against a set of human labels — confirm that when a human says an answer is good, the judge agrees, and vice versa. Skipping that validation step means you're trusting an unverified grader to make ship/no-ship decisions, which defeats the entire point of building a rigorous eval in the first place.
# A minimal mixed-methodology eval harness
results = []
for example in held_out_set:
output = run_agent(example.input)
if example.type == "structured":
# Code grading: deterministic, cheap, unambiguous
score = 1.0 if output == example.reference else 0.0
else:
# LLM-as-judge: scalable, but the judge was separately
# validated against a sample of human-labeled examples first
score = llm_judge(output, rubric=example.rubric)
results.append(score)
pass_rate = sum(results) / len(results)
assert pass_rate >= 0.95, "Regression: below the defined success bar"4.1.4 — Exam Trap
Trusting LLM-as-judge blindly is a named exam trap. The judge needs its own validation against human labels before you rely on it — an unvalidated judge can confidently misgrade output, which defeats the purpose of automating the eval. Watch for scenarios that skip this validation step and present the judge's score as ground truth.
4.1.5 Automate It, or It Isn't an Eval
There's one more requirement hiding in the definition from 4.1.3: "run repeatably." An eval you run once, by hand, right before a big launch, is barely better than the spot-check from 4.1.1 — it tells you about that one moment and nothing about the next fifty prompt tweaks or the next model version bump. The fix is automation: wire the eval into your pipeline so it runs on every prompt change and every model-version bump, without anyone having to remember to trigger it.
This is what actually catches regressions. A prompt edit that reads perfectly reasonable in a code review can silently shift the model's behavior on a category of inputs the reviewer never tried by hand — the same way a one-character typo in application code can pass a casual read but fail a test suite. An automated eval running the full held-out set on every change is the difference between finding that regression in five minutes and finding it three weeks later in a customer complaint.
It's worth being precise about the single most common exam distractor here: "running the new prompt once and reading the answer" is described in almost every plausible wrong-answer option as if it were a legitimate check. It is a spot-check, not a valid evaluation. If a question's scenario describes exactly one example being tried and judged by eye, the correct answer is always the option describing a representative, scored, repeatable test set — never the one-off read, and never "ask the model if it's confident," which isn't measurement at all.
- •✗ Running the new prompt once and reading the answer — a spot-check, not an eval.
- •✗ Asking the model to self-report whether its answer is confident or correct — not measurement.
- •✗ Shipping and waiting for user complaints to surface problems — reactive, not evaluative.
- •✓ A representative held-out test set, scored by a defined method (code, validated LLM judge, or human), against a metric, run automatically on every change.
4.1.5 — Key Concept
Automate the eval to run on every prompt change and every model-version bump — this is what catches regressions that a one-time or manual check would miss. A single passing example is not an evaluation; you need a scored, repeatable test set.
Where this shows up on the exam
4.1 questions almost always give you a scenario and ask you to spot either (a) a missing metric dimension beyond accuracy, or (b) a spot-check disguised as a real evaluation, or (c) an unvalidated LLM judge being trusted blindly. Anchor on: five dimensions, held-out set, validated judge, automated and repeatable.
4.1.6 Reliability as a Testable Layer: Backoff, Fallback Chains & Circuit Breakers
The five dimensions in 4.1.2 describe a system operating under normal conditions. But production dependencies don't stay in normal conditions forever -- a provider rate-limits you, a retrieval backend times out, a downstream API has a rough ten minutes. RELIABILITY is what happens next, and it deserves to be treated as its own testable layer rather than an assumed side effect of otherwise-good engineering, because it cuts across several of the five dimensions at once: a retry storm against a failing dependency shows up as a cost spike, a full retry schedule shows up as a latency SLA breach, and an unguarded fallback path can quietly erode both accuracy and security. An eval built only from happy-path examples never exercises any of this, because a happy-path eval never simulates the dependency actually failing.
| Control | What it does | Where it lives (layer) | What breaks without it |
|---|---|---|---|
| Exponential backoff | Retries a transient error (5xx, 429 rate-limit) with increasing delay between attempts | Individual API-call layer -- right where the call is made | A momentary blip becomes a hard, user-visible failure; a provider's one-second hiccup looks identical to the provider being fully down |
| Fallback chains | Routes to an alternate model, provider, or cached response when the primary path fails | Service-boundary layer -- where your system talks to an external dependency | A single provider incident takes down an entire feature, even when a slower or slightly lower-quality alternate response was available |
| Circuit breakers | Stops sending requests to a failing dependency for a cooldown period, rather than retrying forever | Orchestration layer -- coordinating calls across the whole pipeline | Retries pile up against a dependency that's clearly down, multiplying latency and burning spend on calls that were never going to succeed, and can starve shared resources unrelated healthy requests also depend on |
Three named reliability controls, each solving a different problem at a different layer. Stacking the wrong one at the wrong layer either does nothing or makes an outage worse.
Picture a payment-status lookup service that calls a third-party card processor's API on every checkout confirmation. Under normal load it's fine. Then the processor starts intermittently returning 429s during its own traffic spike. With no reliability controls at all, every 429 is a failed checkout confirmation -- customers see failures exactly as noisy and unpredictable as the processor's own strain, and there's nothing your system does about it. Add exponential backoff alone and most transient 429s now succeed on the second or third attempt with a short increasing delay -- a real improvement. But if the processor goes fully down for eight minutes rather than just strained, every checkout arriving during that window still burns through its full retry schedule before eventually failing, multiplying end-to-end latency across the whole queue and multiplying the API calls you're billed for on requests that were never going to succeed. Add a circuit breaker at the orchestration layer -- after, say, six consecutive failures, the breaker trips open and the service stops calling the processor entirely for a 90-second cooldown, so no more retries pile up. Finally, add a fallback chain: while the breaker is open, route to a secondary payment processor, or queue the confirmation for asynchronous retry with a "payment received, confirming" state shown to the customer instead of an outright failure. Together, the three controls turn an eight-minute processor outage from a wave of failed checkouts into a brief, mostly invisible dip in confirmation speed.
# A minimal composition of all three controls, layered correctly
class CircuitBreaker:
def __init__(self, failure_threshold=6, cooldown_s=90):
self.failures = 0
self.opened_at = None
self.failure_threshold = failure_threshold
self.cooldown_s = cooldown_s
def is_open(self, now):
if self.opened_at is None:
return False
if now - self.opened_at > self.cooldown_s:
self.opened_at = None # cooldown elapsed; allow a trial request
self.failures = 0
return False
return True
def record_failure(self, now):
self.failures += 1
if self.failures >= self.failure_threshold:
self.opened_at = now
def call_with_backoff(primary_fn, fallback_fn, breaker, now_fn, max_attempts=3):
now = now_fn()
if breaker.is_open(now):
return fallback_fn() # orchestration layer: don't even try the primary
delay = 0.5
for attempt in range(max_attempts):
try:
return primary_fn() # individual API-call layer: backoff on transient errors
except TransientError:
time.sleep(delay)
delay *= 2
breaker.record_failure(now_fn())
return fallback_fn() # service-boundary layer: degrade, don't fail outright4.1.6 — Key Concept
Reliability is a distinct, testable layer covering three named controls: exponential backoff (individual API-call layer, retries transient errors with increasing delay), fallback chains (service-boundary layer, routes to an alternate model/provider/cached response), and circuit breakers (orchestration layer, stops calling a failing dependency for a cooldown period). Reliability failures show up as symptoms inside the existing five dimensions -- cost, latency, safety -- rather than as a metric of their own.
4.1.6 — Exam Trap
A naive, fixed-interval or immediate retry loop against a struggling dependency is not a reliability control -- it's a self-inflicted amplifier that, from the dependency's side, can look indistinguishable from a denial-of-service burst. A scenario describing repeated failed retries against a dependency that's fully down is asking for a circuit breaker (stop trying), not more aggressive backoff, which still eventually tries and still eventually fails. Watch for these being presented as interchangeable when the scenario names a specific layer.
4.1.7 ROI and Payback Period: Quantifying the Business Case
Domain 1's business value pillars (efficiency, transformation, productivity, cost, performance-SLAs) tell you WHICH kind of value a design decision is meant to serve. They don't tell you HOW MUCH value it actually delivers, or how long the system takes to pay for itself. That's a distinct, quantitative discipline -- and it belongs here in Domain 4 because, like every other metric in this certification, a return-on-investment claim has to be measured against a defined baseline, not asserted from intuition or a vendor's marketing deck.
- 1.Baseline the business-unit metric BEFORE deployment -- measure the real current-state cost or time, not an estimate of it
- 2.Predict, then MEASURE, the post-deployment state -- once the system is live, use the actual measured metric, not the number you hoped for going in
- 3.Subtract the running cost of the AI system -- the monthly cost of tokens, infrastructure, and any vendor fees the system itself incurs
- 4.Arrive at a payback period -- implementation cost divided by net monthly savings -- with a sensitivity analysis on the assumptions that number depends on
Here's a worked example. A support team currently spends 12 minutes of loaded agent time drafting each ticket reply by hand, at a loaded labor rate of $22/hour, across 500 tickets/week. They deploy an AI drafting assistant that produces a draft in about 3 minutes of agent review-and-send time per ticket -- but roughly 20% of drafts still need a full manual rewrite (an escalated or unusually sensitive ticket the model handles poorly), each of which still costs the original 12 minutes. The system costs $3,100/month to run, and the one-time implementation cost was $28,000.
| Step | Calculation | Result |
|---|---|---|
| 1. Baseline (measured, pre-deployment) | 500 tickets/wk x 12 min = 6,000 min/wk = 100 hrs/wk x 4 wks = 400 hrs/month x $22/hr | $8,800/month |
| 2. Post-deployment state (measured, not estimated) | (500 x 3 min) + (0.20 x 500 x 12 min) = 1,500 + 1,200 = 2,700 min/wk = 45 hrs/wk x 4 wks = 180 hrs/month x $22/hr | $3,960/month labor + $3,100/month run cost |
| 3. Net monthly savings | ($8,800 - $3,960) labor savings - $3,100 running cost | $1,740/month |
| 4. Payback period | $28,000 implementation cost / $1,740 net monthly savings | ~16.1 months |
A worked ROI/payback calculation. Note that the running cost of the AI system ($3,100/month) is subtracted before the payback period is calculated -- not treated as a one-time cost.
Sensitivity analysis is the step teams skip most often, and it's the one that turns a hopeful number into a defensible one. The 20% rewrite-rate assumption drives the whole result here. If the true rewrite rate turns out to be 35% instead of 20% once the system has been live for a full quarter, post-deployment labor cost rises to 500 x 3 min + 0.35 x 500 x 12 min = 3,600 min/wk = 60 hrs/wk x 4 x $22 = $5,280/month, net savings fall to ($8,800 - $5,280) - $3,100 = $420/month, and the payback period stretches from roughly 16 months to nearly 67 months -- a case that looked solidly positive at 20% barely breaks even at 35%. That's not a hypothetical edge case; rewrite rates are exactly the kind of number that's easy to underestimate before a system has handled real edge cases at scale, which is precisely why the framework calls for testing the assumption's sensitivity rather than reporting a single point estimate as if it were certain.
4.1.7 — Two Errors to Watch For
(1) Using an ESTIMATED, not measured, baseline. "We think drafting takes about 12 minutes" is a guess dressed up as a baseline -- the real number has to come from actually measuring the current-state process, or the entire ROI calculation is built on sand. (2) Assuming 100% automation when human review is still required. Dropping the rewrite-rate line entirely and claiming the full $8,800 -> near-zero swing overstates savings substantially; any compliance-, safety-, or quality-driven human-in-the-loop requirement reduces real labor savings below the naive full-automation number, and this is the single most common way an ROI case turns out to be wrong once it's measured in production.
4.1.7 — Key Concept
ROI/payback is a CALCULATION (how much value, how long to pay back), distinct from the business value pillars (a CLASSIFICATION of which value a decision serves). The four-step framework: measure the baseline before deployment, measure (not estimate) the post-deployment state, subtract the AI system's running cost, and compute payback period with sensitivity analysis on the driving assumptions.
Key Takeaways
- ✓A production eval measures FIVE dimensions — accuracy, latency, cost, safety, and security — not accuracy alone; a design that's accurate but too slow, expensive, unsafe, or insecure still fails its requirement.
- ✓Good success criteria are specific, measurable, and tied to the use case ("95% match on the held-out set"), and trace back to the business value pillars the solution was designed to deliver.
- ✓An evaluation = a REPRESENTATIVE, HELD-OUT test set + a defined scoring method + a metric, run REPEATABLY — every word rules out a shortcut.
- ✓Three scoring methodologies — code/exact-match, LLM-as-judge, human evaluation — each fit different tasks; most production evals mix all three.
- ✓LLM-as-judge must be VALIDATED against human labels before you trust it; an unvalidated judge can confidently misgrade output.
- ✓Automate the eval to run on every prompt change and model-version bump — this is what actually catches regressions.
- ✓A single passing example, a self-reported confidence check, or shipping-and-waiting-for-complaints are all spot-checks, not evaluations.
- ✓Reliability is a distinct, testable layer -- exponential backoff (API-call layer), fallback chains (service-boundary layer), and circuit breakers (orchestration layer) prevent cascading failure and wasted spend when a dependency fails.
- ✓ROI/payback period is a CALCULATION (how much value, how long to pay back) distinct from the business value pillars (a CLASSIFICATION of which value is served); the framework is baseline -> measure post-deployment -> subtract running cost -> payback period, with sensitivity analysis -- watch for estimated baselines and assumed 100% automation.
Check Your Understanding
Test what you learned in this lesson.
Q1.A team's RAG-based document extractor scores 98% accuracy on its test set, but the eval never measured anything else before launch. Three weeks after shipping, the team discovers the system costs $6 per document processed and takes 40 seconds per request against a 5-second SLA. What was wrong with the eval?
Q2.Before relying on an LLM-as-judge to score thousands of open-ended support-ticket responses, what must an architect do first?
Q3.A developer tests a new prompt by running it once against a tricky customer question, reads the reply, decides it looks good, and merges the change. Why doesn't this qualify as a valid evaluation?
Q4.Which practice ensures an evaluation actually catches regressions introduced by ongoing prompt edits and model-version upgrades, rather than only reflecting quality at one point in time?
Q5.A team's checkout-confirmation service calls a third-party payment processor. During a processor outage lasting several minutes, every incoming request retries with increasing delay before eventually failing, but requests keep queuing up and retrying against the dead dependency for the entire outage, multiplying latency and API spend. What control is missing?
Q6.A document-automation team claims their new AI system has a 4-month payback period, based on an estimated 40-minutes-per-document baseline ("we think it used to take about that long") and the assumption that the system fully replaces manual processing. What is wrong with this ROI claim?
Practice This Lesson