Evaluation Metrics: Accuracy, Latency, Cost, Safety, Security

Core

Define evaluation metrics across accuracy, latency, cost, safety, and security · Difficulty 2/5

evaluationmetricssuccess-criteriaaccuracysafety

Explanation

You can't improve what you don't measure. Before building an eval, define success criteria across every dimension the business cares about -- not just whether the output is correct.

The Five Metric Categories

Metric	What it captures
Accuracy / quality	Correctness against a reference or rubric (exact match, semantic match, or graded score)
Latency	Time-to-first-token and end-to-end response time vs. SLA
Cost	Tokens and dollars per request / per task
Safety	Rate of harmful, policy-violating, or unsafe outputs
Security	Resistance to prompt injection / jailbreak, data-leakage rate

Writing Good Success Criteria

Good criteria are specific, measurable, and tied to the use case:

"95% of extracted fields match the reference on the held-out set" -- specific and measurable
"The model should be accurate" -- not a usable success criterion

Metrics should reflect the Business Value Pillars established during solution design, not be chosen in isolation from what the business actually needs the system to do.

Common exam traps

Measuring only accuracy. A production eval also tracks latency, cost, safety, and security -- a design that's accurate but too slow, too expensive, or unsafe still fails its requirement. The exam will present a scenario where accuracy looks fine but the system fails on one of the other four dimensions; the correct diagnosis is that the eval was incomplete, not that accuracy measurement failed.

Key Takeaways

A production eval measures five dimensions: accuracy, latency, cost, safety, and security
Good success criteria are specific, measurable, and tied to the use case
Eval metrics must trace back to the business value pillars defined during solution design
An accurate-but-slow, -expensive, or -unsafe design still fails its requirement
Define success criteria before building the eval, not after

Glossary Terms

Business Value Pillars

The five categories -- efficiency, transformation, productivity, cost, and performance/SLAs -- that an architect maps every architectural tradeoff to when justifying a design decision in the language of the business. A decision the architect cannot trace back to one of these pillars is not yet justified.

Related Concepts

Eval Datasets & Scoring Methodologies

An eval = a representative test set + a scoring method + a metric, run repeatably

Verifying Optimizations Against the Eval

Re-run the eval after every optimization before adopting it