Evaluation, Testing & Optimization

Knowledge of

• The five metric categories a production eval should cover: accuracy/quality, latency, cost, safety, and security
• What each metric category captures -- accuracy against a reference or rubric, latency as time-to-first-token and end-to-end response time vs. SLA, cost in tokens and dollars per request or task, safety as the rate of harmful or policy-violating outputs, and security as resistance to prompt injection/jailbreak and data-leakage rate
• That good success criteria are specific, measurable, and tied to the use case rather than vague aspirations
• That eval metrics should reflect the business value pillars established during solution design

Skills in

• Writing success criteria before building the eval, e.g. "95% of extracted fields match the reference on the held-out set" instead of "the model should be accurate"
• Selecting a metric mix appropriate to the use case instead of defaulting to accuracy alone
• Tying eval metrics back to the business value dimensions the solution was designed to deliver
• Recognizing that a design which is accurate but too slow, too expensive, or unsafe still fails its requirement

Knowledge of

• That an eval is a test set + a scoring method + a metric, run repeatably -- not a single manual spot-check
• The properties of a representative dataset: drawn from real usage, including edge cases and known failure modes, large enough that results aren't noise, and including a held-out set the design hasn't been tuned on
• The three scoring methodologies -- code/exact-match grading, LLM-as-judge, and human evaluation -- and when to mix them to fit the task
• That LLM-as-judge scoring must itself be validated against human labels before it can be trusted
• That automating an eval to run on every prompt change and every model-version bump is what catches regressions

Skills in

• Assembling a representative, held-out test set that includes edge cases and known failure modes rather than only common-path examples
• Choosing code/exact-match grading for structured or verifiable output where a fast, cheap, unambiguous check is possible
• Using LLM-as-judge for scalable grading of open-ended or subjective output while validating the judge against human labels
• Reserving human evaluation for nuanced or high-stakes quality judgments that automated methods can't reliably make
• Automating the eval to run on every prompt change and model-version bump so regressions are caught before reaching users
• Recognizing that a single passing example is not a valid evaluation

Knowledge of

• That improvement is empirical: change one variable, measure it against the eval, keep what wins
• That A/B testing compares competing prompts, models, retrieval configs, or parameters on the same dataset against the same defined metrics
• The ablation mindset -- changing one thing at a time so the effect can be attributed
• Contextual retrieval as a canonical A/B testing example: measuring retrieval-failure rate across embeddings-only vs. embeddings+BM25 vs. +reranking
• The iterate loop: hypothesize, change, measure, adopt or revert

Skills in

• Designing A/B tests that compare exactly one changed variable across prompts, models, retrieval configuration, or parameters on the same dataset
• Attributing quality changes to a specific variable by holding everything else constant (an ablation mindset)
• Using retrieval-failure rate to A/B embeddings-only, embeddings+BM25, and +reranking configurations and choosing on evidence
• Running the hypothesize-change-measure-adopt/revert loop rather than shipping changes based on intuition
• Rejecting the approach of switching multiple variables at once and eyeballing a few answers

Knowledge of

• That when quality drops, the cause must be localized before fixing -- patching the wrong layer just moves the symptom
• The symptom-to-cause mapping for six common failure patterns: retrieval/indexing issues, hallucination, model mismatch, prompt failure, max_tokens truncation, and context bloat/drift
• Trace analysis: using logs of every call, tool invocation, retrieval hit, and result to find the earliest deviation, not just the final symptom
• That confident-but-wrong output after a document refresh (with model and latency unchanged) points to retrieval/indexing, while a regression right after a model-version change points to model mismatch

Skills in

• Matching an observed symptom (confident-but-wrong after a data refresh, well-formed but invented output, regression after a version bump, ignored instructions, truncated output, late-session quality drop) to its likely cause and first diagnostic move
• Using trace analysis across call/tool/retrieval logs to find the earliest deviation in a multi-step pipeline rather than fixing only the final visible symptom
• Distinguishing a hallucination (well-formed but factually invented content, fixed by grounding/constraints/citations) from a model-mismatch regression (fixed by re-running evals and pinning/rolling back the version)
• Recognizing stop_reason: max_tokens truncation as a token-limit issue rather than a prompt or model defect
• Avoiding the trap of blaming "the model got worse" when the actual cause is a retrieval or context problem

Knowledge of

• That optimization is a tradeoff against measured quality -- never applied blind
• Prompt caching as a way to cut latency and cost on repeated stable prefixes with no accuracy loss, when stable content is ordered first
• Model right-sizing: routing steps to the cheapest tier that still passes the eval, reserving Opus/extended thinking for where it's needed
• Context trimming: pruning stale tool output and over-fetched chunks to lower cost and reduce context rot
• That the Batches API is async and cheaper for latency-tolerant bulk jobs but does not reduce per-request latency
• That every optimization must be verified against the eval so a cost win doesn't silently cost accuracy

Skills in

• Ordering stable content first so prompt caching can cut cost and latency on repeated prefixes with no accuracy loss
• Routing pipeline steps to the cheapest model tier that still passes the eval instead of defaulting to the most capable model everywhere
• Trimming stale tool output and over-fetched context to reduce both cost and context rot
• Choosing the Batches API for latency-tolerant bulk jobs to cut cost, while recognizing it does not reduce per-request latency
• Capping max_tokens to bound output cost without truncating needed content
• Re-running the eval after every cost/latency optimization to confirm quality hasn't silently regressed

Knowledge of

• That evaluation continues after launch -- monitoring turns the offline eval into a continuous production quality signal
• What to log in production: request/response pairs, tool calls, retrieval results, stop_reason, and per-hop token usage
• What to dashboard: latency, error rates, cost, and per-domain quality metrics
• What to alert on: a spike in retrieval failures, a latency SLA breach, a cost anomaly, or a drop in eval score after a change
• That sampling and re-scoring live traffic periodically catches drift the offline eval didn't cover
• That monitoring feeds the next iteration of the lifecycle -- closing the loop back into design and eval

Skills in

• Logging request/response pairs, tool calls, retrieval results, stop_reason, and per-hop token usage for every production request
• Building dashboards for latency, error rates, cost, and per-domain quality metrics
• Configuring alerts for retrieval-failure spikes, SLA breaches, cost anomalies, and eval-score drops following a change
• Sampling and re-scoring live traffic periodically to catch drift that the offline eval didn't anticipate
• Feeding monitoring signals back into the next design/eval iteration rather than treating monitoring as a terminal step