4.2 A/B Testing, Iterative Improvement & Diagnosing System Issues

You've now got an eval — a scored, repeatable measurement of quality. What do you do with it? The temptation, once a system is live, is to make several changes at once because they all seem obviously good: swap in a better embedding model, add reranking, tweak the prompt's tone, and bump the context window, all in the same release. Then quality goes up two points and everyone celebrates — but nobody can say which of the four changes actually helped, or whether one of them quietly hurt while another compensated.

Task Statement 4.3 is about doing this properly: improvement is empirical. You change ONE variable, measure it against the eval you built in 4.1, and keep whatever wins. This is called an A/B test — comparing two (or more) competing variants of a prompt, model, retrieval configuration, or parameter on the SAME dataset, scored on the SAME defined metrics. Whichever variant scores higher is adopted; the other is discarded. Simple to state, and surprisingly easy to violate under launch pressure.

The discipline underneath A/B testing is called an ABLATION MINDSET: change one thing at a time so you can attribute the effect to that specific change. If you change the prompt AND the retrieval config in the same release and quality moves, you've learned that something in that bundle mattered — but you haven't learned which one, and you can't safely keep half the bundle and drop the other half next time, because you never isolated their individual effects.

Put A/B testing on a repeating cycle and you get the iterate loop that underlies continuous improvement of any Claude-based system: HYPOTHESIZE → CHANGE → MEASURE → ADOPT OR REVERT. You form a specific, falsifiable belief ("adding a reranking step will reduce retrieval-failure rate"), make exactly that one change, run it through the eval, and either keep it because the metric improved or discard it because it didn't. Then you pick the next hypothesis and go again.

The domain's canonical illustration of this loop is CONTEXTUAL RETRIEVAL. Suppose you're deciding how to retrieve context for a RAG pipeline: embeddings-only search, embeddings plus BM25 keyword search, or embeddings plus BM25 plus a reranking step. Each of these three configurations is a hypothesis about what improves retrieval. The correct way to decide among them is to define a metric — RETRIEVAL-FAILURE RATE, the fraction of queries where the retrieved chunks don't actually contain the answer — and then A/B test the three configurations on the same dataset, changing only the retrieval strategy while holding everything else constant. Whichever configuration produces the lowest retrieval-failure rate is the one you ship.

Why does this matter beyond retrieval specifically? Because the same shape — one metric, several variants, one held-constant dataset — applies to model choice, prompt phrasing, tool descriptions, and system parameters alike. Once you internalize contextual retrieval as the worked example, you can transplant the pattern onto almost any "which of these options is actually better" decision the exam throws at you.

A/B testing is how you improve a system that's already working reasonably well. But Task Statement 4.4 covers a different, more urgent moment: quality has visibly DROPPED, and you need to find out why before you can fix it. The single most important habit here is resisting the urge to patch the first thing you notice. If the fault is actually three steps upstream in a multi-step pipeline, patching the visible symptom just moves where the bug shows up next time — it doesn't remove it.

Think of it like a chain of dominoes where only the last one is visible to you. You see the last domino has fallen (the wrong final answer); your job is to find which domino fell FIRST, because that's the one that actually needs fixing. Push the last domino back upright and the next run just knocks it down again, because the real cause — the first domino — was never addressed.

Fortunately, quality regressions cluster into a small number of recognizable patterns, and each pattern has a distinctive symptom signature. The exam's core skill in this task statement is symptom-to-cause matching: given a description of what changed and what didn't, name the layer most likely at fault and the first diagnostic move — not the eventual fix, the FIRST move.

The elimination logic is the real skill, not memorizing the table row by row. "A document refresh happened, model and latency are unchanged, and now answers are confidently wrong" tells you the model itself almost certainly didn't change — so don't waste time re-running model evals — but the DATA feeding retrieval did, which is exactly what points at retrieval and indexing. Conversely, "quality regressed the day we bumped to a new model version, and nothing else changed" points squarely at model mismatch, and the first move is re-running your evals against the new version, not rewriting the prompt.

Of all six rows in that table, one misdiagnosis recurs constantly on the exam and in real incident channels alike: blaming "the model got worse" for what is actually a retrieval problem or a context problem. It's an appealing story because it's simple and it doesn't implicate your own pipeline — but it's usually wrong, and the giveaway is right there in the constants.

If the model version and the latency are both unchanged, the model itself is the least likely thing to have changed. What's left that COULD have changed — the underlying documents, an index that silently went stale, a session that's grown long enough to bury the original instructions — is where you should look first. Reflexively re-running a full model evaluation, or worse, ripping out and replacing the model, wastes real effort chasing a cause the evidence already argues against.

The flip side of the same trap: a genuine model-version regression sometimes gets misdiagnosed as a prompt failure or a hallucination, and someone spends a day rewriting the system prompt when the fix was simply to re-run the eval against the new version and pin back to the old one. The discipline that prevents both mistakes is the same: check what actually changed before you decide what to fix.

The symptom-to-cause table works well for single-hop failures, but real production agents are multi-step pipelines — a request might trigger a retrieval call, then a tool call, then a second retrieval, then the final generation. When the final answer is wrong, which of those four steps was the one that actually went sideways? Guessing from the visible symptom alone often gets this wrong, because different root causes can produce the exact same-looking final symptom.

TRACE ANALYSIS is the practice of walking the logs of every call, tool invocation, retrieval hit, and result in a request, in order, to find the EARLIEST deviation from expected behavior — not just the final visible symptom. If the final answer cites a fact that isn't in any retrieved chunk, trace analysis lets you confirm whether the retrieval step actually returned the right chunk and the model ignored it (a prompt/attention problem), or whether the retrieval step never returned the right chunk at all (a retrieval problem) — two very different fixes that look identical from the final output alone.

Trace analysis has a hard prerequisite that's easy to overlook until you need it and don't have it: the logs it depends on — request/response pairs, tool calls, retrieval results — have to already be captured, in production, before the incident happens. There is no trace to analyze after the fact if nothing was logged in the first place. This is exactly why the monitoring practices you'll meet in 4.3.6 aren't a separate topic from diagnosis — they're the prerequisite for it.

The ablation mindset from 4.2.1 -- change one variable, measure it -- tells you WHAT to change. It doesn't, by itself, guarantee the number you measured means anything. A test that changes exactly one variable can still be an invalid test if it lacks the structural discipline underneath a genuine A/B test: a hypothesis declared in advance, a real treatment/control split, one pre-declared primary metric, and enough traffic run through the comparison that the result isn't just noise dressed up as a finding. "One variable at a time" is necessary but not sufficient, and the exam tests both halves.

• 1.A stated hypothesis, declared BEFORE running the test -- "I believe change X will move metric Y by roughly Z" written down in advance, not reconstructed afterward to fit whatever happened
• 2.Explicit treatment/control assignment -- a defined population actually receives the new variant, a defined population doesn't, and the split is deliberate, not "whoever happened to use it this week"
• 3.A single, pre-declared primary metric -- decided before the test runs, not chosen afterward from whichever number happened to move favorably
• 4.A sample size sufficient to detect the effect size you actually care about -- calculated in advance, so a result isn't declared a win or a loss on a sample too small to distinguish a real effect from noise

Here's an original worked failure mode. A support team rewrites the prompt used for their ticket-triage classifier, hoping to improve routing accuracy. They roll the new prompt out to the first 40 tickets that happen to arrive on a Tuesday afternoon, compare routing accuracy against the old prompt's historical average, and see a 6-point lift -- 91% versus a historical 85%. The team is thrilled and ships the change company-wide that same day. Two weeks later, once the new prompt has run against real volume -- several thousand tickets spanning every day of the week, every ticket category, every shift -- routing accuracy has settled back to 84.6%, statistically indistinguishable from the old prompt. Forty tickets on one Tuesday afternoon was a small enough sample that a few easy, well-worded tickets landing in that window could swing the score by six points purely by chance. There was never a real effect to detect; nobody calculated whether 40 tickets was even large enough to distinguish a genuine 6-point improvement from ordinary variation before declaring victory and shipping.

Once a change is ready to test against real traffic rather than just the offline held-out set, there are two fundamentally different ways to expose it, and they answer different questions. SHADOW TESTING runs a new version in parallel on real production traffic, logging its outputs, but NEVER serving them to users -- zero user-facing risk, because if the shadow version crashes, hallucinates wildly, or triples in latency, no user ever sees it. The tradeoff is exactly that safety: because users never see the shadow output, you get no real behavioral or business-metric feedback (click-through, satisfaction, conversion). Shadow testing is for latency, cost, and error-rate comparison, and for catching crashes before any user exposure -- not for learning whether users actually prefer the new behavior.

LIVE A/B TESTING is a real user-facing split test -- a defined fraction of real users actually receive the treatment variant. This is what gets you real engagement and quality signal, because real users are genuinely experiencing and reacting to the new behavior. The tradeoff is real risk: if the treatment turns out to be subtly worse, some real users had a worse real experience before anyone found out.

Consider a coding-assistant product about to switch its default model to a new version. Because a model swap touches every request the product serves, the team shadow-tests the new version for a week: every real request is also sent to the new model in parallel, its output logged but never shown to any user. The shadow run surfaces two problems for free -- p99 latency is 40% higher on the new version, and a narrow category of multi-file edits triggers a formatting bug in about 2% of shadow runs. Both get fixed with zero user exposure. Only once a second shadow run comes back clean does the team promote the change to a live A/B, serving the new model to 10% of real users for two weeks and measuring actual task-completion rate and user-reported satisfaction -- the quality signal a shadow test could never have produced, because shadow output is never seen by anyone who could react to it.

You now have every piece of the improvement-and-repair cycle: A/B testing, the iterate loop, and the structural rigor (sample size, hypothesis discipline, outcome-shopping) that makes an A/B result trustworthy; shadow testing to de-risk a change before it ever reaches real users; and symptom-to-cause matching plus trace analysis for diagnosing an unexpected regression. In practice these chain together -- a regression you diagnose gets fixed via the same A/B discipline: form a hypothesis about the fix, change exactly that one thing, measure it against the eval with a real sample size and a pre-declared metric, and adopt it only if it wins.

The next lesson turns from diagnosis to prevention and efficiency: how do you optimize cost, latency, and token usage without silently trading away the quality you just spent this lesson learning to protect — and how do you keep watching the system once it's live, so the next regression gets caught by a dashboard instead of a customer?