A/B Testing Rigor: Sample Size, Hypothesis Discipline & Outcome-Shopping

The ablation mindset -- change one variable, measure it -- tells you WHAT to change. It does not, by itself, guarantee that the number you measured means anything. A test that changes exactly one variable can still be an invalid test if it lacks the structural discipline underneath a real A/B test: a hypothesis declared in advance, a real treatment/control split, one pre-declared primary metric, and enough sample size to trust the result. Skip any of the four and "one variable at a time" is necessary but not sufficient.

The Four Structural Requirements

1. A stated hypothesis, declared BEFORE running the test -- "I believe change X will move metric Y by roughly Z" written down in advance, not reconstructed afterward to fit whatever happened
2. Explicit treatment/control assignment -- a defined population actually gets the new variant, a defined population doesn't, and the split is deliberate (not "whoever happened to use it this week")
3. A single, pre-declared primary metric -- decided before the test runs, not chosen afterward from whichever number moved in a favorable direction
4. A sample size sufficient to detect the effect size you care about -- calculated in advance, so a result isn't declared a win or a loss on a sample too small to distinguish a real effect from noise

A Worked Failure Mode

A support team rewrites the prompt used for their ticket-triage classifier, hoping to improve routing accuracy. They roll it out to the first 40 tickets that happen to come in on a Tuesday afternoon, compare the routing-accuracy score against the old prompt's historical average, and see a 6-point lift -- 91% versus a historical 85%. Encouraged, they ship it team-wide immediately.

Two weeks later, once the change has run against real volume (several thousand tickets across every day of the week, every ticket category, every shift), routing accuracy has settled back to 84.6% -- statistically indistinguishable from the old prompt. What happened? Forty tickets on one Tuesday afternoon is a sample small enough that a few easy, well-worded tickets landing in that window can swing the score by six points purely by chance -- there was never a real effect to detect, just noise that looked like a win because nobody calculated whether 40 tickets was even enough to distinguish a genuine 6-point improvement from random variation before declaring victory.

Naming the Anti-Pattern: Outcome-Shopping

OUTCOME-SHOPPING is choosing your success metric AFTER seeing which one moved favorably, rather than declaring it upfront. Picture a team that tracks five metrics on a prompt change -- accuracy, latency, cost, user satisfaction, escalation rate -- with no primary metric declared in advance. The change makes accuracy slightly worse and cost slightly worse, but escalation rate ticks down. The team writes up "escalation rate improved" as the headline result and ships. That's outcome-shopping: with five metrics in play and no metric declared as primary beforehand, at least one of them moving favorably by chance is likely even if the change did nothing real -- and the team gets to retroactively decide that's the one that mattered.

Common exam traps

• A large lift on a small sample presented as a settled result. The exam's version of the support-ticket scenario above: a compelling early number from too few sessions, presented as proof, before volume gives you a real sample. The correct response is always to ask whether the sample size was sufficient to detect the claimed effect BEFORE trusting the number, not to treat an early, exciting result as final.
• A post-hoc metric swap. If a scenario describes multiple metrics being tracked and the "winning" metric only gets named as primary after results come in, that's outcome-shopping -- the fix is declaring the primary metric before the test runs, every time, not cherry-picking the friendliest number after the fact.