Shadow Testing vs. Live A/B Testing

Once a change is ready to test against real traffic (not just the offline held-out set), there are two fundamentally different ways to expose it to that traffic, and they answer different questions.

The Two Modes

SHADOW TESTING runs a new version in parallel on real production traffic, logging its outputs, but NEVER serving them to users. Zero user-facing risk -- if the shadow version crashes, hallucinates wildly, or triples in latency, no user ever sees it. The tradeoff is exactly that safety: because users never see the shadow output, you don't get real behavioral or business-metric feedback (click-through, satisfaction, conversion) -- shadow testing tells you about latency, cost, and error rate under real production load, and lets you catch crashes or malformed output before any user exposure, but it cannot tell you whether users actually prefer the new behavior.

LIVE A/B TESTING is a real user-facing split test -- a defined fraction of real users actually receive the treatment variant. This gets you real engagement and quality signal, because real users are actually experiencing and reacting to the new behavior. The tradeoff is real risk to that fraction of users: if the treatment is subtly worse, some real users had a worse real experience before you found out.

Decision Criteria

The Sequencing Pattern

For a genuinely risky change, the correct sequence is shadow-test FIRST, to catch infrastructure-level problems cheaply and with zero user exposure, THEN promote to a live A/B once shadow testing shows the new version is stable -- to measure the actual quality and business impact that only real user exposure can reveal. Skipping straight to a live A/B on a risky, unproven change means the first time you discover a crash-inducing edge case or a latency regression is in front of real users in the treatment group, rather than in a shadow log nobody but you ever sees.

Common exam traps

• Treating shadow testing as a substitute for a live A/B, or vice versa. Shadow testing cannot answer "do users prefer this," and a live A/B on an unproven, risky change exposes real users to a failure mode a shadow test would have caught for free. The correct answer names the sequence -- shadow first for risky changes, live A/B second to measure real impact -- not one mode standing in for the other.
• Assuming shadow testing is always required. For a low-risk, well-understood change, going straight to a live A/B (or even skipping straight to full rollout with monitoring) can be the right call; shadow testing exists to de-risk changes where the downside of a live failure is meaningfully worse than the cost of the extra step.