A/B Testing & the Iterate Loop

Improvement is empirical: change one variable, measure it against the eval, keep what wins. Intuition and a few eyeballed examples are not a substitute for measurement.

A/B Testing

• A/B test competing prompts, models, retrieval configs, or parameters on the same dataset and compare on the metrics defined for the eval
• Change one thing at a time -- an ablation mindset -- so you can attribute the effect to the specific variable that changed

Contextual Retrieval: The Canonical Example

Contextual retrieval is the domain's standard illustration of this pattern: measuring retrieval-failure rate lets you A/B embeddings-only vs. embeddings+BM25 vs. +reranking, and choose the winning configuration on evidence, not intuition.

The Iterate Loop

Iteration is a loop, not a one-shot decision:

hypothesize -> change -> measure -> adopt or revert

Each pass through the loop tests one hypothesis against the eval and either keeps the change (if it wins) or reverts it (if it doesn't).

Common exam traps

• Switching everything at once and eyeballing a few answers. This breaks attribution -- if quality changes, you can't tell which of the several simultaneous changes caused it. The correct approach is always a measured A/B test on a representative dataset that changes one variable at a time and reports a defined metric (e.g., retrieval-failure rate), not a holistic before/after glance.