UX Glossary · A
A/B testing
Also: Split testing · Online controlled experiment
Randomly splitting live traffic between two versions to measure which performs better on a chosen metric.
An A/B test randomly assigns visitors to one of two versions and compares a chosen metric between them. Because assignment is random, differences in outcome can be attributed to the change rather than to differences between the groups. It is a randomised controlled trial applied to an interface.
What it can and cannot tell you
A/B testing measures what happened with precision, and says nothing about why. It tells you variant B converted better. It does not tell you that variant A confused people about shipping costs, which is the finding that would have helped you design variant C.
This is why it pairs with usability testing rather than replacing it. Qualitative work generates the hypothesis, the experiment measures it.
Where it goes wrong
Stopping a test as soon as it looks significant inflates false positives badly, because you get a fresh chance to stop at every check. Sample size and duration should be fixed before the test starts.
Low-traffic products often cannot run valid tests at all. Detecting a small improvement needs a large sample, and a test that never reaches it produces noise that looks like a result.
Tests also optimise the metric you picked. A change that lifts click-through and raises returns 30 days later scores as a win on the wrong measure.