A/B Testing

011 min

A/B Testing at a glance

• What it is: A randomized experiment that measures which version of something performs better on one clear metric.
• Origin: Kohavi, Henne, and Sommerfield formalized web experimentation practice at Microsoft in a 2007 KDD paper.
• Confused with: A multivariate test, which studies several changes and their interactions at once.
• Watch for: Peeking at results early inflates the false-positive rate far past the intended 5%.

021 min

Why A/B Testing matters

A product manager watching a live test dashboard has to decide when a result is trustworthy enough to ship on. Get that call wrong and the company ships changes that never really worked, or kills changes that did work but had not run long enough to prove it. Watching a metric move after a launch cannot settle this by itself, because a holiday, a competitor's outage, or a marketing push landing the same week is an equally good explanation for the same move. A/B testing exists to answer the question directly, by comparing two groups that differ only in which version they saw.

Most of that confidence is misplaced to begin with. Only one third of the ideas tested on Microsoft's own experimentation platform improved the metric they were designed to improve, which is exactly why an objective measurement matters more than how good an idea sounds in a meeting. The cost of misreading it is concrete, not abstract. A growth team that ships on a peeked, not-yet-significant result can spend months building on a change that was actually neutral or harmful, and only find out once a later, properly run test contradicts it — after the engineering cost is already spent.

032 min

What A/B Testing actually measures

An A/B test splits traffic into two groups by chance: a control group that keeps the existing version, and a treatment group that sees the change. Every other factor — the day of the week, the mix of new and returning users, a marketing campaign running that week — lands on both groups in roughly equal proportion, because the split is random rather than chosen. That is what separates a controlled experiment from watching a metric move after a launch and guessing why. Correlation and Causation covers the general version of this gap: a metric changing after a release does not show the release caused it, because anything else that changed at the same time is an equally good explanation. A controlled experiment closes that gap by holding every other variable constant across the two groups through randomization alone, not through argument.

At the end of the planned run, the two groups' values on one chosen metric — a conversion rate, a click-through rate, a revenue figure — are compared with a statistical significance test, usually a two-proportion or two-mean test. The test produces a p-value: the probability of seeing a difference this large between the groups if the change actually had no effect at all. A result is typically called statistically significant when that p-value falls below a threshold fixed in advance, commonly 0.05. The threshold, the metric, and the number of users needed all have to be decided before the test starts. Deciding them afterward, once the data already point somewhere, is one of the ways a test stops being controlled.

041 min

Where A/B Testing comes from

Randomized experiments predate the web by decades — Ronald Fisher formalized the method for agricultural research in the 1920s — but running them continuously on live software traces to a specific paper. In 2007, Ron Kohavi, Randal Henne, and Dan Sommerfield of Microsoft published a paper at the ACM KDD conference making the case for measurement over opinion in software development. Kohavi, Henne, and Sommerfield argued that teams should listen to their customers, not to the Highest Paid Person's Opinion, or HiPPO. The term HiPPO stuck because it named something every team recognizes: a senior person's guess overriding a measured result. Their paper set out the statistical and engineering practices — randomization, sample size planning, variance reduction — that most A/B testing tools still implement today, and Kohavi went on to lead Microsoft's own Experimentation Platform, the source of much of the evidence in this article.

051 min

A worked example

Suppose a checkout page currently converts 4% of visitors into buyers, and the team wants to know whether a redesigned checkout button raises that rate by at least one percentage point, to 5%. Using the standard formula for a two-proportion test at 95% confidence and 80% power, that requires roughly 6,030 visitors in each group before the test can reliably tell a real one-point lift from noise.

If the team instead wants to detect a smaller, more realistic lift of half a percentage point — to 4.5% — the same formula requires roughly 24,110 visitors per group: about four times as many, because the required sample size grows with the square of how small an effect the test needs to catch.

Halving the effect size does not double the required sample; it roughly quadruples it. A team that skips this calculation and just runs the test for a fixed two weeks, whatever traffic that happens to bring, often ends up with a result too noisy to trust in either direction.

061 min

How A/B Testing shows up in product and engineering work

A growth engineer ships a new onboarding flow behind a feature flag, splits new signups 50/50 between the old and new flow, and watches a dashboard update the split's activation rate hour by hour. Two days in, the new flow is ahead by three points, and someone on the team wants to roll it out to everyone immediately, before the planned two-week run finishes. This is the moment a controlled experiment either stays controlled or quietly stops being one: shipping on day two means shipping on a sample size the team never calculated as sufficient, using a result nobody planned to trust yet.

The same dashboard shows something else worth noticing: dozens of secondary metrics move along with the primary one, because a redesigned flow touches load time, error rate, and a handful of unrelated buttons on the same page. A PM who scans all of them looking for anything that moved is running dozens of informal tests inside one experiment, and a few of those will look significant by chance alone even when nothing real changed. The experimental mindset that makes A/B testing valuable — deciding the metric and the stopping point before looking at data — is also the discipline most under pressure the moment a dashboard starts moving in a direction somebody wanted.

071 min

When to use A/B Testing — and when not to

A/B testing fits a specific precondition: a metric that can be measured within the test's planned duration, a baseline value for that metric already known from existing traffic, and enough visitors passing through the experience to reach a sample size fixed before the test starts. Below a few thousand users per variant, most day-to-day product changes take weeks or months to reach a sample size large enough to detect anything but a huge effect, and the honest answer is that A/B testing is the wrong tool at that traffic level — a smaller-sample method such as a moderated usability study answers the question faster.

A/B testing also assumes one group's experience does not leak into the other's. A pricing change tested on one side of a marketplace can shift behavior on the other side even though that side saw no change at all, which breaks the assumption the test relies on. And it assumes the team already knows which metric decides success. Running the test first and deciding afterward which of the metrics that moved counts as the win is not A/B testing with an unclear goal — it is the multiple-comparisons problem, covered next, wearing a different name.

083 min

Where A/B Testing goes wrong

Four mechanisms account for most A/B tests that produce a confident, wrong answer.

091 min

A/B Testing vs. nearby concepts

The concept most often mistaken for A/B testing is the multivariate test. An A/B test compares two or more complete versions of an experience against each other. A multivariate test instead varies several elements at once — a headline, an image, a button color — independently, and measures not just each element's effect but how the elements interact with each other. The deciding fact is traffic: a multivariate test needs enough visitors to fill every combination of elements being varied, which multiplies the sample-size problem from the worked example above by the number of combinations, so it only works at traffic volumes far above what most A/B tests need.

A/B testing is also confused with an A/A test, which compares two identical versions against each other on purpose. An A/A test measures nothing about a real change — its only job is to confirm that the randomization and measurement system itself produces a result close to no difference when there is no real difference to find, which is exactly the kind of check that would have caught the Skype sample ratio mismatch below before it reached a real experiment.

102 min

A second case: what a trustworthy result looks like, and what a broken one looks like

The calculation above shows how to size a test properly. A real case shows what a trustworthy result looks like once the test is actually run. At Microsoft's Bing, a team added clickable links to individual sections of an advertiser's site directly inside the ad listing, then measured the change with a controlled experiment. The site links alone increased revenue but also degraded user metrics and page-load time; only after the team offset the extra space by showing fewer mainline ads per query did the feature end up improving revenue by tens of millions of dollars per year with neutral user impact. The full result needed both halves of that change to be true at once, which is exactly the kind of trade-off a single before/after number would have hidden.

A different case shows what happens when the measurement itself breaks. Fabijan and colleagues describe a Skype experiment that tested a machine-learning model for setting audio-buffering behavior against a fixed setting, with each call randomly assigned to one variant. In this Skype experiment, a logging bug meant the experiment analysis system collected 30% fewer sessions in one group than the design called for, triggering a sample ratio mismatch alert. The root cause traced back to the logging system: a variant ID tracked in memory kept updating mid-call even though the new setting itself was not applied mid-call, so a chunk of sessions ended up logged under the wrong variant. Every metric computed before that bug was found and fixed was comparing the wrong two groups.

This is why the discipline behind A/B testing includes checking the experiment's own bookkeeping, not just its headline result. A team that restricts its analysis to users who were still active weeks into the test runs into a related trap: analyzing users who have been active for a long time introduces survivorship bias, because the users who dropped out along the way are exactly the ones a fair comparison needs to keep.

112 min

Where the evidence on A/B Testing is contested

A controlled experiment is only as good as the metric it optimizes, and practitioners disagree about how hard that metric is to choose well. Kohavi and Longbotham warn that market share can be a long-term goal, but it is a terrible short-term criterion. A worse search engine can force people to issue more queries in the short run, which looks like more engagement even as users are quietly finding better alternatives elsewhere. The disagreement is not about whether controlled experiments work; it is about whether any short-run metric, measured over the one or two weeks a typical test runs, can be trusted to predict a change's effect months later.

Novelty and primacy effects sharpen the same worry. A visibly new design element often performs unusually well in its first days simply because it is new and draws extra attention — a primacy effect — before settling into whatever its steady-state performance actually is, and the reverse can happen when a familiar option is removed and users need time to adjust. A test that stops the moment an initial spike looks good, or the moment an initial dip looks bad, is measuring the novelty of the change rather than its real effect. Where practitioners land, in practice, is not abandoning short-run tests but running them past the point where the metric first looks decided, specifically to let a novelty spike fade before trusting the number underneath it.

122 min

Frequently asked questions about A/B Testing

?7 questions

Questions people ask

▶3 videos

Watch

• 

  Understanding A/B Testing

• 

  Simple explanation of A/B Testing In Hindi

  codebasics Hindi

• 

  A/B Testing In Data Science

  Krish Naik

§5 sources

Sources