How A/B Test Statistics Work

Principles of statistical group comparison in simple terms.

The Main Idea

An A/B test compares two groups of users: one sees variant A, the other sees variant B. We measure a metric in each group (for example, conversion) and check if there is a significant difference.

But even if groups are identical, metrics differ slightly due to randomness. The task of statistics is to distinguish real effect from random fluctuations.

Randomness and Distributions

Imagine you flipped a coin 100 times. Theoretically it should be 50 heads and 50 tails, but in practice it might be 48 heads and 52 tails. This is normal randomness.

The same happens in experiments: even without changes, group metrics differ slightly. Statistics helps understand where randomness ends and real effect begins.

Hypotheses: Null and Alternative

Null hypothesis (H0): the change has no effect, the difference between groups is random.

Alternative hypothesis (H1): the change has an effect, the difference is real.

A statistical test checks whether we can reject the null hypothesis. If yes — the effect is real. If no — there is no proof of effect.

The Role of Sample Size

The more users in the experiment, the more accurate the effect estimate and the easier it is to detect even small changes. A small sample may "not notice" a real effect, while a large one will reliably detect it.

This is exactly why it's important to calculate sample size in advance to guarantee test statistical power.