Common Mistakes in A/B Tests

What to avoid when planning and analyzing experiments.

Peeking: Looking at Results

Mistake: checking results multiple times during the experiment and stopping the test as soon as you see a significant result.

Why this is bad: each check increases the probability of a false positive result. If you look at p-value 10 times, the chance of randomly getting p < 0.05 is much higher than 5%.

How to avoid: determine sample size and experiment end date in advance. Check results only after completion.

Insufficient Sample Size

Mistake: launching an experiment without calculating sample size and stopping it too early.

Why this is bad: a small sample will not detect a real effect (low statistical power). You risk missing a successful change or, conversely, implementing an ineffective one.

How to avoid: always calculate sample size before launch. Use a calculator with realistic effect expectations.

Changing Hypothesis After Results

Mistake: looking at results and changing the key metric or interpretation to "find success".

Why this is bad: this is called HARKing (Hypothesizing After Results are Known). You're fitting conclusions to data rather than testing a real hypothesis.

How to avoid: fix the key metric and success criteria before launch. Analyze additional metrics, but don't change the main criterion retroactively.

Ignoring Practical Significance

Mistake: stopping only at p-value and not looking at effect size.

Why this is bad: a statistically significant change of 0.01% is useless for business, even if p < 0.001. Implementing the change requires resources that won't be recouped by a microscopic effect.

How to avoid: always assess effect size and confidence interval. Ask yourself: is this effect worth the effort spent?