When to Stop an Experiment
Rules for completing A/B tests for reliable results.
Main Rule: Reach Planned Sample
The most reliable way to complete an experiment is to wait until the planned sample size (calculated before launch) is reached. This guarantees that the test has sufficient statistical power.
If you stop earlier, even at p < 0.05, the result may be unreliable. Random fluctuations in early stages often create an illusion of an effect that disappears with a larger sample.
Exceptions: When You Can Stop Earlier
Critical problem: if the experiment causes obvious harm (drop in key metrics, technical errors, user complaints), stop it immediately. Safety is more important than statistics.
Huge effect: if the effect is so large that it's obvious on a small sample (for example, conversion doubled), you can stop earlier. But this must be a truly dramatic result.
What to Avoid
Don't stop at the first p < 0.05. This is the most common mistake. If you check results every day and stop as soon as p-value drops below 0.05, the probability of false positive sharply increases.
Don't continue indefinitely. If the planned sample is reached and the result is not significant — the experiment failed. Don't extend it hoping to "catch" significance, this violates statistical correctness.
Extending an Experiment
Extending an experiment is only possible in two cases:
- Planned sample is not reached due to technical problems or low traffic.
- Before launch there was an agreement on "Sequential Testing" — a special method that corrects p-value during multiple checks.
Simply wanting to "see more data" is not a reason to extend the test.