Sample Size Calculator

A tool for determining the required number of users in an experiment.

Why Calculate Sample Size

Sample size determines how many users need to be included in an experiment to detect the expected metric change with given statistical reliability.

Sample too small: the test won't detect a real effect (false negative result), even if the change works.

Sample too large: the experiment will drag on, wasting unnecessary resources on data collection.

The tool helps find the minimum sample size for a reliable result.

How to Open the Tool

Navigate to Tools → Sample Size Calculator via the sidebar menu. You'll see a form with two tabs: Conversion and Numeric.

Select the tab depending on your key metric type.

Calculation for Conversion Metrics

Used for binary events (registration, purchase, click) — when the metric takes values of 0 or 1.

Fill in the fields:

Baseline: Current metric level before changes. For example, if 5% of users currently register, enter 5.

Uplift (%): Minimum effect you want to detect, as a percentage relative to baseline value.

For example: +10 means growth from 5% to 5.5% (5% × 1.1 = 5.5%). You can specify negative values to check metric decrease.

Significance Level (Alpha): Acceptable probability of false positive result. Default 0.05 (5%) — industry standard. Means 95% confidence in result.

Test Power (Power): Probability of detecting an effect if it exists. Default 0.8 (80%) — recommended value. Means that in 80% of cases the test will detect a real change.

Traffic Allocation: How to split users between control (A) and treatment (B). Default 50/50. Can be changed with slider if you need an unbalanced design (for example, 70/30).

Click "Calculate". The tool will show the required sample size for each group.

Calculation for Numeric Metrics

Used for continuous indicators (revenue per user, session time, number of actions).

Fill in the fields:

Baseline Mean: Current average metric value. For example, if average revenue per user is 50 rubles, enter 50.

Standard Deviation: Measure of data variability. Calculated as sqrt(variance) from historical data. The higher the variability, the larger the sample needed.

If you don't know SD, use an approximate estimate: SD ≈ Mean / 2 for moderate variability.

Uplift (%): Minimum effect as a percentage relative to baseline value. For example: +15 means growth from 50 to 57.5 rubles.

Alpha and Power: Same parameters as for conversion metrics (default 0.05 and 0.8).

Traffic Allocation: Similar to conversion metrics, default 50/50.

Click "Calculate". The result will show sample size accounting for data variability.

Result Interpretation

The tool shows three cards:

1. Test Parameters: Confirmation of entered alpha (significance level) and power (test power) values.

2. Metric Parameters:

• Baseline
• Uplift
• Target value (baseline + uplift)

3. Required Sample Size: Main result — minimum number of users for each group.

For balanced design (50/50): One number is displayed — size of each group. Total experiment size depends on number of groups (for A/B test = size × 2, for A/B/C = size × 3, etc.).

For unbalanced design: Separate numbers are displayed for each group (control and treatment variants).

Duration Planning

Knowing the sample size and average traffic, you can estimate experiment duration.

Example calculation:

• Required size: 10,000 users per group (20,000 total)
• Daily traffic: 2,000 users
• Duration: 20,000 / 2,000 = 10 days

Add a buffer of 10-15% for:

• Uneven traffic (weekends/weekdays)
• Technical failures
• Exclusion of anomalous data

Unbalanced Design

Use unequal traffic distribution (for example, 80/20 or 90/10) if:

• Testing a potentially dangerous change — limit treatment group to 10-20%
• Want to minimize risks when experimenting with new functionality
• There are technical limitations on treatment group size

Important: unbalanced design requires a larger total sample than 50/50 for the same statistical power.

Common Mistakes

Underestimating uplift: If you specify too small expected uplift (+1%), a huge sample will be required. Be realistic — what minimum effect is worth detecting?

Overestimating baseline for conversion: Baseline 50% requires a smaller sample than 5%. Don't artificially inflate baseline — this will lead to insufficient sample size.

Ignoring variability for numeric: The higher the standard deviation, the larger the sample needed. If SD is unknown, it's better to overestimate than underestimate.

Stopping before calculated size: If the experiment showed p < 0.05 at half the planned sample, don't stop it — this violates statistical correctness.