SRM Analysis (Sample Ratio Mismatch)
Checking the correctness of user distribution between experiment groups.
What is SRM
Sample Ratio Mismatch (SRM) is a skew in traffic distribution between experiment groups. If you planned to split users 50/50, but actually got 52/48, this may be random chance or a sign of error.
SRM analysis checks whether the actual distribution matches the expected one. If not — experiment results may be unreliable.
SRM Block in Results
The SRM block is always displayed first, before tables with metrics. It has two possible states:
✓ SRM Not Detected (green block)
✓ SRM Not Detected
Traffic distribution matches expected
p-value: 0.4520 (α = 0.01)Interpretation:
- User distribution between groups matches the planned distribution
- P-value ≥ 0.01 → differences in group sizes are explained by chance
- Experiment is correct, results can be trusted
✗ SRM Detected (red block)
✗ SRM Detected
Traffic distribution skew detected
p-value: 0.0003 (α = 0.01)Interpretation:
- Actual user distribution significantly differs from expected
- P-value < 0.01 → skew is too large to be chance
- Experiment results are unreliable!
How the Check Works
AB-Labz Workbench uses a Chi-squared test to check for SRM:
- Compares actual number of users in each group with expected (usually 50/50)
- Calculates p-value
- If p-value < 0.01 (stricter threshold than for metrics) → SRM detected
Threshold α = 0.01 (not 0.05) is used intentionally to reduce the probability of false alarms.
Causes of SRM
If SRM is detected, possible causes:
1. Technical Error in Split
- Error in randomization algorithm
- Incorrect hash function for splitting users
- Bug in code that handles groups differently
Example: If some users in the control group do not register due to a bug, they will not appear in the final dataset, creating a skew.
2. Post-Randomization Filtering
- Users from one group are more often excluded from analysis
- Different conditions for inclusion in final dataset for groups
Example: If the treatment variant causes technical errors, users with errors are excluded from analysis → skew.
3. Temporal Factors
- Experiment was not launched simultaneously for all groups
- Different data collection times for groups
4. Segmentation or Filters
- Dataset is already filtered by some conditions that unevenly affect groups
Example: If you analyze only users from Russia, but the split was global, a skew may occur.
What to Do If SRM Is Detected
Step 1: Do Not Trust Experiment Results
Even if metrics show significant differences, they may be an artifact of skew, not a real effect.
Step 2: Find the Cause
- Check randomization code
- Make sure both groups are processed identically
- Check logs for errors specific to one group
- Make sure filters are applied identically to both groups
Step 3: Fix the Problem and Restart the Experiment
SRM cannot be "fixed" with statistical methods. It is necessary to find and eliminate the technical cause, then restart the experiment.
Step 4: If Cause Is Not Found
If you are sure that randomization is correct, but SRM is still detected:
- There may be artifacts in the dataset (duplicates, gaps)
- Check data quality
- Try to rebuild the dataset
Common Misconceptions
Misconception 1: "SRM is small, can be ignored"
Even a small skew (e.g., 51/49 instead of 50/50) may be a sign of systematic error. If SRM is statistically significant — it is a signal of a problem.
Misconception 2: "Results can be corrected with weights"
Statistical correction does not solve the problem if skew is caused by technical errors. Weighting may hide the real problem.
Misconception 3: "SRM does not matter if metrics are not significant"
SRM is a problem of experiment design, not of result. Even if metrics show "no effect," this may be due to skew.
Summary
- SRM analysis is mandatory for every experiment
- Green block → everything is fine, continue analysis
- Red block → stop, find the cause, restart the experiment
- Do not try to bypass SRM with statistical methods — eliminate the technical cause