Forecasting
Bayesian estimation of success probability for early experiment stopping.
When to Use Forecast
Forecast is a tool for early stopping of an active experiment when business cannot wait for complete sample collection.
Typical scenario: You planned an experiment for 3 weeks, but business needs a decision faster. You run for 1 week, upload interim data and get a forecast: what is the probability that the experiment will be successful if completed.
When this is useful:
- Heavy metrics require long sample collection (weeks/months)
- Business requires urgent decision
- Costs or risks of lengthy experiment are high
Important: Forecast is a compromise between speed and reliability. The more data already collected, the more accurate the forecast.
How Forecast Works
The system uses a Bayesian approach with simulation of experiment continuation.
Step 1: Prior distributions Based on current experiment data, prior distributions are built for metrics of each group.
Step 2: Simulation The system simulates thousands of scenarios of experiment continuation to target sample, updating prior distributions to posterior.
Step 3: Probabilities Probabilities are calculated: P(effect is positive), P(will reach significance), P(best variant).
Golden rule: The longer the experiment runs, the more accurate the prior distributions and the more reliable the forecast. Forecast based on 10% of sample is less reliable than based on 70%.
Forecasting Interface
Navigate to Forecast via navigation after uploading interim experiment data.
Steps:
- Upload data with current experiment state (via CSV or API)
- Specify target sample (what size the experiment was planned for)
- Select significance level α (usually 0.05)
- If there are segments — select the needed one for analysis
- Click "Run Forecast"
The system will show a table with results for each metric:
- Effect (Δ) — effect estimate based on current data
- CI Low / CI Up — confidence interval of effect at target sample
- P(Δ>0) — probability of positive effect
- P(α) — probability of reaching statistical significance
- P(Top) — for ABC+, probability of being the best variant
Result Interpretation
P(Δ>0) > 90% and P(α) > 80%: Strong success signal. Can stop experiment early and implement change.
P(Δ>0) = 75-90% and P(α) = 60-80%: Moderate signal. There is prospect, but better to wait for more data or assess business risks.
P(Δ>0) < 75% or P(α) < 50%: Weak signal. Low probability of success even if completed. Can stop early and NOT implement change.
Wide confidence interval: High uncertainty. Better to continue experiment and collect more data.
Forecast Limitations
Too early forecast is unreliable: Forecast based on < 30% of sample has high uncertainty. Let the experiment collect at least a third of planned sample.
Effect may be unstable: If metric fluctuates strongly day to day or effect has short-term character (novelty), extrapolation will be inaccurate.
Outliers are handled automatically: The system uses a data preprocessor that corrects anomalies and outliers (logarithm, winsorization, IQR). Technical failures or holidays will not distort the forecast.
Forecast is not a guarantee: This is a probabilistic estimate, not an exact prediction. Actual result may differ.
Use wisely: Forecast is a tool for quick decision-making under business constraints, but not a replacement for full A/B test.