Forecast Table (Bayesian Forecast)
Bayesian estimate of experiment success probability with a given target sample size.
When to Use Forecast
Forecast is used for early stopping of an active experiment when the business needs a quick decision.
Typical Scenario
You planned an experiment for 3 weeks (calculated sample size). But the business cannot wait that long — a decision is needed faster. You run the experiment for 1 week, collect intermediate data and use forecast to assess:
- What is the probability of success if the experiment is completed?
- Should you continue or can you already make a decision?
Forecast extrapolates current data to the target sample, showing probabilities of achieving a significant result.
When Forecast is Especially Useful
- Heavy metrics: Require huge sample sizes, waiting for full collection is too long
- Business pressure: Urgent decision needed for high-priority change
- Expensive experiments: Risk or cost of long-term execution is high
Important: Forecast is a compromise between speed and reliability. It is less accurate than a full-fledged A/B test, but allows making a decision earlier.
How Bayesian Forecast Works
Forecast uses a Bayesian approach with prior and posterior distributions.
Prior Distributions
Based on current experiment data, the system builds prior distributions of metric parameters for each group. This is an initial estimate of how metrics behave.
The longer the experiment runs, the more accurate the prior distributions:
- After 1 day → high uncertainty, wide distributions
- After 1 week → medium accuracy
- After 2 weeks → high accuracy, narrow distributions
Posterior Distributions
The system simulates continuing the experiment to the target sample, updating prior distributions to posterior. Based on thousands of simulations, success probabilities are calculated.
Key idea: The more data already collected, the less uncertainty in the forecast. A forecast based on 10% of the sample is less reliable than one based on 70% of the sample.
Meta-Information
At the top of the results, forecast parameters are shown:
Target Sample per Variant
The number of users in each group at which you plan to make a decision. The system extrapolates current data to this sample.
Alpha (α)
Significance level (usually 5%). Used to calculate the probability of achieving statistical significance.
Table Structure (A/B Forecast)
| Metric | Processing | Effect (Δ) | CI Low | CI Up | P(Δ>0) | P(α) |
|---|---|---|---|---|---|---|
| purchase_conversion | — | 1.50 ( +15.00% ) | 0.80 | 2.20 | 92.3% | 87.5% |
Metric
Metric name from the uploaded file.
Processing
Type of preprocessing applied to the metric. Same as in A/B test.
Effect (Δ)
Estimate of average effect based on current data. Format: absolute (relative %).
Important: This is not the final result, but an extrapolation to the target sample. The real effect may differ.
CI Low / CI Up
Bayesian confidence interval of the effect on the target sample. Shows the range of possible effect values with 95% probability.
Interpretation:
- Wide interval → high uncertainty
- Narrow interval → more accurate estimate
P(Δ>0) — Probability of Positive Effect
Key forecast metric. Shows the probability that the effect is positive (treatment variant is better than control).
Interpretation:
- > 90% (green) → high probability of success
- 75-90% (yellow) → moderate probability
- < 75% → low probability of success
Example:
- P(Δ>0) = 95% → in 95 out of 100 scenarios, the treatment variant will be better
P(α) — Probability of Achieving Statistical Significance
Probability that at the target sample the experiment will show p-value < α (usually 0.05).
Interpretation:
- > 80% → high probability of getting a significant result
- 50-80% → medium probability
- < 50% → low probability of significance
Important: High P(α) does not guarantee success, but shows that the experiment has sufficient power.
Table Structure (A/B/C+ Forecast)
For multiple variants, the table groups results by metrics, showing for each variant:
| Variant | Effect (Δ) | CI Low | CI Up | P(Δ>0) | P(α) | P(Top) |
|---|---|---|---|---|---|---|
| 0 (Control) | — | — | — | — | — | 12.5% |
| 1 | +1.80 ( +18.00% ) | +0.90 | +2.70 | 95.3% | 89.2% | 78.3% |
| 2 | +0.50 ( +5.00% ) | -0.20 | +1.20 | 68.5% | 42.1% | 9.2% |
Variant
Variant number. Variant 0 is usually the control group.
P(Top) — Probability of Being Best
Unique column for multiple variants. Shows the probability that this variant will be better than all others.
Interpretation:
- Sum of P(Top) across all variants = 100%
- Variant with highest P(Top) is the most promising candidate
Example:
- Variant 1: P(Top) = 78.3% → in 78 out of 100 scenarios, this variant is best
- Variant 2: P(Top) = 9.2% → low chances of being best
Making Decisions Based on Forecast
When to Stop the Experiment Early
Strong success signal:
- P(Δ>0) > 90% and P(α) > 80% → high probability of success, can stop and implement the change
- Confidence interval is narrow and does not include 0 → confidence in positive effect
Moderate signal:
- P(Δ>0) = 75-90% and P(α) = 60-80% → there is potential, but better to wait for more data
- Decision depends on business risks and urgency
Weak signal:
- P(Δ>0) < 75% or P(α) < 50% → low probability of success, even if completed
- Can stop early and NOT implement the change
When to Continue the Experiment
- P(Δ>0) = 50-75% → uncertainty is too high, need more data
- Confidence interval is very wide → forecast is unreliable
- Current sample < 30% of planned → too early for forecast
For Multiple Variants
- P(Top) > 70% for one variant → clear leader, can stop early
- P(Top) = 30-70% for several variants → no clear winner, better to continue experiment
- Look at effect size, not just P(Top)
Forecast Limitations
Forecast is an extrapolation, not a guarantee. It shows probabilities based on current data, but may be inaccurate if:
- Current sample is too small (< 30% of planned) → high uncertainty, forecast is unreliable
- Effect is unstable over time → if metric fluctuates heavily day to day, extrapolation will be inaccurate
- There are outliers or anomalies in current data → distort effect estimate
- Novelty effect → if change has a short-term effect (e.g., novelty), forecast will overestimate long-term result
Golden rule: The more data already collected, the more reliable the forecast. If you collected 70% of the sample, forecast is almost as reliable as the full experiment. If you collected 20% — forecast is approximate.
Recommendation: Use forecast as a tool for quick decision-making, but remember — it is a compromise between speed and reliability.
Common Mistakes
Mistake 1: Stopping the Experiment Too Early
Forecast based on 10-20% of the sample is extremely unreliable. Let the experiment accumulate at least 30-40% of the planned sample before using forecast.
Mistake 2: Making a Decision Based Only on P(Δ>0)
Even if P(Δ>0) = 95%, look at effect size. A microscopic effect (+0.1%) with high probability is not a reason for early stopping and implementation.
Mistake 3: Ignoring Confidence Interval
Wide CI means high uncertainty. If CI = [-2%, +10%], even with P(Δ>0) = 70%, the real effect may be negative. In such case, better to continue the experiment.
Mistake 4: Not Accounting for Metric Instability
If metric fluctuates heavily day to day (e.g., due to seasonality or external factors), forecast will be inaccurate. Make sure the effect is stable over several days.
Mistake 5: Comparing Forecast with Final Result
If you still complete the experiment, do not expect the result to match the forecast. Forecast is a probabilistic estimate, not an exact prediction.