A/B Test Table

The main results table for analysis of two variants (control vs. treatment).

Table Structure

The table contains one row for each metric from the uploaded file. Columns show key statistics comparing the two groups.

Metric

The metric name from the uploaded CSV file. Metrics with a P marker are primary metrics that form a family for FDR correction (if correction is enabled in analysis settings).

Green row color means a statistically significant result — p-value is less than the specified significance level (alpha), which is configured in analysis parameters.

Processing

The type of preprocessing applied to the metric:

• — (empty) — for conversion metrics (binary), preprocessing is not applied
• log — logarithmic transformation
• IQR3 — outlier removal using 3×IQR method
• winsor5% — winsorization at 5% level
• winsor1% — winsorization at 1% level

Preprocessing is automatically selected by the system for numeric metrics to minimize the impact of outliers.

Effect (Δ)

The difference between treatment and control groups. Format: absolute change (relative change in %).

For conversion metrics:

• Absolute change in percentage points (pp)
• Example: 1.50 ( +15.00% ) → conversion increased by 1.5 pp, which is +15% relative to control

For numeric metrics:

• Absolute change is shown per user
• Example: -12.70 ( -1.25% ) → GMV per user decreased by 12.70 units, which is -1.25% relative to control

CI Low / CI Up

Confidence interval of the effect (95% confidence interval). Shows the range within which the true value of the effect falls with 95% probability.

Interpretation:

• If interval does not include 0 → effect is statistically significant
• If interval includes 0 → no confidence in the presence of an effect
• Narrow interval → high estimation accuracy
• Wide interval → low accuracy, larger sample needed

Example:

• CI Low: 0.80, CI Up: 2.20 → effect is in the range from +0.80 to +2.20 (significant)
• CI Low: -0.50, CI Up: 1.50 → effect can be from -0.50 to +1.50 (not significant)

P-value

The probability of obtaining the observed or more extreme difference if, in reality, the change has no effect.

Result is statistically significant if p-value is less than the specified significance level (alpha). By default, α = 0.05 is used, but this value can be configured in analysis parameters.

For primary metrics (with P marker): if FDR correction is enabled, a corrected value (q-value) is displayed here, which accounts for multiple testing across the metric family.

Important: P-value shows whether there is an effect, but does not show its size. Always look at the magnitude of the effect and confidence interval.

ProbW

Available only when "Small Samples" method is selected.

The probability that the treatment variant is better than control (Probability of Winning).

Interpretation:

• > 95% — very high confidence in improvement
• 90-95% — high confidence
• 75-90% — moderate confidence
• 50-75% — low confidence
• < 50% — the change most likely worsens the metric

Example:

• ProbW = 89.17% → in 89% of simulations, the treatment variant shows a better result
• ProbW = 9.23% → in 91% of simulations, control is better than treatment (the change likely worsens the metric)

Decision Making

Do not rely only on p-value. Consider:

1. Effect size: Is it large enough for the business?
2. Confidence interval: How accurate is the estimate?
3. Practical significance: Is it worth implementing the change considering the costs?
4. Side effects: How did other metrics change?

Example of correct decision:

• Metric purchase_conversion: effect +2.30 ( +23.00% ), CI [+1.50, +3.10], p-value 0.0012
• Conclusion: Statistically significant increase in conversion by 23%, confidence interval is narrow and does not include 0. The change is clearly positive, worth implementing.

Example of cautious decision:

• Metric revenue_per_user: effect +0.05 ( +0.50% ), CI [+0.01, +0.09], p-value 0.0180
• Conclusion: Statistically significant, but the effect is microscopic (+0.5%). Need to assess whether the costs of implementation justify such a small increase.