A/B/C+ Test Table

Results of analysis of multiple variants (3 or more groups). Consists of two tables: global test and pairwise comparisons.

Table 1: Global P-value

The first table shows global test: is there at least one difference between all groups.

Structure

Metric	Global p-value
purchase_conversion	0.0023
revenue_per_user	0.1450

What Global P-value Shows

This is the result of an overall statistical test that tests the null hypothesis: all groups are the same.

If Global p-value < α (significance level):

• There are statistically significant differences between groups
• Proceed to pairwise comparisons to find which specific variants differ

If Global p-value ≥ α:

• No confidence that groups differ
• Pairwise comparisons don't make sense

Important: Global p-value doesn't tell which variant is better. Use pairwise comparisons for that.

Table 2: Pairwise Comparisons

The second table shows all possible variant pairs and their differences for each metric.

Table Structure

The table is organized as follows:

• Rows — pairs of compared groups (for example, 0-1, 0-2, 1-2)
• Columns — grouped by metrics, for each metric several characteristics are shown

For standard analysis:

• p — comparison p-value (with correction for multiple comparisons)
• CI — confidence interval in format [CI Low, CI Up]

For small sample analysis:

• p — comparison p-value
• CI — confidence interval
• ProbW — probability that the second group is better than the first (P(win))

Example table:

Group	cr_item			cr_cart
	p	CI	ProbW	p	CI	ProbW
0-1	0.7360	[-0.02, 0.01]	36.80%	0.5373	[-0.02, 0.01]	26.87%
0-2	0.7360	[-0.02, 0.01]	20.70%	0.3940	[-0.03, 0.00]	6.57%
1-2	0.7360	[-0.02, 0.01]	28.53%	0.5373	[-0.02, 0.01]	18.10%

Group Pairs

Notation:

• 0-1 — comparison of group 0 (control) with group 1
• 0-2 — comparison of group 0 with group 2
• 1-2 — comparison of group 1 with group 2

The effect is calculated as: second group minus first. For example, for pair 0-1, a positive effect means that group 1 is better than group 0.

P-value (with correction for multiple comparisons)

Critical: For multiple comparisons, the system automatically applies correction for multiple testing to reduce the probability of false positives.

Without correction: if making 10 comparisons at α=0.05, the probability of at least one false positive is ~40%.

With correction: the probability of false positive is controlled at the specified level.

Threshold: p < α (already with correction), where α is the significance level set in analysis settings.

Confidence Interval (CI)

Shows the range within which the true difference between groups falls with 95% probability.

Interpretation:

• Does not include 0 → difference is statistically significant
• Includes 0 → no confidence in presence of difference

ProbW (probability of "win")

Available only for small sample analysis.

Shows the probability that the second group in the pair is better than the first. For example:

• 0-1, ProbW = 36.80% → probability that group 1 is better than group 0 is 36.80%
• 1-2, ProbW = 28.53% → probability that group 2 is better than group 1 is 28.53%

Interpretation:

• > 95% — very high confidence that the second group is better
• 90-95% — high confidence
• 75-90% — moderate confidence
• 50-75% — low confidence
• < 50% — most likely the first group is better than the second

Making Decisions with Multiple Variants

Step 1: Check Global P-value

If it's not significant — no point digging into pairwise comparisons. Result: all variants are approximately the same.

Step 2: Find the Best Variant

Look at pairwise comparisons. The variant that is statistically significantly better than all others is a candidate for implementation.

Example:

• Variant 1 vs Variant 0: p = 0.002 (significant, +20%)
• Variant 2 vs Variant 0: p = 0.150 (not significant)
• Variant 2 vs Variant 1: p = 0.080 (not significant)

Conclusion: Variant 1 is clearly better than control, while Variant 2 showed no advantage. Implement Variant 1.

Step 3: Consider Practical Significance

Even if all variants are statistically better than control, choose the one with the largest effect at comparable implementation costs.

Common Mistakes

Mistake 1: Ignoring Global P-value Don't start analysis with pairwise comparisons. If Global p-value is not significant, pairwise differences may be random.

Mistake 2: Multiple comparison "manually" Don't run many A/B tests instead of one A/B/C test. This increases the probability of false positives.

Mistake 3: Choosing "best" without considering CI The variant with the largest effect may have a wide CI, which means low estimation reliability. Look at CI, not just point estimate.