Confidence Intervals

Assessing not only the presence of an effect, but also its possible magnitude.

What is a Confidence Interval

A confidence interval is a range of values within which the true effect of the change falls with a certain probability (usually 95%). It shows not only "whether there is an effect", but also "how large it is".

Example: conversion change +2.5%, confidence interval [+1.2%, +3.8%]. This means: with 95% probability, the true change is between +1.2% and +3.8%.

Why an Interval is Needed

P-value says "there is an effect or not", but doesn't show its size. Confidence interval gives the full picture: minimum and maximum possible effect.

If the interval is narrow ([+2.3%, +2.7%]), the estimate is accurate. If wide ([+0.5%, +5.0%]), there is uncertainty — perhaps a larger sample is needed.

Interval Interpretation

Interval does not include 0: effect is statistically significant. For example, [+1.2%, +3.8%] — all values are positive, effect is real.

Interval includes 0: effect is not significant. For example, [-0.5%, +2.0%] — both positive and negative values are possible, no confidence.

Interval is narrow: high estimation accuracy, large sample or low data variability.

Interval is wide: low accuracy, small sample or high variability. Result is less reliable.

Practical Application

Confidence intervals help assess business risks. If the interval is [+0.1%, +5.0%], worst case is minimal improvement (+0.1%), best case is significant (+5.0%). This helps make an informed decision accounting for uncertainty.