P-value in Simple Terms

What statistical significance means and how to interpret it.

What is P-value

P-value is the probability of seeing such (or more extreme) difference between groups if, in reality, the change has no effect. The smaller the p-value, the less likely the difference is random.

Example: p-value = 0.03 means that if there were no effect, such a difference would occur in only 3% of cases. This is unlikely, so we conclude: the effect is real.

Significance Threshold

Usually the threshold p < 0.05 (5%) is used. If p-value is less than 0.05, the result is considered statistically significant — we reject the null hypothesis and acknowledge that there is an effect.

If p-value ≥ 0.05, the result is not significant — there is insufficient evidence that the change had an effect. This doesn't mean there is definitely no effect, just no confidence in its presence.

Common Interpretation Errors

P-value is not the probability that there is no effect. It is the probability of seeing such data in the absence of an effect.

P-value doesn't indicate effect size. A statistically significant effect can be microscopic and useless from a business perspective. Always look at the size of the change, not just p-value.

P-value depends on sample size. With a huge sample, even tiny differences become significant. With a small one — a real effect may not reach significance.

How to Use P-value

P-value is a tool, not a final verdict. Look at it together with effect size, confidence intervals, and business context. Statistical significance is a necessary but not sufficient condition for making a decision.