FDR Correction

Mechanism for controlling false positive results when analyzing multiple metrics simultaneously.

Why FDR Correction is Needed

When you analyze multiple metrics in one experiment, the probability of getting a "randomly" significant result for at least one of them increases.

Multiple testing problem:

• When analyzing one metric with α = 0.05, probability of false positive = 5%
• When analyzing 10 metrics, probability of at least one false positive ≈ 40%
• When analyzing 20 metrics, this probability reaches ≈ 64%

False Discovery Rate (FDR) is the expected proportion of false discoveries among all significant results. FDR correction is a method that controls this indicator at a given level.

How It Works in AB-Labz Workbench

Before running analysis, you can select key metrics (up to 5), which will be analyzed as a single family.

Selecting Key Metrics

When configuring analysis, you will be offered to select key metrics:

What this means:

• Selected metrics form one "family" for statistical correction purposes
• FDR correction is applied only within this family
• Other metrics remain independent and are analyzed without correction

Selection recommendations:

• Select only those metrics on which you plan to make decisions
• Usually this is 1-3 key product metrics (for example, conversion, revenue, retention)
• Do not include auxiliary or exploratory metrics

Enabling Correction

In analysis settings there is a checkbox "p correction".

If enabled:

• Applies Benjamini-Hochberg (FDR) correction to selected key metrics
• This reduces the probability of "random" significant results when analyzing a group of metrics
• Correction is not applied to other metrics

If disabled:

• All metrics are analyzed independently
• Use if looking at metrics separately or correction is not needed

When to Use FDR Correction

Use correction when analyzing multiple key metrics simultaneously and making a decision based on a group of metrics. If you look at conversion, revenue, and retention together and are ready to implement a change when at least one metric improves, correction will protect you from random false positives.

Do not use correction if analyzing only one metric or metrics are independent and decisions are made separately. Correction is also not needed for auxiliary metrics used for diagnostics after the main decision is already made.

Interpreting Results with FDR Correction

After applying FDR correction, p-values of key metrics will be adjusted (usually increased).

Example:

Without correction:

• conversion: p = 0.03 (significant)
• revenue: p = 0.04 (significant)
• retention: p = 0.08 (not significant)

With FDR correction (3 metrics):

• conversion: q = 0.09 (NOT significant after correction)
• revenue: q = 0.09 (NOT significant after correction)
• retention: q = 0.12 (not significant)

Interpretation: Although conversion and revenue showed significance without correction (p < 0.05), after accounting for multiple testing, these results may be random. This protects you from premature conclusions.