Terminology

A brief glossary of key AB-Labz Workbench terms.

Hypothesis

An assumption that a specific change in the product will lead to an improvement in a key metric. Includes the name of the change, hypothesis text, target metric, and expected effect.

Experiment

A controlled test of a hypothesis in which the audience is randomly divided into groups to compare different product variants. Can be two-variant (A/B test) or multivariate (A/B/C/... test).

Randomization Unit

The entity by which division into experiment groups occurs (usually a user, but can be a session or device).

Sample Size

The minimum number of randomization units (users) required to achieve statistically significant results. Calculated based on the baseline metric value, expected change, and statistical power.

Control Group (Control)

A group of users who see the current version of the product without changes. In notation, this is variant A or simply Control.

Treatment Groups (Treatment)

Groups of users who see new versions of the product with changes:

• In an A/B test (2 variants): one treatment group — variant B (Treatment)
• In multivariate tests: several treatment groups — variants B, C, D, etc. (all of them are called Treatment)

Statistical Significance

Confidence that the observed difference between groups is not random. Usually measured by p-value: the result is significant when p < 0.05.

P-value

The probability of obtaining the observed or more extreme difference between groups if, in reality, the change has no effect.

Confidence Interval

A range of values within which the true value of the effect falls with 95% probability. Helps assess not only the presence of the effect but also its possible magnitude.

Baseline Value (Baseline)

The current level of the metric before making changes, used to calculate sample size and evaluate the effect.

Minimum Detectable Effect (MDE)

The minimum change in the metric that needs to be detected in the experiment to make a decision.

Statistical Power (Power)

The probability of detecting an effect if it actually exists. Usually set at 80%.