Statistical Significance in A/B Testing

In an A/B test, one version can appear to win simply because the test observed a limited and variable sample. A significance test compares the data with a specified null hypothesis, usually that there is no underlying difference. A p-value indicates how unusual a result at least as extreme would be under the assumed null model. It does not tell us the probability that the winning version is truly better.

A common 5% significance threshold is a decision convention, not a guarantee that a finding will reproduce. The choice of metric, random assignment, sample size and stopping rule all matter. Repeatedly checking and ending a test as soon as it appears significant can inflate false positives.

Statistical significance is separate from practical significance. A tiny uplift may reach a low p-value with enough traffic but still be too small to justify the cost of implementation. Report effect sizes and uncertainty alongside any hypothesis-test result.