A/B Significance

Estimate the difference between variants and the uncertainty around it after a planned test.

When to use this

Your test has finished its full planned duration. You have conversion counts and visitor counts for each variant. You want to know whether the difference is real or could be explained by chance.

Limits

This two-proportion z-test assumes independent observations and enough successes and failures for a normal approximation. Sparse counts may require an exact method. Continuous outcomes need a different test. If the experiment stopped when significance appeared, its p-value does not account for that repeated stopping rule.

Inputs

Visitors per variant: Independent users or other units assigned to each version.
Conversions per variant: How many assigned units met the binary outcome, counted at most once per unit.
Significance level: Choose before the test, commonly 0.05.
Test type: Two-sided checks differences in either direction. A one-sided test must specify its direction before observing results; the toolkit's one-sided option tests an improvement.

The math

Two-proportion z-test. The p-value answers a narrow question: if there were actually no difference between A and B, what's the chance you'd see a gap this big or bigger from random variation alone?

The full derivation:

p_control   = conversions_control / n_control
p_treatment = conversions_treatment / n_treatment
 
p_pooled = (conversions_control + conversions_treatment) / (n_control + n_treatment)
 
SE = sqrt(p_pooled x (1 - p_pooled) x (1/n_control + 1/n_treatment))
 
z = (p_treatment - p_control) / SE
 
p-value (two-tailed) = 2 x (1 - CDF(|z|))

If p-value < alpha, you reject the null hypothesis that the variants are equal. A p-value of 0.04 means results this extreme or more extreme would occur 4% of the time under the no-difference assumption and the test's conditions. Below 0.05 is the convention for "significant." It is not proof.

The confidence interval for the difference, at 95% confidence:

SE_difference = sqrt(p_control x (1-p_control) / n_control
                     + p_treatment x (1-p_treatment) / n_treatment)
CI = (p_treatment - p_control) +/- 1.96 x SE_difference

The interval uses the unpooled standard error, unlike the pooled error used for the test statistic. An interval of "+2pp to +18pp" supports a positive difference but leaves substantial uncertainty about its size. If it crosses zero, such as "-3pp to +12pp," the data remain compatible with no difference.

A worked example

You ran a test. Here are the results:

Variant	Visitors	Conversions	Conversion rate
A (control)	12,000	504	4.20%
B (variant)	11,800	555	4.70%

Using unrounded rates for the calculation:

p_control = 504 / 12,000 = 0.042
p_treatment = 555 / 11,800 = 0.0470339
p_pooled = 1,059 / 23,800 = 0.0444958
SE_pooled = 0.00267321
z = 0.0050339 / 0.00267321 = 1.88309
p-value (two-sided) = about 0.0597

Relative lift is about 12.0%. The unpooled 95% interval for absolute lift is approximately [-0.0002075, 0.0102753], or -0.021 to +1.028 percentage points.

Verdict: NOT significant at 95% (p > 0.05). And more usefully, the CI crosses zero. The test can't rule out "no effect."

Because the planned test has ended, report the result as inconclusive at this threshold. Decide whether to plan a new test or act on other evidence; extending it solely because the result missed significance changes the stopping rule.

In PM Toolkit

The calculator reports rates, absolute and relative lift, a p-value, and an approximate confidence interval. It cannot determine from counts alone whether randomization, tracking, or stopping rules were valid. Review those conditions before using the output.

Common mistakes

Treating p below 0.05 as proof. Under a true null and valid test assumptions, alpha describes the long-run false-positive rate; it is not the fraction of significant findings that are false.
Reporting relative lift without uncertainty or the baseline rate.
Stopping repeatedly on significance without a sequential method.
Treating a non-significant result as proof of no effect. Check which effects the interval still allows.
Comparing many variants without a suitable multiplicity adjustment. Five independent null tests at alpha 0.05 give 1 - 0.95^5, about a 23% chance of at least one false positive; shared-control tests need not be independent.
Treating a statistically detectable effect as automatically worth shipping. Review costs, risks, and business value.

NIST documents the difference-of-proportions interval. Investigate unexpected results with tracking checks and follow-up evidence rather than assuming either the result or intuition must be right.