A/B Significance

After your A/B test ends, did the variant actually beat control, or did you see noise?

When to use this

Your test has finished its full planned duration. You have conversion counts and visitor counts for each variant. You want to know whether the difference is real or could be explained by chance.

When NOT to use this

The test stopped early because results "looked good." Reading significance on a peeked test is misleading -- the p-value math assumes one check at the end. Tests with fewer than ~100 conversions per variant (noise dominates; the result isn't meaningful even if "significant"). Tests where the metric isn't binary (revenue per user, session length) -- those need a different test, not a two-proportion z-test.

Inputs

Visitors per variant: How many people saw each version.
Conversions per variant: How many of them converted (signed up, bought, clicked -- whatever your primary metric is).
Significance level (alpha): Usually 0.05 (5%). Don't change this mid-test to make a result look better.
Test type: One-tailed (you only care about improvements) or two-tailed (you care about differences in either direction). Two-tailed is the standard.

The math

Two-proportion z-test. The p-value answers a narrow question: if there were actually no difference between A and B, what's the chance you'd see a gap this big or bigger from random variation alone?

The full derivation:

p_control   = conversions_control / n_control
p_treatment = conversions_treatment / n_treatment
 
p_pooled = (conversions_control + conversions_treatment) / (n_control + n_treatment)
 
SE = sqrt(p_pooled x (1 - p_pooled) x (1/n_control + 1/n_treatment))
 
z = (p_treatment - p_control) / SE
 
p-value (two-tailed) = 2 x (1 - CDF(|z|))

If p-value < alpha, you reject the null hypothesis that the variants are equal. A p-value of 0.04 means there's a 4% chance you'd see this result if the variants were truly equal. Below 0.05 is the convention for "significant." It is not proof.

The confidence interval for the difference, at 95% confidence:

CI = (p_treatment - p_control) +/- 1.96 x SE

The confidence interval is more useful than the p-value. If the CI is "+2pp to +18pp," you have a real effect with uncertainty about its size. If the CI crosses zero ("-3pp to +12pp"), you can't rule out that the variant did nothing.

A worked example

You ran a test. Here are the results:

Variant	Visitors	Conversions	Conversion rate
A (control)	12,000	504	4.20%
B (variant)	11,800	555	4.70%

Step through the math:

p_control = 504 / 12,000 = 0.0420
p_treatment = 555 / 11,800 = 0.0470
 
p_pooled = (504 + 555) / (12,000 + 11,800) = 1,059 / 23,800 = 0.0445
 
SE = sqrt(0.0445 x 0.9555 x (1/12,000 + 1/11,800))
   = sqrt(0.0425 x 0.0001680)
   = sqrt(0.00000714)
   = 0.00267
 
z = (0.0470 - 0.0420) / 0.00267 = 1.87
 
p-value (two-tailed) = 2 x (1 - CDF(1.87)) = ~0.062

Relative lift: (4.70 - 4.20) / 4.20 = 11.9%.

95% CI on the absolute lift: 0.0050 +/- 1.96 x 0.00267 = [-0.0003, +0.0103], or roughly -0.03pp to +1.03pp.

Verdict: NOT significant at 95% (p > 0.05). And more usefully, the CI crosses zero. The test can't rule out "no effect."

The right call is to either run longer to tighten the CI, or accept that you don't have evidence and don't ship.

How pmtoolkit does it differently

We surface the confidence interval first, p-value second. "p = 0.04" and "p = 0.06" feel binary but the CI shows the underlying range. We also flag tests that were stopped before reaching their planned sample size as suspect -- peeking is the single most common way bad calls get made. The calculator shows statistical and practical significance side by side, and flags when they diverge (a "significant" 0.1% lift, or a 5% observed lift that was just underpowered).

Common mistakes

Treating p < 0.05 as proof. It's evidence, not truth. Roughly 1 in 20 "significant" results is a false positive by construction.
Reporting relative lift without the CI. "+12% lift" sounds great until you see the CI is -1% to +25%.
Stopping the test the moment significance hits. Peeking inflates your false positive rate by 3-5x.
Concluding a test "failed" when it was underpowered. If your sample couldn't detect the effect you cared about, a non-significant result tells you nothing.
Running multiple variants without correcting for multiple comparisons. Test 5 variants at p < 0.05 and your chance of at least one false positive is ~23%. Apply a Bonferroni correction or use a method built for multi-arm tests.
Ignoring practical significance. "Statistically significant but a 0.1% lift on a feature that took 6 weeks to build" is not a win.

A "significant" result that contradicts your intuition warrants a re-run, not a celebration. Most "wins" don't replicate. Industry replication studies tend to find 30-50% of declared A/B wins fail to repeat (illustrative; check your own org's history).