A/B Sample Size

Estimate visitors per variant for a planned effect size, then estimate duration at your available traffic.

When to use this

You're planning an A/B test and want to know whether it can even work before you start. You have a current conversion rate and a clear minimum lift you'd care about. You know roughly how much traffic the surface gets per week.

Limits

This calculation is for two independent groups and a binary outcome. Continuous outcomes, multiple variants, clustered assignment, and sequential testing need suitable methods. Low traffic may make the planned effect impractical to detect; calculate the requirement rather than applying a fixed monthly-conversion cutoff.

Inputs

Baseline conversion rate: Use recent data representative of the eligible test population.
Minimum detectable effect (MDE): The improvement size you plan to detect. Distinguish percentage points from relative percent.
Statistical power: How likely the test is to detect that planned-size improvement. 80% is a common choice; 90% requires more observations.
Significance level (alpha): The false-positive rate under the no-effect assumption and the test's conditions. 5% is common; choose it before starting.
Eligible traffic: Account for the share assigned to the test and its split between groups.

The math

The calculator uses this normal approximation for equal allocation:

p1 = baseline rate
p2 = p1 + absolute MDE
n = ceil((z_alpha x sqrt(2 x p1 x (1-p1))
          + z_power x sqrt(p1 x (1-p1) + p2 x (1-p2)))^2
         / (p2-p1)^2)

Rates and absolute MDE are decimals. For a two-sided test, z_alpha is the normal quantile at 1 - alpha/2; z_power is the quantile at the chosen power. This is an approximation, not an exact test.

At 80% power and 5% two-sided significance, a simpler planning approximation is the rule of 16:

n per variant = 16 x p x (1-p) / MDE^2

The factor is approximately 2 x (0.84 + 1.96)^2 = 15.68, rounded to 16. The example below uses that rule, so its result differs from the calculator. For two variants, total required traffic is twice the per-variant sample. Adding variants also requires planning for multiple comparisons.

A worked example

You're testing a checkout change. Current conversion = 4%. You'd care about a 0.5 percentage-point absolute lift (4% to 4.5%, a 12.5% relative lift).

n = 16 x 0.04 x 0.96 / (0.005)^2
n = 16 x 0.0384 / 0.000025
n = 24,576 per variant

You get 5,000 visitors per variant per week. So:

24,576 / 5,000 = ~5 weeks

Now check what happens if you halve the MDE to 0.25 percentage points:

n = 16 x 0.04 x 0.96 / (0.0025)^2 = 98,304 per variant

The calculator returns 24,539 per variant for the original 4% to 4.5% example; the rule-of-16 result above is 24,576.

Four times the sample for half the effect size. That's the standard tradeoff: the required sample grows with the inverse square of the effect size in this approximation.

At this traffic level, two weeks would not meet the planned sample. Consider a larger decision-relevant MDE, more eligible traffic, a longer test, or a different research method.

In PM Toolkit

The sample-size calculator reports the requirement and estimates duration from traffic inputs. Its feasibility labels are planning prompts, not statistical rules that make a long test invalid. Review whether the traffic, product, and measurement conditions can remain suitable for the planned duration.

Common mistakes

Choosing a fixed end date without checking whether it supplies enough observations.
Stopping a fixed-horizon test when the p-value first falls below alpha. Use the stopping rule chosen in advance.
Forgetting to reduce traffic estimates when only part of the audience joins the test.
Treating post-hoc power as an explanation of the result. Report the observed effect and uncertainty instead.
Applying this proportions formula to revenue, time, or other continuous measures.
Discarding the first week after seeing the data. Plan any ramp-up or novelty analysis in advance.

NIST's sample-size discussion explains how alpha, power, and effect size enter proportion-test planning.