A/B Sample Size

How many visitors per variant you need to detect a real lift, and how long that will take at your traffic.

When to use this

You're planning an A/B test and want to know whether it can even work before you start. You have a current conversion rate and a clear minimum lift you'd care about. You know roughly how much traffic the surface gets per week.

When NOT to use this

Low-traffic surfaces (fewer than 1,000 conversions per variant per month). Changes where you can't pick a single primary metric. Qualitative redesigns where the goal is "feels better" -- A/B testing is the wrong tool. Most companies don't have the traffic to run good A/B tests on most things. Be honest about that first.

Inputs

Baseline conversion rate: Where you are now. Pull this from the last 30-60 days of data, not your intuition.
Minimum detectable effect (MDE): The smallest lift you'd actually care about. If a 0.1% absolute lift wouldn't change a decision, don't set MDE to 0.1%. Set it to what matters.
Statistical power (1 - beta): The chance you'll detect a real effect if one exists. 80% is the default (means a 20% chance you miss a real win). Use 90% for high-stakes decisions.
Significance level (alpha): The chance you'll call a noise pattern a win. 5% is the default. Use 1% for irreversible decisions or large financial exposure.

The math

The exact formula uses the normal distribution and isn't practical by hand:

n per variant = f(alpha, power) x [p1(1 - p1) + p2(1 - p2)] / (p1 - p2)^2

Where p1 is your baseline rate, p2 is p1 + MDE, and f(alpha, power) combines the z-scores for your chosen alpha and power.

The standard approximation at 80% power and 5% significance is the rule of 16:

n per variant = 16 x p x (1 - p) / MDE^2

The 16 comes from combining the z-scores for 80% power (~0.84) and 95% significance (~1.96): (0.84 + 1.96)^2 = 7.84, doubled because you have two variants. This gets you close enough for planning. The calculator runs the exact formula.

Multiply n per variant by the number of variants (including control) for total traffic needed.

A worked example

You're testing a checkout change. Current conversion = 4%. You'd care about a 0.5 percentage-point absolute lift (4% to 4.5%, a 12.5% relative lift).

n = 16 x 0.04 x 0.96 / (0.005)^2
n = 16 x 0.0384 / 0.000025
n = 24,576 per variant

You get 5,000 visitors per variant per week. So:

24,576 / 5,000 = ~5 weeks

Now check what happens if you halve the MDE to 0.25 percentage points:

n = 16 x 0.04 x 0.96 / (0.0025)^2 = 98,304 per variant

Four times the sample for half the effect size. That's the standard tradeoff: smaller effects need exponentially more data. Most teams underpower their tests because they want to detect small effects but run for a fixed week regardless of the math.

If your stakeholder wants an answer in 2 weeks, the test can't deliver a 0.5pp detection. The choice is: accept a bigger MDE, wait the 5 weeks, or skip the test and ship the change behind a flag.

How pmtoolkit does it differently

The calculator shows duration in weeks at your actual traffic, not just the abstract sample size. "You need 24k per variant" is useless if you don't know whether that's 2 weeks or 6 months. We surface sample size at three MDE levels simultaneously so the tradeoff between sensitivity and duration is visible. We also flag tests that need more than 8 weeks as practically unrunnable -- by then your product has changed, your traffic mix has shifted, and the test is measuring something other than what you set out to measure.

Common mistakes

Running for "two weeks" without computing sample size. You're either underpowered or burning traffic you didn't need to.
Peeking at results mid-test and stopping when significance hits. Peeking drives false positives. The math assumes you check once, at the end.
Setting MDE based on what you'd like to find, not what's plausible. A 30% lift on a checkout flow is rare. Plan for what's likely, not what's hoped.
Exposing only 10% of traffic to "be cautious" and forgetting to adjust the sample math. Your effective traffic for sample-size purposes is 10% of the total. Plan accordingly.
Calculating sample size after the test (post-hoc power analysis). Methodologically invalid. Calculate before, stick to the plan.
Using the rule-of-16 for non-binary metrics. It assumes proportions. Continuous metrics (revenue per user, time on page) need a different formula and usually more data.
Ignoring novelty effects in the first 7 days. Users behave differently when something looks new. Either exclude the first week or run long enough that novelty washes out.