A/B testing for product managers

It depends on baseline conversion, the minimum detectable effect (MDE), significance level, and desired power. At a 5% baseline, 5% relative MDE, 95% confidence, and 80% power, a two-sided test needs 120,146 visitors per variant in this calculator’s model. Smaller effects need larger samples. Enter your assumptions in the A/B planning calculator.

Plan enough time to reach the target sample and cover relevant business cycles. Seven to fourteen days can cover weekday and weekend patterns, but may be insufficient for low traffic or slower conversion. Define the sample and duration before starting; a significant result alone is not a stopping rule for a fixed-horizon test.

Pre-test planning estimates the sample and duration needed to detect the effect you chose. Post-analysis uses observed group sizes and outcomes to estimate lift, uncertainty, and a p-value. These calculations help assess the evidence; they do not establish that an effect is useful or that the experiment was run correctly.

Repeatedly checking a fixed-horizon test and stopping when it looks significant can raise the false-positive rate. You can monitor safety and data quality, but make the final inference at the planned sample and duration. Alternatively, use a sequential method with a stopping rule designed for repeated checks; a Bayesian label alone does not establish that property.

MDE is the effect size you plan the test to detect with the chosen power. Pick a change large enough to matter to the decision, and distinguish relative lift from percentage points. A smaller MDE needs more traffic; do not increase it merely to make the required sample fit.

Check the confidence interval and whether the sample can detect the effect you planned for. A result near zero may rule out a useful improvement, or it may remain too uncertain to decide. Review prespecified secondary and guardrail metrics; treat new segment patterns as exploratory findings to test.

Yes, but plan the comparisons and account for multiple testing. For example, Bonferroni divides the significance threshold by the number of planned comparisons. This calculator supports two-arm comparisons; use a method appropriate to your outcome and multi-arm design for a larger experiment. ANOVA or sequential testing does not automatically solve every comparison problem.

A p-value describes how unusual the result, or a more extreme result, would be if there were no difference and the test assumptions held. A two-sided p-value of 0.04 includes extremes in either direction; it is not a 96% chance the variant is better. Read it with the effect estimate and confidence interval.