A/B testing for product managers

It depends on three things. Your baseline conversion rate, your minimum detectable effect (MDE), and the statistical power you want. The defaults most teams use are 95% confidence and 80% power. For a 5% baseline conversion and a 5% relative MDE, you're looking at roughly 125,000 visitors per variant. Smaller MDE means much bigger samples. The fastest way to size a real test is to plug your baseline and target lift into the Sample Size Calculator and the A/B Test Calculator's planning mode.

Run it for at least one full business cycle, and ideally two. For most consumer products and SaaS sites, that's 7 to 14 days. This isn't about hitting significance. It's about making sure your sample sees the same mix of weekdays, weekends, and traffic sources. A test that ran Monday to Friday only saw weekday users. If you ship on that, you'll be surprised on Saturday.

Pre-test planning answers "what sample do I need to detect a real effect?" You set baseline, MDE, alpha, and power, and the calculator tells you how many users per variant and how long it'll take. Post-analysis answers "did the effect I saw really happen, or is it noise?" You feed in actual visitors and conversions per arm and read the p-value, confidence interval, and lift. Both modes live inside the A/B Test Calculator.

Your false-positive rate goes up. A lot. Peeking at a fixed-horizon test ten times can turn a nominal 1% significance into a real 5% rate. Researchers like Evan Miller have written about this for years. Two ways out: commit to a sample size before you start and don't read the test until you hit it, or use sequential testing (Bayesian or frequentist) that's designed to allow peeking. Most A/B platforms do not adjust for peeking by default.

MDE is the smallest lift you'd care about detecting. Pick it from the business side, not the stats side. If a 2% lift wouldn't change anything in your roadmap, don't size a test to detect 2%. Size for the effect that would actually trigger a decision. Smaller MDEs need exponentially larger samples, so honest MDE-setting saves weeks.

Flat is information. Most product changes don't move the metric you tested. The right next steps are: check whether the test was actually powered to detect the effect you cared about, look at downstream metrics to see if anything else moved, and check the funnel by step in the Conversion Rate Calculator. A flat headline number can hide a real shift earlier or later in the journey.

You can, but the math gets harder. With three or more arms you need to correct for multiple comparisons, or you're inflating false positives a different way. The simple version: pick the metric, pick one comparison you care about most (usually each variant against control), and apply a Bonferroni correction. The A/B Test Calculator covers two-arm tests well. For more arms, ANOVA or sequential designs are the right call.

A p-value is the probability of seeing the result you got (or more extreme), given that nothing actually changed. It's not the probability that your variant won. P = 0.04 doesn't mean you're 96% likely to be right. It means: if there were no real effect, you'd see this lift or bigger 4% of the time by chance. Pair every p-value with a confidence interval on the lift, and decide based on the size of the effect, not the size of the p-value.