What is Sample Size?
Sample size is the number of observations in a study. Before an A/B test, calculate the sample needed for your target effect size, significance level, and statistical power. An underpowered test may miss a real effect. Survey sample planning also needs to account for the precision you require.
Formula
Survey proportion estimate: n = (Z^2 x p(1-p)) / E^2This formula estimates the sample for a survey proportion in a large population under simple random sampling. Z is 1.96 for 95% confidence, p is the expected proportion, and E is the margin of error as a decimal. With p = 0.5 and E = 0.05, round up to 385 responses. A/B sample planning uses a different calculation that also accounts for statistical power: how likely the test is to detect an improvement of the size you planned for.
Benchmarks and interpretation
- Choose the sample size for the question, test method, baseline rate, and effect size or precision you need
- 80% power and a 5% significance level are common A/B planning choices, not requirements for every test
- The 385-response survey example assumes simple random sampling, a large population, and p = 0.5; more responses do not fix selection bias
- Choose a minimum detectable effect that matters to the business, then check whether you have enough traffic to study it
When to use Sample Size
- Determining how long to run an A/B test before it accumulates sufficient data for a statistically valid decision
- Sizing survey campaigns for the desired precision while planning how to recruit a representative sample
- Deciding whether current traffic volumes are sufficient to run an A/B test within an acceptable timeframe
- Communicating test design requirements to engineering and analytics teams before experiment setup
- Peeking at results before the pre-determined sample size is reached and stopping early when you see a promising result
- Choosing a very small MDE without checking the sample and duration it requires
- Running A/B tests on pages with insufficient traffic, leading to tests that take months to conclude
- Always calculate sample size before starting the experiment, not after seeing the data
- For low-traffic pages, consider testing larger changes (redesigns) that might produce bigger effects rather than incremental tweaks
- Use a sequential testing approach with appropriate corrections if you need the ability to peek at results during the experiment
Related terms
Free Sample Size Calculator
Enter your inputs in the free Sample Size Calculator and review the result alongside its guidance.