What is Sample Size?
The minimum number of observations required to detect a true effect at a specified confidence level and statistical power before running a quantitative study.
Sample Size in statistics is the minimum number of observations required to detect a true effect of a specified magnitude with acceptable levels of confidence and statistical power. In product management, sample size calculations are performed before running A/B tests, user surveys, or usability studies to ensure that experiments are adequately powered to detect meaningful differences. Under-powered tests produce inconclusive results even when real effects exist.
Formula
n = (Z^2 x p(1-p)) / E^2For proportion tests: n = sample size per variant, Z = Z-score for desired confidence level (1.96 for 95%), p = baseline conversion rate, E = minimum detectable effect (absolute). For A/B tests with two variants: total sample = 2n. Example: baseline conversion = 5%, MDE = 1 percentage point, 95% confidence. n = (1.96^2 x 0.05 x 0.95) / 0.01^2 = approximately 1,825 per variant, 3,650 total.
Industry Benchmarks
- Standard A/B test: 80% statistical power and 95% confidence level are industry defaults
- Typical SaaS A/B test: 1,000 to 10,000 users per variant depending on baseline conversion rate
- Minimum detectable effect: 10-20% relative lift is a practical MDE for most product tests
- Survey sample size: 385 responses needed for 95% confidence with 5% margin of error in a large population
- Usability studies: 5 users reveal ~85% of usability issues (Nielsen's Law)
When to Use Sample Size
- Determining how long to run an A/B test before it accumulates sufficient data for a statistically valid decision
- Sizing survey campaigns to achieve representative samples at specified confidence levels
- Deciding whether current traffic volumes are sufficient to run an A/B test within an acceptable timeframe
- Communicating test design requirements to engineering and analytics teams before experiment setup
- Peeking at results before the pre-determined sample size is reached and stopping early when you see a promising result
- Setting an MDE that is too small (e.g. 0.1%) which requires enormous sample sizes and long test durations that are impractical
- Running A/B tests on pages with insufficient traffic, leading to tests that take months to conclude
- Always calculate sample size before starting the experiment, not after seeing the data
- For low-traffic pages, consider testing larger changes (redesigns) that produce bigger effect sizes rather than incremental tweaks
- Use a sequential testing approach with appropriate corrections if you need the ability to peek at results during the experiment
Related Terms
Free Sample Size Calculator
Skip the spreadsheet. Enter your numbers in the free Sample Size Calculator and get a benchmarked Sample Size result in seconds.