Two ways to run an experiment. A/B compares full variants. MVT tests element combinations. The choice usually comes down to traffic.
Last updated: 2026-04-01
A controlled experiment with two or more full variants: control versus one or more challengers. Each variant gets a slice of traffic. The winner beats the control with statistical significance.
Best for testing major page-level or flow-level changes. Reach for A/B when you want to know "does this version work better than the current one".
An experiment that tests combinations of element-level changes simultaneously. A 2x2 MVT (two headlines, two CTAs) creates 4 combinations. A 2x2x2 creates 8. Each combination gets a slice of traffic.
Best when you need to know how elements interact. Useful only at high traffic where every combination can be properly powered.
n per variant ~ (Z_alpha/2 + Z_beta)^2 x (p1(1-p1) + p2(1-p2)) / (p1 - p2)^2Defaults: alpha = 0.05, power = 0.80, MDE = whatever effect size matters to your business. Lower MDE means much higher sample size.
n per combination ~ (same A/B formula) x (number of combinations)Double your variables, quadruple your traffic. A 2x2x2 MVT needs roughly 4x the traffic of an A/B/n test with the same MDE.
| Criteria | A/B Test | Multivariate (MVT) |
|---|---|---|
| What it tests | Two or more full variants | Combinations of element-level changes |
| Sample size needed | Standard for the MDE | A/B sample size x number of combinations |
| Time to ship | Days to weeks | Weeks to months |
| Detects interactions | No | Yes |
| Best fit | Most experiments | High-traffic critical surfaces |
| Risk of under-powering | Low at standard traffic | High unless traffic is very high |
| Typical use ratio | Most teams run 10 A/B tests per 1 MVT | Reserved for highest-value pages |
| Decision output | Winning variant | Best combination + element-level effects |
Pros
Cons
Pros
Cons
Score your own data with both frameworks. Compare results and pick the one that fits your team.
When traffic is high enough to power every combination, you're optimizing a high-value surface, and you need to know how elements interact. Outside those conditions, sequential A/B tests are usually faster to a decision and cheaper in traffic.
Roughly 4x for a 2x2 design, 8x for a 2x2x2, and so on. Rule of thumb: take your A/B sample size and multiply by the number of combinations. If you don't have that traffic, MVT will under-power and the result is noise.
Only if you have the traffic. The trap: teams think MVT replaces ten sequential A/B tests. In practice, a well-designed MVT can substitute for one or two of those tests, not all ten. The other tests still need to run sequentially.
At minimum, run a test through one weekly cycle so you capture both weekday and weekend behavior. For B2B, two weeks is common to cover holidays and stakeholder schedules. For seasonal products, longer.
Probably not, unless your homepage has very high conversion-rate-relevant traffic. Most homepages benefit more from sequential A/B tests on the headline, then the CTA, then the layout. MVT on a homepage usually under-powers.