Two ways to run an experiment. A/B compares full variants. MVT tests element combinations. The choice usually comes down to traffic.
Last updated: 2026-04-01
A controlled experiment with two or more full variants: control versus one or more challengers. Each variant gets a random share of eligible users. Compare the results against the effect size and decision rules you set before the test.
Best for testing major page-level or flow-level changes. Reach for A/B when you want to know "does this version work better than the current one".
An experiment that tests combinations of element-level changes simultaneously. A 2x2 MVT (two headlines, two CTAs) creates 4 combinations. A 2x2x2 creates 8. Each combination gets a slice of traffic. This is a full factorial design.
Best when you need to know how elements interact. Plan enough traffic for the effects and interactions you want to detect.
n per variant ~ (Z_(1-alpha/2) + Z_power)^2 x (p1(1-p1) + p2(1-p2)) / (p1 - p2)^2Approximation for two independent conversion rates with equal allocation. Example settings: alpha = 0.05 and power = 0.80. The Z terms are standard-normal quantiles; a smaller MDE needs more users.
Total sample = sample per combination x number of combinationsA 2x2 design has 4 combinations; a 2x2x2 has 8. Plan sample size for the intended comparisons and interactions, including any multiple-testing adjustment.
| Criteria | A/B Test | Multivariate (MVT) |
|---|---|---|
| What it tests | Two or more full variants | Combinations of element-level changes |
| Sample size needed | Standard for the MDE | Depends on effects, interactions, and comparisons planned |
| Duration planning | Planned sample plus relevant usage cycles | Enough data for combinations and interactions |
| Detects interactions | Not separately within a bundled variant | Yes, with a suitable design and analysis |
| Best fit | Most experiments | High-traffic critical surfaces |
| Risk of under-powering | Depends on sample size and effect size | Depends on the number of combinations and planned effects |
| Planning focus | Compare complete variants | Estimate element effects and interactions |
| Decision output | Winning variant | Best combination + element-level effects |
Pros
Cons
Pros
Cons
Use your own inputs to explore the calculations and compare results.
Use MVT when you need to understand how elements interact and can collect enough data for that analysis. A/B testing is simpler when the decision is which complete version to use.
There is no fixed multiplier. A 2x2 design has four combinations and a 2x2x2 has eight. If each cell needs the same sample as one A/B arm, those totals are two and four times a two-arm A/B test. Interaction tests and multiple comparisons may need different sample sizes.
Yes, when those tests concern factors you can vary together. A full factorial design tests every combination and can estimate interactions. It does not replace experiments about unrelated questions.
Cover the usage patterns that matter to the decision, as well as the planned sample size. For a product with weekday and weekend differences, that includes a full week. Longer purchase or usage cycles may need more time.
Use it if you need to study combinations of headlines, CTAs, or layouts and have enough traffic for the planned effects. Otherwise, an A/B test of complete versions may answer your decision more directly.