Multivariate Testing: Beyond A/B

Test multiple variables together to estimate their individual effects and how they interact.

Why test interactions?

Suppose you want to change a landing page’s headline, hero image, CTA button, and form length. Testing each in isolation may miss interactions. An urgent headline, for example, might work with a red button but perform poorly with a blue one. A multivariate design lets you compare combinations.

Interaction effects can change which combination wins. Testing variables independently won't reveal those effects¹, so you could miss a better-performing combination.

How multivariate testing works

What is MVT?

Multivariate testing examines multiple variables in the same experiment. It can estimate their individual effects and interactions. A full factorial design tests every combination of the selected levels.

MVT vs A/B Testing

Factorial Design Basics

A full factorial design includes every combination of factor levels.²

2x2 Factorial (2 variables, 2 variants each):

Variable A: Headline (Original vs New)
Variable B: Button Color (Blue vs Red)

Combinations:
1. Original Headline + Blue Button
2. Original Headline + Red Button
3. New Headline + Blue Button
4. New Headline + Red Button

2x2x2 Factorial (3 variables): Eight total combinations. If each combination needs the same number of observations, total sample size is eight times the per-combination requirement. Plan that requirement for the effects and comparisons you intend to test.

When Variables Combine: Main vs Interaction Effects

Main Effect: The average impact of a single variable. For example, a red button might increase conversion by 5% overall.

Interaction Effect: When the effect of one variable changes depending on the level of another variable. For example, a red button might decrease conversion by 2% when paired with an urgent headline, but increase it by 10% when paired with a calm headline.

Hypothetical example: combining button and headline

Suppose a test compares headlines ("Limited Time!" vs "Learn More") and button colors (Red vs Blue). The following figures illustrate relative changes against the calm-headline, blue-button baseline.

An initial additive expectation:

• Urgent headline: +5% conversion
• Red button: +3% conversion
• Together: +8% conversion (5% + 3%)

Hypothetical observed lifts:

• Urgent + Red: -2%
• Calm + Red: +7%
• Urgent + Blue: +6%

The combined result differs from the separate changes, suggesting an interaction worth testing statistically. These conversion rates alone cannot establish whether anxiety, trust, or another response explains it.

Sample Size: The Traffic Reality Check

Plan the sample

Multiply the number of levels for each variable to count the combinations in a full factorial design. A 2×2 test has four combinations; three variables with two levels each have eight. Each additional two-level variable doubles that count.

If each group needs 5,000 visitors, an A/B test needs 10,000 in total and a 2×2 test needs 20,000. The per-group requirement depends on the effect size, power, and comparisons you intend to test; these example counts are not universal minimums.

How the number of combinations grows

With two levels per variable:

• One variable: 2 combinations
• Two variables: 4 combinations
• Three variables: 8 combinations
• Four variables: 16 combinations

Check the traffic and analysis requirements before adding another variable.

Check feasibility

Divide the eligible traffic by the number of combinations to estimate the observations available to each group under equal allocation. Compare that figure with a sample-size calculation for your chosen effects, power, and analysis plan. If the required duration is impractical, reduce the number of variables or consider an A/B test.

Try It Now

See how sample sizes grow:

Sample Exercise

Planning MVT for checkout page:

• Variables: 3 (Form Length, Trust Badges, Express Checkout)
• Baseline: 10% conversion
• Target: 20% relative lift (10% → 12%)
• Combinations: 2×2×2 = 8

At 95% confidence and 80% power, PM Toolkit's two-sided, two-arm planning approximation gives 3,623 users per arm for 10% versus 12%. Allocating that many to each of eight combinations would total 28,984 users. This is a traffic illustration, not a powered MVT design: it does not adjust for multiple comparisons or establish power for interactions. Calculate those requirements from the factorial analysis plan before launch.

Obama 2008: testing media and button text

The campaign tested six media options and four button labels, producing 24 combinations. The family photo with "Learn More" increased email sign-ups by 40.6% over the original. Dan Siroker estimated that the additional sign-ups contributed $60 million in donations over the campaign.³ The measured outcome was sign-ups; the donation figure was extrapolated. The account illustrates a combination test, but does not report a formal interaction estimate.

Advanced: When You're Ready for More

Fractional factorial designs

A fractional factorial design tests a selected subset of combinations. It reduces the number of conditions, but some effects become indistinguishable from others, a problem called aliasing.

For example, a four-run half-fraction of a three-factor, two-level design aliases each main effect with a two-factor interaction. It cannot separately estimate both without additional assumptions or runs⁴. Check which effects the design can distinguish before choosing it; there is no general “70% of the insight” rule.

Taguchi Methods (For the Ambitious)

Taguchi methods use orthogonal arrays and assumptions about which effects matter.⁵ Choose a design according to the effects you need to estimate, rather than a fixed minimum number of variables.

Common Pitfalls

1. Starting Too Complex

Mistake: Testing 5 variables immediately. Fix: Start with a manageable design, such as 2×2, and add factors only when the traffic and analysis plan support them.

2. Ignoring Interactions

Mistake: Only analyzing main effects. Fix: Always create interaction plots. That's why you chose MVT.

3. Insufficient Sample

Mistake: Using A/B test sample size. Fix: Calculate MVT requirements first. Be realistic.

4. Confounding factor levels

Mistake: Changing font size and line height together without including combinations that separate their effects. Fix: Vary the factors according to the design. Related choices can be tested together when the combinations are feasible.

5. Analysis Paralysis

Mistake: Overwhelmed by 16 combinations. Fix: Follow the planned comparisons and account for selecting a winner from several combinations. Do not treat the largest observed lift as an unbiased estimate.

AI Prompts for MVT

Design MVT Test

Design a multivariate test for: [list variables]
- Current conversion: [X%]
- Monthly traffic: [Y visitors]
- Test duration: [Z weeks]
Recommend:
1. Full vs fractional design
2. Sample requirements
3. Expected power for effects

Analyze Results

Analyze these MVT results: [paste data]
Identify:
1. Strongest main effects
2. Significant interactions
3. Optimal combination
4. Surprising findings
5. Implementation recommendations

Calculate Sample Size

Calculate MVT sample size:
- Variables: [number and levels]
- Baseline rate: [X%]
- Target effect: [Y%]
- Power: [Z%]
Provide:
1. Total sample needed
2. Sample per combination
3. Test duration at [traffic]/day
4. MVT vs A/B recommendation

Other Advanced Techniques (Optional Reading)

Response Surface Methodology: Models continuous variables (like font size) to find optimal points on curves.

Adaptive MVT: Automatically shifts traffic to winning combinations during the test.

Bayesian MVT: Updates estimates as data arrives. Set out the model, initial assumptions, and rules for making a decision before interpreting results.

Sequential Strategy: Start with screening, then deep-dive on winners.

Your MVT Decision Tree

Should I use MVT?

1. Can eligible traffic support the sample size for your planned effects and comparisons within a useful period? If not, reduce the design or reconsider the test.
2. Do you need to separate the effects of multiple factors or estimate their interactions? If you only need an overall comparison between two designs, an A/B test may be enough.
3. Can you implement the combinations and analyze the results as planned? Resolve those requirements before launching.

Quick Reference Guide

The Hybrid Approach

Phase 1: A/B test for big swings (new design vs old) Phase 2: MVT to optimize the winner (fine-tune elements)

Analyzing Results

Analysis Framework

Step 1: Validate Test

• Sample size achieved?
• Do group sizes match the traffic split you planned?
• No technical issues?

Step 2: Main Effects

• Average performance per variable
• Which variables drive change?

Step 3: Interactions

• Create interaction plots
• Look for non-parallel lines and estimate interaction uncertainty
• Crossovers indicate dependencies

Step 4: Find Optimal

• Rank all combinations
• Account for uncertainty and multiple comparisons⁶
• Consider implementation cost

Step 5: Generate Insights

• Ask why combinations work
• Apply learnings to future tests

Action Items

Start Here (15 min)

List 2-3 variables on your highest-traffic page. Ask: Could these work better together?

This Week (2 hours)

Estimate whether eligible traffic supports the sample required for a 2×2 MVT.

Your First MVT Sprint

Plan a 2×2 test with two variables and two options each. Specify the interaction you want to estimate and the sample needed to detect it.

Key takeaways

• A factorial test can estimate main effects and interactions when its design and sample support them.
• Count combinations separately from calculating statistical power.
• Start with a design you can implement and analyze reliably.
• Account for multiple comparisons and uncertainty when selecting a combination.

Next Steps

Expand your testing toolkit:

1. Calculate sample size with our Sample Size Calculator
2. Run basic tests with A/B Test Calculator
3. Track conversions with Conversion Rate Calculator

Sources