A/B Test & Experiment Designer
Design rigorous experiments with hypothesis, sample size, and analysis plan
- • Hypothesis written If/Then/Because with the behavior theory you're testing.
- • One decision metric, stated MDE, and power-based sample size and duration.
- • Randomization plan, SRM checks, and guardrails to avoid breaking things.
- • Ramp, stop, and rollback criteria agreed before launch, with a tested rollback process.
- • Pre-agreed analysis: peeking rules, segments, and how you decide ship/hold.
- • Tests with too little data to reliably detect the planned effect.
- • Peeking daily and pivoting mid‑run; novelty and regression bite later.
- • Mixed variants (too many changes) so you can't tell what worked.
- • No SRM or bot filters—your split is 55/45 and nobody notices.
- • Measuring the wrong thing: clicks up, revenue down. Guardrails would help.
How do I write a hypothesis that isn't fluff?
If [we remove one step from onboarding], then [activation rate increases by 12%] because [drop‑offs cluster at the second form and users say it feels like paperwork]. If/Then/Because forces you to show your reasoning, not just hope.
What's a reasonable minimum detectable effect (MDE)?
Tie MDE to business impact and available traffic. If a 1% lift matters but traffic only supports detecting 6%, consider a longer test, another surface, or a different research method.
Can I peek at results?
Monitor health during the test. For a fixed-sample analysis, follow the planned sample and duration; a scheduled interim check still needs a method that permits early decisions.
How do I avoid SRM and other traps?
Automate SRM checks, filter bots, and hash randomize on a stable key (user, not session). Also: pause overlapping experiments on the same surface unless you have an interaction design.
My exec wants to ramp early because it "looks good". Help?
Pre‑commit to kill and ship criteria. Put the table in the doc before launch. When pressure hits, point to the rules we all signed. It saves relationships—and your data.
What if results are flat?
Check exposure and measurement, then report the effect estimate and uncertainty. Treat unplanned segment findings as exploratory and verify them in follow-up work.
Which metrics should be guardrails vs. secondary?
Guardrails track unacceptable harm, such as errors or refunds. Secondary metrics help explain the result. Define the response to guardrail changes before launch.
How long should we run an experiment?
At least one full business cycle (weekly patterns matter) and until your powered sample size is reached. Stop‑early rules must be defined a priori; “feels done” is not a rule.
Do I need an A/A test?
An A/A test can help check assignment and measurement after changes. A significant result can occur by chance, so investigate rather than treating it as proof of a bug.
How do I make the decision obvious at the end?
Document criteria for shipping, rejecting, or running a follow-up. An inconclusive result does not justify extending a fixed-sample test until it becomes significant.
When to use it
Feature validation and optimization
Before you use the output
- •Fill in the variables with the facts and constraints you have.
- •Check the output against your source material and revise any mistakes.
- •Enable web search where available and check the dates and sources it returns.
Expected output
Comprehensive experiment design document