A/B test design: hypothesis, randomization, primary metric

Turn “make the homepage better” into a fair coin toss that can actually tell you what worked.

The idea

An A/B test isn’t “ship it and watch the numbers.” It’s a small experiment with three parts written down before you look at data: a falsifiable hypothesis (change X moves metric Y by Z for population P), random assignment so the two groups differ only by the change, and one primary metric plus a couple of guardrails you commit to in advance.

Randomization is the quiet hero. It’s the only thing that lets you say the change caused the difference, instead of “the two groups were just different people.”

How it works

The whole game is making the two groups exchangeable — identical on average in every way except the change. Then any difference in the primary metric is caused by the change. Here is the arithmetic the simulator runs:

# 60 visitors. Early sign-ups skew high-intent (they'd convert anyway).
# base convert: high-intent 40%, low-intent 10%.  True effect: +8 pp.

Split by signup date  (a confounder rides along)
  control  = first 30 (mostly high-intent) -> base 34%
  treatment= last  30 (mostly low-intent)  -> base 16%, +8 = 24%
  measured lift = 24% - 34% = -10 pp   # sign flipped! looks harmful.

Random coin flip  (groups balance)
  control   base ~= 25%
  treatment base ~= 25%, +8 = 33%
  measured lift ~= 33% - 25% = +8 pp   # matches the truth.

Same visitors, same real effect. The only thing that changed is how you split them — and that decided whether your number was true or a lie.

When to use it

Reach for an A/B test when…	The trade-off
You can randomly assign users and wait for enough of them.	Needs traffic and patience — small samples give noisy, unstable lifts.
The metric responds within a testable window (a click, a signup).	Slow outcomes (annual churn) are hard; use proxies carefully.
The change is isolable to one variable you can toggle.	Bundled changes tell you something moved, not what.

Watch out for

• Peeking and stopping early. Checking daily and stopping the moment it’s “significant” inflates false positives. Fix the sample size (or use a sequential method built for peeking) up front.
• Switching your primary metric after the fact. If you pre-committed to signup rate, don’t ship because “time-on-page” happened to look nice. That’s how noise becomes a “win.”
• Broken randomization. Assigning by signup date, geography, or “whoever we could migrate first” smuggles a confounder in. Randomize at the user level.
• Ignoring guardrails. A signup lift that quietly tanks revenue or load time isn’t a win. Name 2–3 guardrail metrics before you start.
• Too many variants. Testing ten things at once splits your traffic thin and multiplies false alarms. Keep it focused.

Worked example

An interviewer asks: “The team wants to make the pricing page ‘better.’ Design the test.” A strong answer names all three parts. Hypothesis: adding annual-plan social proof will lift the paid-conversion rate by ~3 pp for logged-out visitors. Primary metric: paid-conversion rate; guardrails: refund rate and page load. Assignment: randomize each visitor 50/50, bucketed by a hashed user id so the split is stable and independent of when or where they arrived. Then estimate the sample size needed to detect a 3 pp lift, run to that size, and read the primary metric once. If someone suggests “just show it to the East region and compare” — that’s the signup-date trap in disguise, and you’d flag the confounder.