Experiment pitfalls: peeking, novelty, multiple comparisons

A clean test can still lie to you — three ways — and each one has a boring, reliable antidote.

The idea

You can randomize perfectly and still reach a false conclusion, because three habits quietly manufacture “wins” out of pure noise. Peeking — stopping the moment it turns significant — inflates false positives several times over. Novelty — a week-one bump that fades — fools you if you decide early. Multiple comparisons — checking twenty metrics — all but guarantees a lucky green somewhere.

The antidotes are dull on purpose: pre-register the sample size, the primary metric, and the decision before you look; run an A/A test to feel how much your own pipeline lies. Play with each lab below and watch the lie form.

Three mini-labs

How it works

Peeking. Each test at the 5% level has a 5% false-positive rate at one fixed look. But the test statistic wanders as data arrives, so the more often you look, the more chances it has to stray past the line at least once:

looks (peeks)      P(a false "win" at some point)
   1 (correct)          ~5%
   5                    ~14%
  20                    ~25%
  40  (daily, 6 wks)    ~30%      ← this lab

fix: fixed sample size, or a sequential design (alpha-spending / group-sequential)
     that budgets the 5% across all the looks you intend to take.

Novelty. The measured lift is durable-effect plus a fading novelty bump. Decide during the fade and you bank the bump:

lift(day) = durable + novelty × e^(-(day-1)/6)     durable = 0,  novelty = 8 pp
  day  1   ->  +8.0 pp   "ship it!"      (almost all novelty)
  day  9   ->  +2.1 pp   "still a win"
  day 21   ->  +0.3 pp   "no real effect" (the truth)

fix: run past the decay (2-3+ weeks), judge the plateau, not the spike.

Multiple comparisons. Under the null, p-values are uniform, so each metric independently trips 5% of the time:

P(at least one false win) = 1 - (1 - 0.05)^20 = 1 - 0.95^20 = 64%
expected false winners     = 20 × 0.05 = 1.0 per run

fix (Bonferroni): test each at 0.05 / 20 = 0.0025
     ->  1 - (1 - 0.0025)^20 = ~4.9%   (family-wise rate back to ~5%)

When to use it

Watch out for

• “It hit significance, ship it.” If you didn’t pre-commit the sample size, that p < 0.05 is likely a peeking artifact. Significance without a stopping plan is not a result.
• Reading a launch in week one. The biggest, most exciting lift is exactly when novelty is strongest. Early enthusiasm is the tell, not the signal.
• The metric dredge. “The primary was flat, but engagement-per-session in the returning-mobile cohort is up!” With enough slices, something is always up. Decide the metric first.
• Switching the primary metric after the fact. Picking whichever metric happened to win is peeking in a different costume — the false-positive rate balloons the same way.
• No A/A baseline. If you never ran the test against itself, you don’t know how noisy your own pipeline is — and you’ll read that noise as effects.

Worked example

An interviewer says: “We shipped a redesign. Day 3, conversion is up 6% and significant — do we roll out?” The disciplined answer names all three traps at once. Was day 3 the pre-planned readout, or a peek? If you’ve been checking daily, that significance is inflated — hold to the pre-set sample size. Is 6% durable or novelty? A redesign is precisely the change that spikes then fades; wait two to three weeks and read the plateau. And is conversion the one primary metric you committed to, or the best of many you scanned? If it’s a lucky secondary, correct for the family or don’t count it. A strong candidate closes with the cheap insurance: pre-register next time, and keep an A/A test running so you always know your own noise floor.