Read the result in whole numbers, so two services never disagree about who won.
An experiment platform is asked the same three questions every time. What is each arm converting at, is one better than the other, and can we stop yet.
All three are easier and safer in integers. Conversion goes in basis points, hundredths of a percent, so 4.12% is 412 and a whole number is the unit everyone agrees on. Comparisons use cross-multiplication instead of division, so nothing rounds before the comparison happens. And before any of that, the split itself gets checked, because a result computed on a broken assignment is not a small error, it is a meaningless one.
The word "peek" carries the last piece. Stopping the moment a lead appears is the single most common way an honest experiment produces a false answer.
Auto play is off because your system asks for reduced motion. Run a day still works.
Three small functions carry almost every experiment readout. None of them needs a float.
def rate_bps(conversions, visitors):
"""Conversion in basis points, floored. 412 means 4.12%."""
return (10000 * conversions) // visitors
def variant_wins(c_a, v_a, c_b, v_b):
"""Is arm A converting better than arm B? No division, so no rounding."""
return c_a * v_b > c_b * v_a
def srm_ok(counts, expected_pct, total):
"""Chi-square on the split itself. Above 3.841 is a 1-in-20 coincidence."""
chi = 0
for observed, pct in zip(counts, expected_pct):
expected = total * pct / 100
chi += (observed - expected) ** 2 / expected
return chi <= 3.841 # 1 degree of freedom, p = 0.05
Compute the rates in basis points for display, and compare with cross-multiplication for decisions. Doing both with the same floored number is what makes a dashboard and a decision agree.
| Operation | Time | What it costs you |
|---|---|---|
| Rate in basis points | O(1) | Up to 0.01 percentage point of precision, discarded by the floor |
| Compare two arms | O(1) | Nothing, cross-multiplication is exact |
| Sample-ratio check | O(arms) | Nothing, and skipping it costs the whole experiment |
| Sequential rule over d days | O(d) | One prefix sum per arm, so one pass |
The trade is deliberate. Integers give up a hundredth of a percentage point and get back exact reproducibility: the same inputs give the same answer on every machine, in every language, forever.
c_a * v_b > c_b * v_a knows which is bigger. Floor for the dashboard, cross-multiply for the decision.-1 // 2 and int(-1/2) disagree. Decide whether you want floor or truncation, write it down, and test the negative case, because it only shows up when the variant loses.Control took 5,003 visitors and 206 conversions. Variant took 4,998 and 224.
Control is 10000 * 206 // 5003, which is 411 basis points. Variant is 10000 * 224 // 4998, which is 448. The uplift is 37 basis points, and both numbers are exact integers that any service can recompute.
Now check that they are really different rather than just differently rounded: 224 * 5003 = 1,120,672 against 206 * 4998 = 1,029,588. The variant is genuinely ahead, and no division happened. Finally, the split: 5,003 against 4,998 out of 10,001 expected 50/50 gives a chi-square of about 0.0025, nowhere near 3.841, so the assignment is healthy and the result is worth reading.
An experiment intended a 50/50 split. After a week, control has 48,900 visitors and variant has 51,100. The variant shows a 60 basis point lift. What do you do?