Tell a real benchmark regression from noise
Your task
A benchmark regressed somewhere in the last 200 commits, and benchmark numbers are noisy. Before a bisecting agent names a commit, and the person behind it, it has to tell a real slowdown from noise.
Write classify_bench_regression(baseline_runs, candidate_runs, noise_band_pct): the rule your agent follows to compare two commits.
baseline_runs,candidate_runs(list of float): repeated timings for the known-good commit and for the commit under test. Higher is slower.noise_band_pct(float): the declared noise, in percent.- Return
"regression","clean","inconclusive"or"insufficient data".
Rules, in order:
- Fewer than 3 runs on either side:
"insufficient data". - Take the median of each side. With an even count, the median is the average of the two middle values. If the baseline median is 0 or less:
"insufficient data". - The band is the larger of
noise_band_pctand the baseline's own spread,(max - min) / baseline median * 100. - The effect is
(candidate median - baseline median) / baseline median * 100. - An effect at or below the band, a speedup included:
"clean". - An effect of at least twice the band:
"regression". - Otherwise:
"inconclusive".
Examples
classify_bench_regression(baseline_runs: list[float], candidate_runs: list[float], noise_band_pct: float) -> strExample 1
Input
classify_bench_regression([100, 101, 99, 100.5], [118, 119, 117.5], 3)Output
"regression"Example 2
Input
classify_bench_regression([200, 210, 195], [205, 208, 199], 2)Output
"clean"Example 3
Input
classify_bench_regression([98, 100, 102], [107, 107, 106], 5)Output
"inconclusive"