Tell a real benchmark regression from noise

Agent buildEasy10 minGoogle and Nvidia style

Your task

A benchmark regressed somewhere in the last 200 commits, and benchmark numbers are noisy. Before a bisecting agent names a commit, and the person behind it, it has to tell a real slowdown from noise.

Write classify_bench_regression(baseline_runs, candidate_runs, noise_band_pct): the rule your agent follows to compare two commits.

  • baseline_runs, candidate_runs (list of float): repeated timings for the known-good commit and for the commit under test. Higher is slower.
  • noise_band_pct (float): the declared noise, in percent.
  • Return "regression", "clean", "inconclusive" or "insufficient data".

Rules, in order:

  1. Fewer than 3 runs on either side: "insufficient data".
  2. Take the median of each side. With an even count, the median is the average of the two middle values. If the baseline median is 0 or less: "insufficient data".
  3. The band is the larger of noise_band_pct and the baseline's own spread, (max - min) / baseline median * 100.
  4. The effect is (candidate median - baseline median) / baseline median * 100.
  5. An effect at or below the band, a speedup included: "clean".
  6. An effect of at least twice the band: "regression".
  7. Otherwise: "inconclusive".

Examples

classify_bench_regression(baseline_runs: list[float], candidate_runs: list[float], noise_band_pct: float) -> str
Example 1
Inputclassify_bench_regression([100, 101, 99, 100.5], [118, 119, 117.5], 3)
Output"regression"
Example 2
Inputclassify_bench_regression([200, 210, 195], [205, 208, 199], 2)
Output"clean"
Example 3
Inputclassify_bench_regression([98, 100, 102], [107, 107, 106], 5)
Output"inconclusive"