Benchmark regression bisect
A benchmark regressed sometime in the last two hundred commits, and benchmark numbers are noisy. Design an agent that finds the culprit: how it searches, how it separates real regressions from noise, and what it does once it believes it has found the commit — remembering there's a person attached to every commit it blames. How would you validate the agent on regressions you've already root-caused, and what's the pitch for building it?
classify_bench_regression(baseline_runs: list[float], candidate_runs: list[float], noise_band_pct: float) → str[[100,101,99,100.5],[118,119,117.5],3]out"regression"[[200,210,195],[205,208,199],2]out"clean"[[98,100,102],[107,107,106],5]out"inconclusive"Turn the fuzzy goal into a bounded agent task. Say what the agent reads, what it may do on its own versus draft for a human, and what it must never touch. Then earn the trust: how you’d verify it before rollout, and the pitch (the problem, what you built, why it matters).
Agent build: design the agent in plain language (or narrate it). The coach grades the decomposition, the guardrails, the verification plan, and the pitch.