AI-assisted engineering/Agent evaluation

Fix the acceptance gate for a performance refactor

Review the AI's codeEasy10 min

Your task

An agent rewrote a Python data-processing function with vectorized pandas and claimed it was 10x faster. On your spot check, the outputs looked the same. Before your team accepts a performance refactor, it runs a gate: a differential test against the old function's outputs, then a benchmark you reproduce yourself at several input sizes.

An assistant wrote verify_perf_refactor(baseline_out, candidate_out, baseline_ms, candidate_ms). It is in the editor. It has a bug. Find it and fix it.

Input and output:

baseline_out, candidate_out: the old and the new function's outputs on the same inputs, each written as a canonical string.
baseline_ms, candidate_ms: measured times in milliseconds for the old and the new version, one entry per input size, smallest size first. Index i is the same size in both lists.
Return "accept", "reject_output_mismatch", "reject_no_speedup" or "reject_no_benchmark".

Rules the tests check. Return the first that applies:

"reject_output_mismatch": the two output lists differ in length, or differ at any position. Compare the strings exactly, so "7" and "7.0" differ.
"reject_no_benchmark": baseline_ms is empty, the two timing lists differ in length, or any time in either list is 0 or less.
Compute the speedup at the largest size only, the last entry: the last baseline time divided by the last candidate time. At least 2: "accept". Otherwise "reject_no_speedup".

Examples

verify_perf_refactor(baseline_out: list[str], candidate_out: list[str], baseline_ms: list[float], candidate_ms: list[float]) -> str

Example 1

Inputverify_perf_refactor(["1", "2", "3"], ["1", "2", "3"], [40, 400], [10, 50])

Output"accept"

Example 2

Inputverify_perf_refactor(["1.0", "nan", "3.0"], ["1.0", "0.0", "3.0"], [40, 400], [4, 20])

Output"reject_output_mismatch"

Example 3

Inputverify_perf_refactor(["a", "b"], ["a", "b"], [100, 50], [10, 50])

Output"reject_no_speedup"