Fix the revert check for an agent's regression tests
Your task
You found a bug and asked an AI agent to fix it and write a regression test for it. It returned a one-line fix and a passing test. A test written by the same agent can pass whether or not the fix is right. So you revert the fix, run each test on the buggy code and then on the fixed code, and score each pair.
An assistant wrote classify_regression_tests(result_lines). It is in the editor. It has a bug. Find it and fix it.
Input and output:
result_lines: one string per test, in the form"name|on_buggy|on_fixed", like"test_rounding|fail|pass".on_buggyis the test's result with the fix reverted, andon_fixedits result with the fix in place. A result is normallypassorfail, but a runner can write other words. The list can be empty.- Return one
"<name>:<verdict>"string per scored line.
Rules the tests check:
- Split each line on
|. Skip a line with fewer than 3 parts. Parts after the third are ignored. - Trim whitespace at both ends of all three parts. Skip a line whose name is empty after trimming.
- Results compare without case, so
PASScounts aspassandFailasfail. - The verdict is
unreadablewhen either result is anything other thanpassorfail. - Otherwise the verdict comes from the pair:
guards: fails on the buggy code, passes on the fixed code.vacuous: passes on both.broken: fails on both.inverted: passes on the buggy code, fails on the fixed code.
- Return the verdicts in input order. Keep duplicates: a name listed twice is scored twice.
Examples
classify_regression_tests(result_lines: list[str]) -> list[str]Example 1
Input
classify_regression_tests(["test_zero_qty_discount|fail|pass"])Output
["test_zero_qty_discount:guards"]Example 2
Input
classify_regression_tests(["test_zero_qty_discount|pass|pass", "test_rounding|fail|pass"])Output
["test_zero_qty_discount:vacuous", "test_rounding:guards"]Example 3
Input
classify_regression_tests(["test_a|fail|fail", "test_b|pass|fail"])Output
["test_a:broken", "test_b:inverted"]