Fix the revert check for an agent's regression tests

Review the AI's codeEasy10 minAnthropic and OpenAI style

Your task

You found a bug and asked an AI agent to fix it and write a regression test for it. It returned a one-line fix and a passing test. A test written by the same agent can pass whether or not the fix is right. So you revert the fix, run each test on the buggy code and then on the fixed code, and score each pair.

An assistant wrote classify_regression_tests(result_lines). It is in the editor. It has a bug. Find it and fix it.

Input and output:

  • result_lines: one string per test, in the form "name|on_buggy|on_fixed", like "test_rounding|fail|pass". on_buggy is the test's result with the fix reverted, and on_fixed its result with the fix in place. A result is normally pass or fail, but a runner can write other words. The list can be empty.
  • Return one "<name>:<verdict>" string per scored line.

Rules the tests check:

  • Split each line on |. Skip a line with fewer than 3 parts. Parts after the third are ignored.
  • Trim whitespace at both ends of all three parts. Skip a line whose name is empty after trimming.
  • Results compare without case, so PASS counts as pass and Fail as fail.
  • The verdict is unreadable when either result is anything other than pass or fail.
  • Otherwise the verdict comes from the pair:
    • guards: fails on the buggy code, passes on the fixed code.
    • vacuous: passes on both.
    • broken: fails on both.
    • inverted: passes on the buggy code, fails on the fixed code.
  • Return the verdicts in input order. Keep duplicates: a name listed twice is scored twice.

Examples

classify_regression_tests(result_lines: list[str]) -> list[str]
Example 1
Inputclassify_regression_tests(["test_zero_qty_discount|fail|pass"])
Output["test_zero_qty_discount:guards"]
Example 2
Inputclassify_regression_tests(["test_zero_qty_discount|pass|pass", "test_rounding|fail|pass"])
Output["test_zero_qty_discount:vacuous", "test_rounding:guards"]
Example 3
Inputclassify_regression_tests(["test_a|fail|fail", "test_b|pass|fail"])
Output["test_a:broken", "test_b:inverted"]