Fix the diff auditor for agent test fixes

Review the AI's codeEasy10 minAnthropic and OpenAI style

Your task

An AI agent was asked to make a failing test, test_refund_caps_at_original_charge, pass. It reported success and CI is green. The risk is that it changed the test to accept the code instead of fixing the code. Your team runs every agent diff through an auditor that flags the usual ways of gaming a test.

An assistant wrote audit_test_fix(diff_lines), the auditor. It is in the editor. It has a bug. Find it and fix it.

Input and output:

  • diff_lines: the changed lines of the diff, each written path|sign|content, where sign is + for an added line and - for a removed one. The content may itself contain |.
  • Return the findings in the order listed below, each at most once.

Rules the tests check:

  • The path is the text before the first |, the sign is the text between the first and second |, and the content is everything after the second. Skip a line with fewer than two |, or whose sign is not + or -.
  • A path is a test file when it contains test anywhere, case included. Every other path is production code.
  • Findings, in this order:
    1. assertion_removed: a removed line in a test file contains assert.
    2. skip_added: an added line in a test file contains skip or xfail.
    3. mocking_added: an added line in a test file contains mock( or patch(.
    4. broad_exception_added: an added line in any file contains except Exception.
    5. no_production_change: none of the lines that were not skipped is in production code. An empty diff counts.
  • "Contains" means the text appears anywhere, case included.

Examples

audit_test_fix(diff_lines: list[str]) -> list[str]
Example 1
Inputaudit_test_fix(["tests/test_refund.py|-| assert refund <= charge", "tests/test_refund.py|+| assert True"])
Output["assertion_removed", "no_production_change"]
Example 2
Inputaudit_test_fix(["billing/refund.py|-| return amount", "billing/refund.py|+| return min(amount, charge)", "tests/test_refund.py|+| assert cap(150, 100) == 100"])
Output[]