Fix the diff auditor for agent test fixes
Your task
An AI agent was asked to make a failing test, test_refund_caps_at_original_charge, pass. It reported success and CI is green. The risk is that it changed the test to accept the code instead of fixing the code. Your team runs every agent diff through an auditor that flags the usual ways of gaming a test.
An assistant wrote audit_test_fix(diff_lines), the auditor. It is in the editor. It has a bug. Find it and fix it.
Input and output:
diff_lines: the changed lines of the diff, each writtenpath|sign|content, where sign is+for an added line and-for a removed one. The content may itself contain|.- Return the findings in the order listed below, each at most once.
Rules the tests check:
- The path is the text before the first
|, the sign is the text between the first and second|, and the content is everything after the second. Skip a line with fewer than two|, or whose sign is not+or-. - A path is a test file when it contains
testanywhere, case included. Every other path is production code. - Findings, in this order:
assertion_removed: a removed line in a test file containsassert.skip_added: an added line in a test file containsskiporxfail.mocking_added: an added line in a test file containsmock(orpatch(.broad_exception_added: an added line in any file containsexcept Exception.no_production_change: none of the lines that were not skipped is in production code. An empty diff counts.
- "Contains" means the text appears anywhere, case included.
Examples
audit_test_fix(diff_lines: list[str]) -> list[str]Example 1
Input
audit_test_fix(["tests/test_refund.py|-| assert refund <= charge", "tests/test_refund.py|+| assert True"])Output
["assertion_removed", "no_production_change"]Example 2
Input
audit_test_fix(["billing/refund.py|-| return amount", "billing/refund.py|+| return min(amount, charge)", "tests/test_refund.py|+| assert cap(150, 100) == 100"])Output
[]