Fix the evidence rule for a flaky test diagnosis
Your task
One test fails about 1 in 20 CI runs. An AI agent called it a timing flake and suggested a 200 ms sleep before the assert. Before your team accepts a diagnosis like that, you rerun the test many times alone and many times after the rest of the suite, and a rule names the cause the counts support.
An assistant wrote classify_flake_evidence(isolated_runs, isolated_failures, after_suite_runs, after_suite_failures). It is in the editor. It has a bug. Find it and fix it.
Input and output:
isolated_runs,isolated_failures: how many times the test ran on its own, and how many of those runs failed.after_suite_runs,after_suite_failures: how many times it ran after the rest of the suite, and how many of those runs failed.- Return one verdict string.
Rules the tests check. Return the first verdict that applies:
"invalid-evidence"when any count is negative, or either side has more failures than runs."insufficient-evidence"when either side has fewer than 100 runs."always-fails"when both sides failed on every run."not-reproduced"when neither side failed at all."order-dependent"when the isolated runs never failed."order-amplified"when the after-suite failure rate is more than twice the isolated failure rate. Compare exactly, with no rounding:after_suite_failures * isolated_runs > 2 * isolated_failures * after_suite_runs."inherent-race"otherwise.
Examples
classify_flake_evidence(isolated_runs: int, isolated_failures: int, after_suite_runs: int, after_suite_failures: int) -> strExample 1
Input
classify_flake_evidence(200, 0, 200, 11)Output
"order-dependent"Example 2
Input
classify_flake_evidence(200, 10, 200, 12)Output
"inherent-race"