Fix the evidence rule for a flaky test diagnosis

Review the AI's codeEasy12 min

Your task

One test fails about 1 in 20 CI runs. An AI agent called it a timing flake and suggested a 200 ms sleep before the assert. Before your team accepts a diagnosis like that, you rerun the test many times alone and many times after the rest of the suite, and a rule names the cause the counts support.

An assistant wrote classify_flake_evidence(isolated_runs, isolated_failures, after_suite_runs, after_suite_failures). It is in the editor. It has a bug. Find it and fix it.

Input and output:

  • isolated_runs, isolated_failures: how many times the test ran on its own, and how many of those runs failed.
  • after_suite_runs, after_suite_failures: how many times it ran after the rest of the suite, and how many of those runs failed.
  • Return one verdict string.

Rules the tests check. Return the first verdict that applies:

  1. "invalid-evidence" when any count is negative, or either side has more failures than runs.
  2. "insufficient-evidence" when either side has fewer than 100 runs.
  3. "always-fails" when both sides failed on every run.
  4. "not-reproduced" when neither side failed at all.
  5. "order-dependent" when the isolated runs never failed.
  6. "order-amplified" when the after-suite failure rate is more than twice the isolated failure rate. Compare exactly, with no rounding: after_suite_failures * isolated_runs > 2 * isolated_failures * after_suite_runs.
  7. "inherent-race" otherwise.

Examples

classify_flake_evidence(isolated_runs: int, isolated_failures: int, after_suite_runs: int, after_suite_failures: int) -> str
Example 1
Inputclassify_flake_evidence(200, 0, 200, 11)
Output"order-dependent"
Example 2
Inputclassify_flake_evidence(200, 10, 200, 12)
Output"inherent-race"