PreviousNext
AI-assisted engineering/Reviewing AI code

Fix the review rule for a flaky test fix

Review the AI's codeEasy10 min

Your task

A test failed about 1 in 20 CI runs. An AI agent added sleep(500) before the assertion, the test passed 50 times in a row locally, and the team wants to merge. Your team reviews fixes for flaky tests with a rule that reads the patch and the evidence.

An assistant wrote review_flaky_fix(patch_lines, green_run_count, baseline_one_failure_in_runs). It is in the editor. It has a bug. Find it and fix it.

Input and output:

  • patch_lines: the lines of the unified diff (the usual patch format, with + for added lines and - for removed ones).
  • green_run_count: how many runs in a row passed with the fix.
  • baseline_one_failure_in_runs: the failure rate before the fix, as "1 failure in N runs". For example 20.
  • Return the list of findings, or ["approve"] when there are none.

Rules the tests check:

  • Only added lines count: lines that start with + but not with +++. Drop the leading + and lowercase the rest.
  • An added line contains a sleep when it contains any of sleep(, settimeout( or delay( anywhere, as plain text.
  • An added line contains a condition wait when it contains any of wait_until, waitfor, await (with the space), poll_until or barrier anywhere, as plain text.
  • Findings, in this order:
    1. "wall_clock_sleep" when any added line contains a sleep.
    2. "no_condition_wait" when no added line contains a condition wait.
    3. "insufficient_runs" when green_run_count is less than 30 times baseline_one_failure_in_runs.

Examples

review_flaky_fix(patch_lines: list[str], green_run_count: int, baseline_one_failure_in_runs: int) -> list[str]
Example 1
Inputreview_flaky_fix(["--- a/tests/test_checkout.py", "+++ b/tests/test_checkout.py", "- assert order.status == 'paid'", "+ sleep(500)", "+ assert order.status == 'paid'"], 50, 20)
Output["wall_clock_sleep", "no_condition_wait", "insufficient_runs"]
Example 2
Inputreview_flaky_fix(["+ wait_until(order_is_paid, timeout_ms=5000)", "+ assert order.status == 'paid'"], 600, 20)
Output["approve"]
Example 3
Inputreview_flaky_fix(["+ poll_until(order_is_paid)", "+ assert order.status == 'paid'"], 50, 20)
Output["insufficient_runs"]