Next

Decide what a CI triage agent does next

Agent buildEasy10 minMicrosoft style

Your task

Your team's CI fails about 40 times a day, and engineers lose hours working out whether a red build is their fault, the platform's, or a flaky test. An agent reads each failed build and picks one action.

Write triage_ci_failure(flaky_fingerprint, infra_degraded, touches_changed_files, confidence, reruns_used): the rule your agent follows for each failed build.

  • flaky_fingerprint (bool): the failure matches a known flaky-test fingerprint, a log signature that has passed on rerun before.
  • infra_degraded (bool): the CI platform reports an outage or slowdown right now.
  • touches_changed_files (bool): the failing step covers files this change edited.
  • confidence (float, 0 to 1): how sure the agent's classifier is about its read of the failure.
  • reruns_used (int): how many times this job has already been rerun.

It returns "rerun", "page_oncall", "notify_author" or "human_queue". Apply these rules in order. The first one that matches decides:

  1. confidence below 0.70: "human_queue". Low confidence outranks every other signal.
  2. infra_degraded: "page_oncall".
  3. flaky_fingerprint and confidence of at least 0.95: "rerun" if reruns_used is below 2, otherwise "human_queue", even if the change touched the files.
  4. touches_changed_files: "notify_author".
  5. Otherwise: "human_queue".

A flaky match below 0.95 skips rule 3 and is treated as a real failure.

Then, in a sentence or two: why does a rerun need a higher bar than every other action?

Examples

triage_ci_failure(flaky_fingerprint: bool, infra_degraded: bool, touches_changed_files: bool, confidence: float, reruns_used: int) -> str
Example 1
Inputtriage_ci_failure(True, False, False, 0.97, 0)
Output"rerun"
Example 2
Inputtriage_ci_failure(False, True, True, 0.9, 0)
Output"page_oncall"
Example 3
Inputtriage_ci_failure(False, False, True, 0.85, 0)
Output"notify_author"