AI-assisted engineering/Debug the agent trace

The release gate blocked a version over one task

Debug the agent traceMedium30 min

A release gate agent compares each candidate version with the one in production on the nightly eval. Last night it held v18 back over a single task, paged the release owner, and kept the block after a rerun.

Your task

Name the span where the gate first went wrong, then fix gate() so the replay passes.

  • root_cause_span() returns the FIRST span whose plan contradicted what the agent already knew
  • gate(old, new) returns "ship", "rerun", "block", or "escalate"; old and new are one task's runs, oldest first, "P" for a pass and "F" for a failure
  • no old runs, no new runs, or any letter other than P and F escalates before anything else; otherwise a new version that passes every run ships
  • a task that passed every old run is stable, and any failure on the new version blocks it
  • a task that failed at least once in its old runs is flaky: with fewer than 3 new runs, a failure earns a rerun; with 3 or more, it blocks when every new run failed and ships otherwise
  • leave call_tool() and check() alone; the eval store is not the bug

The replay runs your gate() across task histories at submit, and your named span is checked against the key.