AI-assisted engineering/Debug the agent trace

A version the report agent promoted failed its canary

Debug the agent traceMedium30 min

A report agent compares each week's new version with the current one on the main eval and promotes the winner to a canary. This week it promoted v42, and the canary rolled it back.

Your task

Name the span where the agent first went wrong, then fix decide() so the replay passes.

  • root_cause_span() returns the FIRST span whose plan contradicted what the agent already knew
  • decide(base, cand) returns "promote", "keep", "rerun", or "escalate"; base is the current version's eval summary and cand the new version's, each with the counts tasks, graded and passed
  • a summary with a missing or negative count, more passes than graded tasks, or more graded tasks than tasks escalates, and so do two runs over different numbers of tasks
  • a task that was not graded has no result: a run with more than 10 percent of its tasks ungraded escalates, and any other ungraded task earns a rerun
  • when both runs graded every task, promote if the new version passed more tasks, and keep the current one otherwise
  • leave call_tool() and check() alone; the eval store is not the bug

The replay runs your decide() across pairs of summaries at submit, and your named span is checked against the key.