Canary verdict from error and escalation rates

Agent buildEasy15 min

Your task

A support agent's new version is in canary: a slice of tickets goes to it while the rest stay on the current version, the baseline. Every hour the rollout controller compares the two and decides what happens next.

Write canary_verdict(baseline, canary, min_tasks). baseline and canary are dicts {"tasks": int, "errors": int, "escalations": int}: tickets handled, tickets that ended in an error, and tickets the agent handed to a human. The error rate is errors / tasks and the escalation rate is escalations / tasks.

Apply these rules in order and return the first one that matches:

  1. "wait" if the canary has handled fewer than min_tasks tickets: too little data to judge.
  2. "rollback" if the canary's error rate is more than 1 percentage point above the baseline's.
  3. "hold" if the canary's escalation rate is more than 5 percentage points above the baseline's: keep the traffic split where it is and page the owner.
  4. "promote" otherwise.

"More than" is strict: exactly 1 point (or exactly 5) above the baseline is not over the line. A canary doing better than the baseline is never penalized. min_tasks is at least 1, the baseline has at least one task, and every count is a non-negative integer with errors and escalations never above tasks.

The counts are integers, so compare the rates exactly rather than as floats. With ct, ce the canary's tasks and errors and bt, be the baseline's, the canary is more than 1 point worse when 100 * (ce * bt - be * ct) > ct * bt.

Examples, all with baseline {"tasks": 1000, "errors": 20, "escalations": 50} and min_tasks 200:

  • canary {"tasks": 300, "errors": 9, "escalations": 15} returns "promote": 3% errors is exactly 1 point above 2%, which is not over.
  • canary {"tasks": 300, "errors": 10, "escalations": 15} returns "rollback": 3.33% is more than 1 point above.
  • canary {"tasks": 150, "errors": 0, "escalations": 0} returns "wait".

Examples

canary_verdict(baseline: dict, canary: dict, min_tasks: int) -> str
Example 1
Inputcanary_verdict({"tasks": 1000, "errors": 20, "escalations": 50}, {"tasks": 300, "errors": 9, "escalations": 15}, 200)
Output"promote"
Example 2
Inputcanary_verdict({"tasks": 1000, "errors": 20, "escalations": 50}, {"tasks": 300, "errors": 10, "escalations": 15}, 200)
Output"rollback"
Example 3
Inputcanary_verdict({"tasks": 1000, "errors": 20, "escalations": 50}, {"tasks": 150, "errors": 0, "escalations": 0}, 200)
Output"wait"