Staged rollout with guardrail rollback

Agent buildMedium30 min

Your task

A new ticket-triage agent is rolled out in stages: it takes stages[0] percent of tickets, then stages[1] percent, and so on. Once a day, the rollout controller receives a report comparing the agent's tickets with a control group handled the old way, and moves the rollout up, down or not at all.

Write staged_rollout(stages, soak, min_tasks, margin, days):

  • stages: strictly increasing percentages, each from 1 to 100.
  • soak: how many healthy days a stage needs before the rollout moves past it (at least 1).
  • min_tasks: the smallest sample worth judging (at least 1).
  • margin: the guardrail, in tenths of a percent (per mille).
  • days: one report per day, [agent_tasks, agent_bad, control_tasks, control_bad], where "bad" counts tickets that were misrouted. Every count is a non-negative integer of at most 1,000,000, and bad never exceeds tasks.

Classify each day

  • Thin: agent_tasks < min_tasks or control_tasks < min_tasks. A thin day is skipped: it neither counts toward a soak nor breaches.
  • Breach: the agent's bad rate is more than margin per mille above the control's. With integers: 1000 * (agent_bad * control_tasks - control_bad * agent_tasks) > margin * agent_tasks * control_tasks. Exactly margin above is not a breach.
  • Healthy: any other day.

Move the rollout

The rollout starts at stage 0 with no healthy days counted.

  • Healthy day: count it toward the current stage. Once the stage has the healthy days it needs, move up one stage and start counting from zero. A stage needs soak healthy days, or 2 * soak if that stage has breached before. At the last stage, healthy days change nothing.
  • Breach: add one to the current stage's breach count. If that makes two breaches for this stage, or the rollout is at stage 0, halt. Otherwise move down one stage and start counting from zero.
  • Halted: the agent is at 0 percent for that day and every later day, and later reports are ignored.

Return the percentage in effect at the end of each day.

Examples:

  • staged_rollout([5, 25, 100], 1, 100, 10, [[500, 5, 500, 5]] * 3) returns [25, 100, 100].
  • staged_rollout([5, 25, 100], 1, 100, 10, days) with days [500, 5, 500, 5], [500, 20, 500, 5], [500, 5, 500, 5], [500, 5, 500, 5], [50, 5, 500, 5], [500, 5, 500, 5], [500, 40, 500, 5], [500, 40, 500, 5], [500, 5, 500, 5] returns [25, 5, 25, 25, 25, 100, 25, 0, 0]. Day 2 breaches at 25% (4% bad against 1%) and drops to 5%. Back at 25%, that stage now needs two healthy days (days 4 and 6; day 5 is thin). Day 7 breaches at 100% and drops to 25%; day 8 is the second breach at 25%, so the rollout halts.
  • staged_rollout([10, 50], 3, 100, 10, [[100, 10, 100, 0]]) returns [0]: a breach at the first stage halts.

Examples

staged_rollout(stages: list[int], soak: int, min_tasks: int, margin: int, days: list[list[int]]) -> list[int]
Example 1
Inputstaged_rollout([5, 25, 100], 1, 100, 10, [[500, 5, 500, 5], [500, 5, 500, 5], [500, 5, 500, 5]])
Output[25, 100, 100]
Example 2
Inputstaged_rollout([5, 25, 100], 1, 100, 10, [[500, 5, 500, 5], [500, 20, 500, 5], [500, 5, 500, 5], [500, 5, 500, 5], [50, 5, 500, 5], [500, 5, 500, 5], [500, 40, 500, 5], [500, 40, 500, 5], [500, 5, 500, 5]])
Output[25, 5, 25, 25, 25, 100, 25, 0, 0]
Example 3
Inputstaged_rollout([10, 50], 3, 100, 10, [[100, 10, 100, 0]])
Output[0]