Shadow mode agreement report

Agent buildEasy15 min

Your task

Before a ticket-routing agent is allowed to act, it runs in shadow mode for two weeks: for every ticket, the action a person actually took is logged next to the action the agent would have taken. The agent may also abstain, meaning it declined to pick an action. The rollout review needs a short report from that log.

Write shadow_report(pairs). Each pair is [human_action, agent_action]: two strings, except that agent_action is None (null in JavaScript) when the agent abstained. Actions are compared as exact, case-sensitive strings, and an action can contain any character, | and : included. Return a dict with these keys:

  • "agreed": pairs where the agent proposed the same action as the human.
  • "disagreed": pairs where the agent proposed a different action.
  • "abstained": pairs where the agent abstained.
  • "agreement": agreed / (agreed + disagreed) as a whole percentage, rounded down. Abstentions are left out: they are measured separately, and counting them as misses would punish the agent for knowing when it is unsure. When the agent never proposed anything, this is None (null).
  • "top_miss": the most common disagreement as a pair [human_action, agent_action], or None (null) when there are no disagreements. If several pairs are equally common, return the one whose first occurrence comes earliest in the log.

Examples:

  • shadow_report([["refund", "refund"], ["refund", "escalate"], ["close", None], ["close", "close"], ["refund", "escalate"]]) returns {"agreed": 2, "disagreed": 2, "abstained": 1, "agreement": 50, "top_miss": ["refund", "escalate"]}.
  • shadow_report([["close", None]]) returns {"agreed": 0, "disagreed": 0, "abstained": 1, "agreement": None, "top_miss": None}.

Examples

shadow_report(pairs: list[list]) -> dict
Example 1
Inputshadow_report([["refund", "refund"], ["refund", "escalate"], ["close", None], ["close", "close"], ["refund", "escalate"]])
Output{"agreed": 2, "disagreed": 2, "abstained": 1, "agreement": 50, "top_miss": ["refund", "escalate"]}
Example 2
Inputshadow_report([["close", None]])
Output{"agreed": 0, "disagreed": 0, "abstained": 1, "agreement": None, "top_miss": None}