Autonomy level from eval results and blast radius

Agent buildEasy15 min

Your task

Before an agent task type may act on its own, the platform team sets its autonomy level from two facts: how it did on the offline eval suite (passes out of runs) and its blast radius ("low", "medium" or "high": how much damage one bad action can do).

Write autonomy_level(passes, runs, blast_radius). It returns one of four levels, from most to least freedom:

  • "auto": the agent acts on its own.
  • "approve": the agent acts only after a human approves each action.
  • "suggest": the agent drafts the action and a human carries it out.
  • "off": the task type does not run.

Step 1. The pass rate passes / runs picks a level from this table. Thresholds are inclusive: a pass rate exactly on a threshold earns that level. Below the suggest threshold the level is "off".

Blast radiusautoapprovesuggest
low95%85%60%
medium98%90%70%
highnever97%80%

Step 2. A small eval proves little, so the run count caps the level. With fewer than 100 runs the level is at most "approve"; with fewer than 20 runs it is at most "suggest". A cap only ever lowers a level, never raises it.

Fail closed. Zero runs, or a blast radius that is not exactly one of the three strings (case matters), gives "off".

You can rely on 0 <= passes <= runs. Compare without floating-point rounding, for example passes * 100 >= runs * 95.

Examples:

  • autonomy_level(196, 200, "low") returns "auto": 98% clears 95%, and 200 runs is enough evidence.
  • autonomy_level(196, 200, "high") returns "approve": a high blast radius never earns auto.
  • autonomy_level(19, 19, "low") returns "suggest": 100% would be auto, but 19 runs caps it at suggest.

Examples

autonomy_level(passes: int, runs: int, blast_radius: str) -> str
Example 1
Inputautonomy_level(196, 200, "low")
Output"auto"
Example 2
Inputautonomy_level(196, 200, "high")
Output"approve"
Example 3
Inputautonomy_level(19, 19, "low")
Output"suggest"