Trust is earned one rung at a time, against a number somebody wrote down in advance.
Nobody launches an agent. They promote one. It starts by watching, then suggesting, then acting with a signature, then acting alone on the narrow slice it has proven, and each of those is a separate decision with evidence behind it.
What makes it a ladder rather than a gut feel is the measure. Run the agent in shadow beside the people already doing the work, compare its decision to theirs on the same case, and you get an agreement rate. Write down, before you start, the rate and the sample size that earn the next rung.
And write down the way back down at the same time. Traffic drifts, an upstream tool changes, a season turns, and the rate falls. A rollback rule agreed in advance costs an argument you have already had. One invented during the incident costs a much longer one.
The rollout does not run itself because your system asks for reduced motion. Next batch still works.
The promotion rule is a few lines, and writing it as code is the point: it is a decision made once, in advance, rather than every time somebody looks at a dashboard.
RUNGS = ["shadow", "suggest", "act with approval", "act alone"]
def review(state, batch):
"""One batch of cases, scored against what humans decided."""
rate = agreement(batch) # agreed / total, same cases
if rate >= state.bar and len(batch) >= state.min_cases:
state.streak += 1
if state.streak >= state.batches_needed and state.rung < len(RUNGS) - 1:
state.rung += 1 # promote, and reset the evidence
state.streak = 0
else:
state.streak = 0
if rate < state.bar - state.rollback_margin and state.rung > 0:
state.rung -= 1 # demote on the way down too
return state
Two details do the work. The streak resets on any batch that misses, so one good week cannot carry a bad one. And the demotion threshold sits below the promotion bar, so an agent hovering at the line does not oscillate between rungs every batch.
| Rung | Human effort | Time saved | What a failure costs |
|---|---|---|---|
| Shadow | None, it watches | None | Nothing, which is the point |
| Suggest | Read and accept | Some, thinking time | A bad suggestion accepted quickly |
| Act with approval | Sign off | Most of it | An approval given without reading |
| Act alone | Audit after the fact | All of it | Real, and discovered late |
Shadow mode is the cheapest evaluation available and the rung teams skip most often, because it produces no visible benefit while it runs. It is also the only rung where being wrong costs nothing at all.
A clinic wants an agent to manage appointment reminders and offer freed slots from the waitlist.
It starts in shadow: for every appointment, it records what it would have done, and a coordinator's actual decision is recorded beside it. After three batches above the bar it moves to suggesting, which a coordinator accepts or overrides in one click. Then acting with approval, then acting alone, but only for the reminder step. Offering a slot to a specific patient stays on the rung below, because the cost of getting that wrong is a person who took a morning off work.
Then flu season starts. Cancellation patterns change, agreement falls two batches running, and the rule that was written in week one pulls the agent back a rung without anyone needing to be persuaded. Run the panel above and watch that happen. Then raise the promotion bar and run it again: a stricter bar climbs the ladder more slowly and never gets caught out by the shift.
Your agent has agreed with human decisions on 96% of 500 cases in shadow mode for a month. Should it act alone?