Scalable code review protocol

A system produces artifacts your reviewers cannot verify in the time they have. Take the concrete case. An agent submits changes to a large codebase, each one a few thousand lines spanning services no single reviewer owns, at a rate of hundreds a week. Tests pass, because the agent writes the tests. Your reviewers, in aggregate, have enough attention to read carefully about one percent of what arrives.

Design the oversight protocol that decides what merges.

The relaxations that make this easy are unavailable. You have no trusted stronger reviewer, human or model. Every automated helper you can field is drawn from the same capability class as the system under review and shares its blind spots. You have no ground truth labels, because if you had a checker that could tell you which changes were wrong you would run it and be done. You may not cut throughput to what humans can read, since that removes the reason the system exists. You may not escalate on the system's reported uncertainty, because the report is not verifiable.

Hand in the protocol: what each participant sees, what each is asked for, how disagreement resolves, and how the scarce human budget is allocated. Then argue for the property you are buying. Make explicit the claim about why, under your protocol, a flawed artifact is more likely to be surfaced than accepted, and name the assumption that argument rests on. Then attack it. Describe the strategy against which your protocol degrades into accepting whatever it is handed, and say whether you would notice from inside the system.

Finally, give the experiment. Since you have no labels in production, describe the offline construction that would measure whether the protocol catches flaws it was not designed around, and state the result that would tell you it is measuring reviewer convenience rather than artifact quality. Ties break toward protocols that stay useful when the system under review knows the protocol.