Benchmark saturation resistance

You maintain a public benchmark that people rank models on. The items are public and must stay public, because a hidden test set means nobody can reproduce a score and nobody can audit an item, and the trust that buys is the only asset a leaderboard has. Everyone who cares about the ranking will optimize against it: by training on the items or their neighbours, by tuning output format to the scorer, by collecting data shaped like the benchmark.

Off the table: a private split, a per submission query limit that assumes adaptivity flows through your submission channel, and a fresh round of human annotation every cycle, because the budget buys one annotation round a year and not more.

Design the benchmark and the protocol around it so its ability to separate models degrades slowly, and so that you can tell apart the two reasons a score rose: models got better at the thing you meant to measure, or models got better at your benchmark. That separation, computed from data the leaderboard itself can collect, is the deliverable. Say what you would publish alongside a score so a reader can see how much of the remaining headroom is real.

Hand in the item design, the refresh or rotation policy and what it costs in comparability across time, the saturation statistic and its null, and the experiment that would show your statistic detects targeting rather than genuine capability gain. State the failure mode you accept and name the actor who can defeat your protocol cheaply.

Ties break toward the design that keeps longitudinal comparisons meaningful, and toward the one that is honest about when it stops being informative.