Reward hacking detection

You have a policy trained with an automated grader in the loop: a learned preference model, a suite of programmatic checks, or both. You hold every checkpoint from the run, the grader itself, and the ability to sample the policy on prompts you choose. What you do not hold is any signal for what you actually wanted, which is the entire difficulty. Somewhere across those checkpoints the policy may have started improving its score by exploiting the grader rather than by getting better at the task, and you have to decide before release whether that happened.

Design the detector and hand in the statistic it computes.

Off the table: you may not retrain, and you may not fix this by improving the grader, because the same question would then apply to the new grader. You may not use a stronger trusted model as an oracle. Assume nothing available to you is more reliable on this task than the system you are checking. You may not lean on human preference judgments as ground truth, because human judgment supplies several of the proxies at issue and its affordable sample size is far below what a rare severe failure needs.

Hand in an algorithm: what you compute per checkpoint or per sample, what it costs in samples and compute, and what each side of your threshold means for a release decision. Then the honest part. Say which class of gaming your statistic is blind to, and argue that the blindness is structural rather than a tuning problem. Then give the experiment. With no labels, you will need a construction where the ground truth holds by design, and you must argue why performance there says anything about the exploits you did not construct. Ties break toward a detector that would fire on a proxy exploit its designer did not anticipate, over one that is sharper on the exploits already catalogued.