Per-item contamination detection

You are auditing a benchmark you did not build, against a model you reach only through an inference API. The provider will not say what was trained on, and the benchmark has been public long enough that any crawl could have picked it up. Your customer wants more than a dataset level suspicion. They want a per item verdict: for each item, contaminated or not, with a false positive rate you can state and defend, because a flagged item gets pulled from the scored set and a wrong flag quietly deletes a legitimate measurement.

Off the table: the training corpus, any retraining or fine tuning, and any assumption that fresh items can be commissioned from the original authors. You may query the model, you may use token log probabilities where the API exposes them, and you may use public corpora, but you may not treat a public corpus as a stand in for the training set. You also may not use a set you merely believe is uncontaminated as your negative control unless you can argue it is drawn from the same distribution and the same time period as the items under test.

Hand in the statistic you compute per item, the null hypothesis it is tested against, how the null distribution is constructed under the constraints above, and the multiple comparison treatment across the whole benchmark. Then hand in the experiment that would show your statistic tracks memorization rather than item difficulty, item frequency on the open web, or item length. State the false negative rate you accept and who it hurts.

Ties break toward the submission whose negative control is constructed most defensibly, and toward the one most explicit about the regime where its verdict is worthless.