Active learning under drift

You are running one preference-optimization run against a purchased pool of human annotators and a fixed budget of comparisons. Annotators are slow. A batch you send comes back hours later, by which time the policy has taken a few hundred optimizer steps, so every label you receive is about a policy that no longer exists. You must decide continuously which prompts to sample, which candidate pairs to send, and when to stop buying.

The relaxations are closed. You cannot re-run the training to compare two acquisition schedules, because the run is the budget. You cannot run two schedules in parallel against the same pool, because they contend for the same annotators and the pool's own standard drifts as its instructions get clarified. You cannot substitute a stronger model's judgments for the human ones, because the behaviours worth buying labels for are the ones where you already know that model and your annotators disagree. And you cannot measure the value of a label until the run is over, at which point nothing can be rewound.

Two confounds bite. Annotator disagreement is irreducible on a real fraction of pairs, so selecting the most uncertain comparison is partly paying for a coin flip. And an acquisition policy can look excellent by steering toward comparisons that are easy to win, which inflates agreement and reward without moving the thing you wanted.

Hand in the acquisition policy, including how it treats a stale label rather than discarding it. Hand in an estimator of the marginal value of the next label that is computable mid-run without finishing it, with the assumption it rests on. Hand in the stopping rule and what it trades. Hand in an experiment that fits inside one run's budget, could show your policy beats uniform sampling, and could come back negative.

Ties break toward a policy that degrades to uniform sampling when its value estimate is uninformative, over one that concentrates the budget wherever the estimate is highest.