Speculative decoding token selection

Speculative decoding buys throughput wherever the draft and the target agree. The complaint from production is that agreement comes cheapest exactly where it is worth least: boilerplate, formatting, the back half of a familiar identifier. At the tokens that decide an outcome, the branch taken in generated code, the operative digit, the name of the tool about to be called, agreement drops and the scheme degenerates toward one target step per token, which is where the latency budget was already going.

Your setting: a fixed target you may not fine-tune, a generic draft you may not train on the deployment workload, and no offline calibration pass over production traffic, because the traffic you care about is traffic you have not seen. You may keep the standard exactness guarantee, in which case the accepted output distribution must remain the target's, or you may relax it, in which case you owe a defined distortion measure and a bound on it. Quietly accepting close-enough tokens is the failure this question exists to catch.

Hand in one of two things. Either a scheme whose acceptance does not collapse at high-stakes tokens, with the mechanism that makes that true stated as something other than a better draft model. Or an argument that under exact distribution preservation the acceptance probability at a position is controlled by a quantity that is necessarily small at those tokens, which would make the observed behavior a property of the scheme rather than an engineering shortfall.

Either way, hand in a definition of a high-stakes token computable without the reference answer, and the experiment that would show your gains are not simply a corpus with more boilerplate in it.

Ties break toward the sharper argument, not the larger speedup.