Quantization behavior loss detection

You are handed a base checkpoint and a weight-quantized build of it produced by someone else. Your job is to decide, before that build goes near production traffic, which behaviors it has lost. You may run a bounded number of forward passes over unlabeled text you already hold. You may not fine-tune, you may not run quantization-aware training, you have no labeled downstream evaluation, and you have no sample of the deployment workload, because the deployment is new. Assume greedy and low-temperature decoding are both in use and that outputs feed parsers and tools, so one flipped token can fail a whole request.

The difficulty is that every cheap signal is a per-token measurement taken under teacher forcing, and the thing you care about is the divergence between two free-running processes over sequences hundreds of tokens long.

Hand in four things. First, an estimator of sequence-level divergence between base and quantized generation that fits inside your pass budget, with the derivation of what it does and does not bound. Second, a screening procedure that turns the estimator into a ranked list of behavior classes at risk, and states how a class gets onto the list when you hold no labels for it. Third, the failure mode you accept, named as a concrete category of degradation this procedure will systematically miss, not as a caveat about coverage. Fourth, the experiment that would show your estimator tracks behavioral divergence rather than tracking token entropy, which is the confound that makes a useless estimator look predictive.

Ties break toward the submission whose bound is honest about its own looseness and whose experiment could genuinely come back negative. A submission that proposes running the downstream evaluation has answered a different question.