Model routing under uncertainty

You serve a mixed fleet: one small fast model, one large slow one, a couple of specialists. The routing decision has to be made before generation starts, because the interactive slice has a first-token budget that a run-small-then-judge cascade spends on its own. In production you never learn whether the other choice would have been better. The counterfactual is not observed, thumbs-up signals are sparse and biased toward users who already got something usable, and there are no human labels. You have a small fixed exploration budget, a low single-digit share of traffic you may deliberately route the wrong way, and no more, because the large model is the cost problem you were sent to solve.

Design the routing layer and the measurement that keeps it honest. Hand in the router and the features it may use before a single token is generated. Hand in an estimator of what the routing is costing you, built from the exploration budget alone, with its variance and the assumptions that make it valid. Hand in a drift detector, stated with what it can and cannot catch, since the router shapes the traffic it will later be evaluated on. Hand in the experiment that would show the router is tracking difficulty rather than tracking surface features which correlate with difficulty in this quarter's traffic and not the next one's, prompt length, language and customer identity being the obvious three.

Off the table: a judge model on the interactive path, shadow-routing every request to both models, human labeling, and any assumption that the workload is stationary. Also off the table is a router retrained nightly on its own logs without an argument for why that loop converges to something other than its own habits.

Ties break toward the submission clearest about what it cannot measure.