Mixture-of-experts load isolation

A sparse mixture-of-experts checkpoint is your cheapest quality per dollar, and it is shared. Experts are distributed across accelerators, gating is input dependent, and the set of experts a batch activates is therefore chosen by whoever happens to be in the batch. A tenant who understands this can shape traffic to concentrate load on a few experts. Two things follow. Tail latency rises for everyone in that batch, and if you enforce a capacity limit some tokens are dropped at some layers, which no one sees as an error: the request completes, the answer is merely worse, for the wrong tenant, with nothing in the response saying so.

Design the serving layer. You may not retrain or rebalance the gate, since that is a training-time answer to a serving-time problem and the checkpoint is given. You may not hand every tenant dedicated accelerators, since sharing is the economics you were buying. You may not re-run a dropped request without capacity limits to see what it would have said, because at fleet scale that doubles the cost of exactly the traffic that is already expensive.

Hand in an architecture with a per-request tail-latency target and a stated quality invariant, both of which must survive a co-tenant chosen to break them. Hand in a bound on how much one adversarial tenant can degrade a neighbor, in whatever units you are willing to defend. Hand in a server-side detector that flags a request whose answer was degraded by routing pressure, built only from signals you already have, gate distributions, per-layer drop counts, queue depth, and not from a counterfactual run. Hand in the experiment that would show your detector fires on degradation rather than merely on load.

Ties break toward the design that admits which degradations stay invisible.