Inference determinism reproduction

A customer disputes what your assistant told them. The transcript is months old, the request has to be reproduced exactly enough to settle the dispute, and what counts as exactly enough will be argued by someone who is not an engineer. Your fleet is heterogeneous by necessity: several accelerator generations and more than one vendor, because parts age out and supply is what it is. Continuous batching is not negotiable either, so the batch a request lands in depends on which other tenants arrived in the same millisecond, and reduction order inside the fused kernels follows the batch shape. Decoding is greedy, which does not save you: an argmax tie broken the other way sends the continuation somewhere else entirely.

Design the serving architecture that makes replay possible, and be precise about what it promises. Hand in the guarantee, stated as the class of divergence you still admit and the class you have eliminated. Hand in the per-request record: schema, size, and retention cost at fleet scale. Hand in the path you take when the original hardware no longer exists, which it will not. Hand in the adversarial test in which a co-tenant chooses its own traffic to maximize the divergence your guarantee claims to exclude.

Pinning every request to one accelerator model forever is off the table, since the fleet outlives the part. So is retaining activations, majority voting over repeated runs, and asserting semantic equivalence without saying who adjudicates it and against what. Abandoning bit-exactness for a weaker invariant is a legitimate submission, but then the weaker invariant has to be defined, defended to the non-engineer, and shown to be checkable by a party who does not trust you.

Ties break toward the guarantee that is narrow and true over the one that is broad and hand-waved.