Silent arithmetic errors at scale

A training job runs for weeks across thousands of accelerators and consumes the quarter's compute. In any fleet that size some devices return wrong arithmetic without raising anything: data-dependent, sometimes only under a particular thermal or voltage condition, sometimes on one functional unit for one input pattern. Memory correction covers storage and link checks cover transport, and neither covers a multiply that returns the wrong product and is then stored and transmitted correctly.

The relaxations are closed. Full redundant execution doubles a job that is already the budget. Halting the fleet for known-answer tests at a cadence that would catch a transient costs the throughput you are protecting. A bitwise-deterministic stack is not available, because the fast kernels are not deterministic and reduction order shifts with topology and with rank count after any restart, so replaying a suspect step and getting a different number is not evidence and getting the same number does not clear the device. And you cannot re-run the job to see whether the result differs. You have a stated throughput budget for detection, a small percentage, and you must name it.

Detection alone is not the deliverable. Corruption that begins at step two hundred thousand and is noticed at four hundred thousand means the checkpoints you kept may already carry it, so you owe a procedure for localizing to a device and deciding how far back the last trustworthy checkpoint is.

Hand in the detector: what it computes, where it sits in the step, and its throughput cost. Hand in the false-positive arithmetic, because at thousands of devices times thousands of steps any per-device test at a plausible rate fires constantly, and a detector that pages the on-call every hour is not deployed. Hand in the localization and rollback procedures. Name the corruption class you accept missing.

Ties break toward a detector whose cost does not grow with device count, and toward one that says what it does with a device that fails once and passes on retest.