Isolate regression across changes

Between the previous large pretraining run and this one, the calendar forced six changes at once: the data mixture proportions, the tokenizer, one architectural detail, the optimizer schedule, the sequence length, and the numeric precision on new hardware. The new model is better on aggregate held-out loss and clearly worse on one downstream capability the product depends on. You have one run's budget for the next attempt. Decide what to revert, and justify it.

The relaxation everyone reaches for is not banned, it is put on trial. You may run small-scale proxies, and you must defend the transfer for each one, because rank order at proxy scale inverts for some interventions and the capability at issue is near chance on models small enough to sweep, so the metric you care about is not measurable where the experiments are affordable. Ablation at target scale is out, each one being a quarter. The mid-run branch, forking the run at a step and continuing two ways, is available only for changes that can be introduced mid-run, and the tokenizer, the early mixture composition and the architecture cannot, so you must say which of your six qualify. You hold the run's full artifacts: every checkpoint, per-step metrics, per-domain loss curves, and the same for the previous run.

Hand in the attribution plan. For each of the six changes, name the evidence class you would use, proxy experiment, mid-run branch, in-run correlational evidence, or none, and argue that the class is valid for that change rather than merely available. Give the transfer argument for every proxy result, naming the changes for which you would not trust it and why. Give the decision rule that turns the evidence into a revert list, including what you do when two evidence classes disagree. Give the check inside the next run that would show the attribution was wrong.

Ties break toward a plan that names in advance which of its conclusions the next run will falsify, over one that assigns a confidence to all six.