Continual learning regression detection

You fine-tune a vendor's base checkpoint every few weeks on fresh product data, and you have done it about twenty times over a year. Every update passed the gate: a regression suite, a preference evaluation, a canary. Users now report the model is worse at a class of task nobody wrote a test for, and it has been worse for months.

The difficulty is that most of what an update changes is what you paid it to change. A divergence detector against the previous checkpoint fires hardest on exactly the behaviour the update was for, so any threshold loose enough to admit the intended change is loose enough to admit collateral change of the same magnitude. Per update, the collateral loss sits inside the regression suite's confidence interval. Over twenty updates it does not.

What is closed off: the vendor's pretraining corpus, so parameter-importance regularizers have no data to estimate importance from. The update data from ten months ago, deleted under retention, so you cannot redo the chain. Reverting to an old checkpoint, which is not un-applying one update but discarding every update after it, including the behaviour the product currently depends on. And an ever-growing regression suite as the sole guard, since every test in it was written after a failure already shipped.

Design the program. Cover what you measure per update and what you measure continuously, the admission rule that blocks an update, the construction that separates intended change from collateral change and why it separates them, the sentinel design and the argument that it stays sensitive at update fifty rather than saturating, and what rollback means when you cannot un-apply. Name the class of capability whose loss you accept, and say what a user experiences when it goes.

Ties break toward a design whose per-update signal is still informative after a year of updates, over one that is sharper on the first update and decays.