Model ranking score collapse
Your task
Four hours after a routine model promotion, the growth team escalates: checkout conversion is down 6% and add-to-cart from the 'related products' carousel is down 18%, but there are no errors, latency is normal, and the model returns 200s. Dashboards: the new model version v37 went to 100% at 10:00; offline eval looked slightly better than v36. Click-through on recommendations dropped at 10:05 and stayed down; the candidate-recall stage looks unchanged but the final ranking scores are more uniform (lower variance) than v36's. How do you triage and respond?
It covers
- Bad deploy
- Business metric
- Rollback
- Canary