Model ranking score collapse

On-call incidentMedium30 minAmazon style

Your task

Four hours after a routine model promotion, the growth team escalates: checkout conversion is down 6% and add-to-cart from the 'related products' carousel is down 18%, but there are no errors, latency is normal, and the model returns 200s. Dashboards: the new model version v37 went to 100% at 10:00; offline eval looked slightly better than v36. Click-through on recommendations dropped at 10:05 and stayed down; the candidate-recall stage looks unchanged but the final ranking scores are more uniform (lower variance) than v36's. How do you triage and respond?

It covers

  • Bad deploy
  • Business metric
  • Rollback
  • Canary