Model ranking score collapse
Four hours after a routine model promotion, the growth team escalates: checkout conversion is down 6% and add-to-cart from the 'related products' carousel is down 18%, but there are no errors, latency is normal, and the model returns 200s. Dashboards: the new model version v37 went to 100% at 10:00; offline eval looked slightly better than v36. Click-through on recommendations dropped at 10:05 and stayed down; the candidate-recall stage looks unchanged but the final ranking scores are more uniform (lower variance) than v36's. How do you triage and respond?
What a strong answer looks like
Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.
0:00 of about 30 min
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.
Run or narrate your approach, then ask the coach.