Code Room
On-callMediumoc-g567
Subject Model serving bad versionLevel Mid–Senior~30 minCommon in ML systems · Reliability & on-call interviewsIndustries Technology

Question

Four hours after a routine model promotion, the growth team escalates: checkout conversion is down 6% and add-to-cart from the 'related products' carousel is down 18%, but there are no errors, latency is normal, and the model returns 200s. Dashboards: the new model version v37 went to 100% at 10:00; offline eval looked slightly better than v36. Click-through on recommendations dropped at 10:05 and stayed down; the candidate-recall stage looks unchanged but the final ranking scores are more uniform (lower variance) than v36's. How do you triage and respond?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

Diagram & narrate the incident
Loading whiteboard…
Run or narrate your approach, then ask the coach.