Model size GPU saturation

On-call incidentMedium30 minMeta and Nvidia style

Your task

You're paged on the recommendations serving tier. At 14:05 the p99 latency on POST /v1/recommend jumped from 90ms to 2.1s; p50 is still flat at 40ms. Error rate is normal, throughput unchanged. Dashboards: the model server's GPU utilization is pinned near 100% (was ~55%), the dynamic-batching queue-wait histogram has a fat tail, and inflight requests are up 4x. A deploy 30 minutes ago bumped the served model from a distilled checkpoint to the full-size one to 'improve quality.' How do you triage and mitigate?

It covers

  • Latency
  • P99
  • Batching
  • Rollback