On-callMediumoc-g570

Subject Inference queue backlogLevel Mid–Senior~30 minCommon in ML systems · Distributed systems · Algorithms & data structures interviewsIndustries Technology

Question

Your async inference service pulls jobs from a queue (document-embedding for a customer's nightly import). At 22:00 the queue depth starts climbing fast and ETA-to-drain is now 9 hours and rising; SLA says results within 1 hour. Dashboards: enqueue rate jumped 8x at 22:00 (a large enterprise customer's bulk import kicked off), worker GPU utilization is high but not pinned, dequeue rate is flat at its usual ceiling, and there are no errors. The fleet is at its configured max replicas. How do you triage and bring the backlog under SLA?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

Learn the concepts

Diagram & narrate the incident

Loading whiteboard…

Run or narrate your approach, then ask the coach.