On-callMediumoc-g387

Subject Redelivery stormLevel Mid–Senior~35 minCommon in Reliability & on-call interviewsIndustries Technology

Question

An SQS-driven `pdf-render` worker fleet alerts at 04:00: rendered PDFs are appearing in S3 2–3 times for the same job, and downstream gets duplicate 'render complete' webhooks. Dashboards: `NumberOfMessagesReceived` is ~2.3x `NumberOfMessagesDeleted`; `ApproximateAgeOfOldestMessage` is rising; the queue's visibility timeout is 30s. Recent context: a deploy at 03:50 added an embedded-font fetch step that made the average render time climb from ~12s to ~45s. The handler deletes the message only AFTER the render+upload completes. How do you triage and mitigate?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

Learn the concepts

Diagram & narrate the incident

Loading whiteboard…

Run or narrate your approach, then ask the coach.