Message visibility timeout

On-call incidentMedium35 minAmazon style

Your task

An SQS-driven pdf-render worker fleet alerts at 04:00: rendered PDFs are appearing in S3 2–3 times for the same job, and downstream gets duplicate 'render complete' webhooks. Dashboards: NumberOfMessagesReceived is ~2.3x NumberOfMessagesDeleted; ApproximateAgeOfOldestMessage is rising; the queue's visibility timeout is 30s. Recent context: a deploy at 03:50 added an embedded-font fetch step that made the average render time climb from ~12s to ~45s. The handler deletes the message only AFTER the render+upload completes. How do you triage and mitigate?

It covers

  • Visibility timeout
  • Slow processing
  • Duplicate work