Welcome emails delivered multiple times

On-call incidentMedium35 min

Your task

A welcome-email worker consumes a Pub/Sub subscription and calls an email provider's send API per message. At 17:00 support gets a wave of 'I received the same welcome email 3–4 times' complaints. Dashboards: the subscription's oldest_unacked_message_age is elevated and redelivery is high, ack_message_count is healthy but the delivered/ack ratio is well above 1; worker CPU and the email send latency both crept up after a deploy at 16:50 that added an analytics call into the send path. The ack deadline is 60s. How do you triage and mitigate?

It covers

  • At least once
  • Duplicate side effects
  • Non idempotent