Welcome emails delivered multiple times
Your task
A welcome-email worker consumes a Pub/Sub subscription and calls an email provider's send API per message. At 17:00 support gets a wave of 'I received the same welcome email 3–4 times' complaints. Dashboards: the subscription's oldest_unacked_message_age is elevated and redelivery is high, ack_message_count is healthy but the delivered/ack ratio is well above 1; worker CPU and the email send latency both crept up after a deploy at 16:50 that added an analytics call into the send path. The ack deadline is 60s. How do you triage and mitigate?
It covers
- At least once
- Duplicate side effects
- Non idempotent