Malformed address redelivers forever

On-call incidentMedium30 minGoogle style

Your task

A Pub/Sub subscription enrichment feeds a worker that calls an external geocoding API. At 19:00 the subscription's oldest_unacked_message_age starts climbing and a small set of messages keep getting redelivered (high redelivery, low ack). Worker logs show those messages contain an address string that makes the geocoding call return HTTP 400, the worker treats 400 as a failure and nacks, and the message comes straight back. The subscription has no dead-letter topic configured and the worker doesn't cap attempts. Recent context: a data-import job last night ingested a batch with malformed addresses. Triage and mitigate; how do you keep these poison messages from blocking the subscription indefinitely?

It covers

  • Pubsub
  • Dead letter topic
  • Max delivery attempts