Question
A Pub/Sub subscription `enrichment` feeds a worker that calls an external geocoding API. At 19:00 the subscription's `oldest_unacked_message_age` starts climbing and a small set of messages keep getting redelivered (high redelivery, low ack). Worker logs show those messages contain an address string that makes the geocoding call return HTTP 400, the worker treats 400 as a failure and nacks, and the message comes straight back. The subscription has *no* dead-letter topic configured and the worker doesn't cap attempts. Recent context: a data-import job last night ingested a batch with malformed addresses. Triage and mitigate; how do you keep these poison messages from blocking the subscription indefinitely?
Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.