Dead-letter topic overflow

On-call incidentHard40 minStripe style

Your task

Your payments side-effects pipeline uses the Kafka tiered-retry pattern: payments (main) to on handler failure, produce to payments.retry.5s to consumer waits and re-attempts to on failure produce to payments.retry.30s to on failure produce to payments.DLT (dead-letter topic). At 08:30 an alert fires: payments.DLT lag/size is climbing ~600 msg/min, but the main payments topic looks healthy (normal lag, normal error rate). Dashboards: the retry-topic consumers show a steady redelivery flow, and the DLT producer rate exactly tracks the retry-30s failure rate. Recent context: a downstream ledger API started returning HTTP 429 (rate-limited) at 08:25 after a quota change. How do you triage and mitigate?

It covers

  • Retry topic loop
  • Exhausted retries
  • Tier