Orders charged twice

On-call incidentMedium35 minStripe style

Your task

Your order-processing consumer reads from Kafka and writes to Postgres. At 13:00 finance reports some orders are charged twice. Dashboards: the consumer group rebalanced at 12:58 (a pod was OOM-killed and restarted); consumer lag briefly spiked then recovered; you commit Kafka offsets AFTER processing each message; the charge call to the payment provider and the DB insert are separate steps with no shared transaction. No code deploy. How do you triage and mitigate?

It covers

  • Idempotency
  • Exactly once
  • Message redelivery