Duplicate payment captures after rebalance

On-call incidentHard40 minStripe style

Your task

A Kafka consumer group processes payment.capture events and calls a payment gateway, then commits offsets. The team believes processing is 'exactly once'. At 11:05 an autoscaler added two pods, triggering a consumer-group rebalance. Over the next 10 minutes, ~300 captures were sent to the gateway twice. Dashboards: consumer lag briefly spiked then recovered, no errors, no crashes, just the rebalance. Each handler takes ~4s (a slow gateway call) before it commits. How do you triage, stop the double-captures, and reconcile the duplicates?

It covers

  • Consumer rebalance
  • Offset commit gap
  • At least once
  • Kafka