Consumer rebalance duplicate charges

On-call incidentMedium30 minStripe style

Your task

Support tickets report customers charged twice for the same order over the last hour. Dashboards: the payment-service consumer that processes a Kafka order.placed topic shows its rebalance count jumped (3 rebalances in 30 min), processing latency per message rose near the max.poll.interval.ms ceiling, and charge_created counts in your payment provider are ~1.8x the order.placed counts for the window. No code deploy happened; a noisy-neighbor caused GC pauses on two consumer pods earlier. How do you confirm the cause, stop the double-charges now, and reconcile the customers who were already charged twice?

It covers

  • At least once
  • Idempotency
  • Webhooks
  • Double charge