Customers charged twice

On-call incidentHard35 minStripe style

Your task

Finance reports that ~0.3% of customers were charged twice for the same order over the last 24h. Dashboards: the payments worker fleet was auto-scaled from 4 to 12 pods yesterday at noon to handle a backlog; the duplicate-charge rate is near zero before noon and rises sharply after. The order-events queue (SQS-style, at-least-once) occasionally redelivers messages. The charge handler does: read order, if order.status != 'charged' then call Stripe and set status='charged'. No DB row lock around that check-then-set. Triage, contain the bleeding, and explain the root cause.

It covers

  • Duplicate processing
  • Idempotency
  • At least once