Orders charged twice
Your task
Your order-processing consumer reads from Kafka and writes to Postgres. At 13:00 finance reports some orders are charged twice. Dashboards: the consumer group rebalanced at 12:58 (a pod was OOM-killed and restarted); consumer lag briefly spiked then recovered; you commit Kafka offsets AFTER processing each message; the charge call to the payment provider and the DB insert are separate steps with no shared transaction. No code deploy. How do you triage and mitigate?
It covers
- Idempotency
- Exactly once
- Message redelivery