Partition rebalance violates ordering

On-call incidentHard40 minStripe and Coinbase style

Your task

A wallet-balance service consumes a Kafka topic wallet-events keyed by user_id, applying credit/debit/adjust events that MUST be processed in per-user order. Last night, to add headroom, ops increased the topic's partition count from 12 to 24 (online, via kafka-topics --alter). This morning support reports a small number of users whose balance briefly went wrong before self-correcting, and one user is stuck in an inconsistent state. Dashboards: no lag, no consumer errors, no redeliveries, consumers look perfectly healthy. The producer uses the default partitioner (hash of key mod partition count). How do you triage and mitigate?

It covers

  • Partition increase
  • Key to partition
  • Rehash