State store rebuild spikes lag

On-call incidentHard40 min

Your task

A Kafka Streams app doing stateful sessionization (RocksDB state stores backed by changelog topics, 48 partitions, 6 instances) is fine for weeks. At 02:00 one instance's pod is rescheduled to a new node by Kubernetes (node maintenance). Immediately, end-to-end lag on the 8 partitions that instance owns spikes to several million and stays elevated for ~18 minutes, then recovers on its own; the other 40 partitions are unaffected the whole time. CPU on the rescheduled instance is high during the window; disk I/O is saturated. No errors, no poison records. How do you triage and mitigate?

It covers

  • State restore
  • Standby replica
  • Rocksdb