Lag spikes during rolling restart
Your task
A Kafka Streams app doing stateful aggregation (RocksDB state stores, 48 partitions, 8 instances) is deployed via a rolling restart for a routine config change. During the rollout, end-to-end lag spikes to several million and stays elevated for ~25 minutes per instance bounce; the app's rebalance JMX metrics fire on every pod restart, and instances spend minutes in a RESTORING state reading from the __changelog topic. The cluster is otherwise healthy. This rolling restart happens on every deploy and each one causes a multi-minute lag spike. Triage why the lag spikes during deploys and how you'd mitigate it.
It covers
- Kafka
- State store
- Rebalance
- Static membership