Out-of-order processing
Your task
A device-state service consumes a single Kafka partition (per-device events are correctly ordered on the broker) and applies connect/config/disconnect transitions that must be applied in order per device. Support reports a few devices stuck 'connected' that should be 'disconnected.' Dashboards: no lag, no errors, no rebalances, no redeliveries, the consumer reads records strictly in order. Recent context: last week the consumer was 'optimized' by handing each polled record to a thread pool (executor.submit(processRecord)) so processing could parallelize. How do you triage and mitigate?
It covers
- Consumer concurrency
- Async handoff
- Per key thread