Kafka consumer lag climbs
Your task
Your order-fulfillment service consumes from a Kafka topic orders.placed (24 partitions). At 14:20 the consumer-group lag dashboard shows lag climbing linearly across ALL partitions, it's now 1.2M messages and growing ~3k/sec, and customers are reporting orders stuck in 'pending'. The consumers are healthy (not crashed, CPU low), but the processing-rate metric for the group has dropped to nearly zero. Logs show the same record being processed over and over: each attempt throws JsonParseException on a record at a fixed offset on partition 7, the consumer doesn't commit, rebalances, and retries from the same offset. A producer team deployed a schema change 30 minutes before this started. How do you triage, stop the bleeding, and prevent recurrence?
It covers
- Message queue
- Backlog
- Poison message