Consumer crash-loops on malformed message

On-call incidentMedium30 min

Your task

A Kafka consumer for the inventory-updates topic (using a JSON/Avro deserializer) starts crash-looping at 16:20. Logs show a DeserializationException repeated thousands of times for the same offset on partition 3; the process catches the exception, logs it, and the container's restart policy brings it back, which re-reads the same offset and crashes again. Lag on partition 3 climbs while partitions 0–2 keep up. Recent context: a producer team deployed a schema change at 16:15 that emitted a few messages with a malformed payload before they rolled back. How do you triage and get the consumer unstuck, and how do you prevent the crash loop?

It covers

  • Kafka
  • Deserialization
  • Schema