Consumer stuck on oversized record

On-call incidentHard35 min

Your task

An analytics consumer (Kafka, 12 partitions, 6 instances) stops making progress on exactly one partition at 14:50. Dashboards: records-lag for partition 9 climbs linearly, the other 11 are at zero; the owning consumer logs a RecordTooLargeException then retries from the same offset forever; no crash, CPU low. Context: a producer change this morning started occasionally batching multiple events into one record, and a handful exceed the consumer's max.partition.fetch.bytes / fetch.max.bytes. How do you triage, unblock the partition, and prevent recurrence, without losing data?

It covers

  • Partition stall
  • Max message bytes
  • Head of line blocking
  • Kafka