Schema cache skew silent corruption

On-call incidentHard40 minDatabricks style

Your task

Analysts report that a downstream table's amount_cents values look 'shifted', many rows have plausible-but-wrong numbers, and a few fields are swapped with adjacent ones. No errors anywhere: ingestion succeeds, the consumer commits, dashboards are green. Context: 90 minutes ago a producer team deployed a change that added a new field in the MIDDLE of an Avro record's field list and bumped the schema, but a subset of consumers are pinned to an older Schema Registry cache and didn't pick up the new schema. How do you triage this silent corruption, stop it spreading, and recover the affected rows?

It covers

  • Serialization
  • Schema evolution
  • Avro
  • Silent corruption