Ledger events missing from warehouse

On-call incidentHard40 minStripe style

Your task

Finance reports that ~0.2% of ledger-events are missing from the downstream warehouse, reconciliation finds gaps with no corresponding Kafka record. No consumer lag, no errors in consumer logs, exactly-once is NOT enabled. Dashboards: the ledger-events topic has replication.factor=3 but min.insync.replicas=1; broker logs show two unclean leader elections in the last 24h during a rolling broker restart for an OS patch. The producer config is acks=1, retries=0. The gaps correlate in time with the broker restarts. Triage and explain the loss.

It covers

  • Kafka
  • Acks
  • Producer config