Billing charges duplicated and missing

On-call incidentMedium30 minStripe and Google style

Your task

A billing pipeline consumes a Pub/Sub subscription and writes invoice line-items to Postgres. Finance reports two problems this week: (1) a few customers were double-charged, and (2) one batch of charges appears missing. Dashboards: the subscription's ack_message_count looks healthy, oldest_unacked_message_age is low. Recent context: the consumer was changed three days ago to ack the Pub/Sub message first and then write to Postgres (to 'reduce ack-deadline expirations'), and the Postgres write has no uniqueness constraint on (invoice_id, line_item_id). There was a brief Postgres failover two days ago. Triage and explain both the duplicates and the loss.

It covers

  • Pubsub
  • At least once
  • Idempotency
  • Dedup