Change-data-capture sync pipeline
Your task
Design a change-data-capture pipeline that keeps a denormalized search index (Elasticsearch) and a cache continuously in sync with a sharded primary Postgres holding product/listing data, with ~30k row mutations/sec. Requirements: downstream views must never permanently diverge from the DB, updates should land within a few seconds, and a transaction that updates a listing row plus an inventory row must not be observed half-applied. Walk through how you capture changes, preserve order, and bootstrap a new consumer.
It covers
- Outbox pattern
- Ordering
- Snapshot bootstrap