Change-data-capture pipeline
Your task
Design a change-data-capture pipeline that replicates writes from a primary OLTP database to a search index, a cache, and a data warehouse, each downstream needing the database's changes in order and without loss. The DB does 50k writes/sec, downstreams have different speeds (the warehouse batches, the cache is real-time), and during a downstream outage the pipeline must not lose changes or apply them out of order. Describe how you capture changes, guarantee ordering/no-loss, deliver to heterogeneous consumers, and the central correctness trade-off.
It covers
- Cdc
- Change data capture
- Exactly once