Change-data-capture pipeline

DesignHard45 minDatabricks and Snowflake style

Your task

Design a change-data-capture pipeline that replicates writes from a primary OLTP database to a search index, a cache, and a data warehouse, each downstream needing the database's changes in order and without loss. The DB does 50k writes/sec, downstreams have different speeds (the warehouse batches, the cache is real-time), and during a downstream outage the pipeline must not lose changes or apply them out of order. Describe how you capture changes, guarantee ordering/no-loss, deliver to heterogeneous consumers, and the central correctness trade-off.

It covers

  • Cdc
  • Change data capture
  • Exactly once