Multi-consumer CDC with schema evolution

DesignHard50 minDatabricks and Snowflake style

Your task

Design a change-data-capture pipeline that streams every insert/update/delete from a fleet of production OLTP databases into a data lake and several downstream consumers, with no gaps and no double-counting. You must handle initial backfill of huge existing tables, then switch seamlessly to live tailing, survive DB failovers, and propagate schema changes (a column added/renamed) without breaking consumers. Scale: 500 tables, 1M row-changes/sec aggregate.

It covers

  • Wal
  • Schema evolution
  • Snapshot plus stream