Multi-consumer CDC with schema evolution
Your task
Design a change-data-capture pipeline that streams every insert/update/delete from a fleet of production OLTP databases into a data lake and several downstream consumers, with no gaps and no double-counting. You must handle initial backfill of huge existing tables, then switch seamlessly to live tailing, survive DB failovers, and propagate schema changes (a column added/renamed) without breaking consumers. Scale: 500 tables, 1M row-changes/sec aggregate.
It covers
- Wal
- Schema evolution
- Snapshot plus stream