Incremental extraction with deletes

DesignMedium35 minDatabricks and Snowflake style

Your task

Design an incremental extraction system that syncs ~200 large source Postgres/MySQL tables into a warehouse hourly, where full re-extracts are too slow/expensive. The naive approach polls 'rows where updated_at > last_watermark', but it misses hard deletes (rows that vanish), misses rows updated within the same second as the watermark, and double-counts when clocks skew. Design an incremental extract that captures inserts, updates, AND deletes correctly and is safe to re-run after a failure.

It covers

  • Incremental extraction
  • Watermark
  • Deletes