Data lineage impact analysis
Your task
A data platform has ~10k tables, ~4k dbt/Spark transformation jobs, and hundreds of BI dashboards and ML features, all interdependent. An engineer who changes or drops a column has no idea what breaks downstream, so changes ship and silently corrupt dashboards discovered days later, and on-call can't tell which upstream failure caused a given dashboard to go stale. Design a lineage system that gives pre-merge impact analysis ('what does this change break') and post-incident root-cause ('what upstream broke this dashboard').
It covers
- Impact analysis
- Blast radius
- Column level