Question
A write-heavy primary table (3B rows, 120k writes/sec — inserts, updates, deletes) needs a new secondary index added on a column to support a query that's currently doing full scans. You cannot take downtime and cannot let writes stall while the index builds, which will take hours. Design the online index build: how the index is backfilled from existing rows while concurrent writes keep mutating the table, how you guarantee the finished index is consistent (no missing or stale entries), and how you protect foreground write latency during the build.
Clarify scale and constraints first. Propose a clean component breakdown, then go deep on the hard parts — data model, bottlenecks, consistency, failure modes — and name the trade-offs you are making.