Nightly sales rollup for a store chain
Your task
Design the pipeline that gives a chain of 20 hardware stores a daily sales dashboard. Each store's point-of-sale system uploads a CSV file of the day's transactions to a folder in cloud storage after closing, about 4,000 rows per store. Some stores upload late, some upload a corrected file the next day, and now and then a file is cut off halfway. Managers want yesterday's sales by store, department and product, plus trends over two years, ready by 7 am. Describe ingestion, validation, the tables the dashboard reads, and how reruns and corrections work.
It covers
- File ingestion
- Idempotent loads
- Data validation