Batch photo processing pipeline

DesignHard45 minGoogle and Apple style

Your task

Design the batch image-processing pipeline for a photo-cloud product that, on upload, must run a chain of ML and CV steps per photo, EXIF extraction, face detection + clustering, object/scene labels, OCR, and NSFW screening, for 2B photos/day, and must also support re-running the whole chain over the entire 500B-photo back-catalog when a model is upgraded. Design the per-photo pipeline, how you schedule the steps, and how you do a full backfill without melting the live ingest path.

It covers

  • Batch pipeline
  • Ml inference
  • Backfill