Near-duplicate media detection

DesignMedium40 minGoogle style

Your task

Design a near-duplicate detection system for a media platform ingesting 3M images/videos a day that must flag content that is a re-upload or trivial edit (crop, re-encode, watermark, slight color shift) of existing content, for copyright/dedup, not exact-hash matching, which a single re-encode defeats. Walk through the embedding/fingerprinting approach, how you search 2B existing fingerprints fast enough to check each upload at ingest, and how you set the match threshold without drowning in false positives.

It covers

  • Near duplicate
  • Dedup
  • Lsh
  • Clustering