Hot partition throttled

On-call incidentMedium30 minAmazon style

Your task

Your file-storage product keeps object metadata (path, size, ACLs, versions) in a partitioned NoSQL store (DynamoDB-style, partition key = parent_directory_id). On-call is paged: a subset of API calls return throttling/ProvisionedThroughputExceeded errors and p99 metadata-read latency spiked, while overall provisioned capacity utilization across the table reads only ~35%. Dashboards: one partition's consumed throughput is pegged at 100% while the rest are nearly idle; the errors all involve operations on one enormous directory that a customer is using as a dumping ground (millions of objects under a single parent_directory_id); a new 'sync everything' client started listing that directory in a tight loop this morning. How do you triage a hot-partition saturation on the metadata store?

It covers

  • Hot partition
  • Metadata store
  • Saturation
  • Throttling