Coordinator bottleneck on scale

On-call incidentHard30 minDatabricks and Snowflake style

Your task

Your distributed query engine scales workers horizontally and normally near-linearly. After onboarding a large analytics customer you added 3x more worker nodes to handle their load, but total throughput barely improved and per-query latency got worse. Worker CPU is moderate. Tracing shows every query, on start and on each stage transition, calls a single central coordinator/metadata service to fetch the schema and assign splits, and that coordinator is now at 100% CPU with a growing request queue. How do you triage, and what's the real fix?

It covers

  • Coordinator bottleneck
  • Metadata service
  • Fan out