Coordinator bottleneck on scale
Your task
Your distributed query engine scales workers horizontally and normally near-linearly. After onboarding a large analytics customer you added 3x more worker nodes to handle their load, but total throughput barely improved and per-query latency got worse. Worker CPU is moderate. Tracing shows every query, on start and on each stage transition, calls a single central coordinator/metadata service to fetch the schema and assign splits, and that coordinator is now at 100% CPU with a growing request queue. How do you triage, and what's the real fix?
It covers
- Coordinator bottleneck
- Metadata service
- Fan out