Question

Design the online inference serving layer for a 70B-parameter LLM behind a public API. Traffic is bursty and request shapes vary wildly: some requests generate 10 tokens, some generate 4000, and prompt lengths range from 50 to 100k tokens. You must maximize expensive-GPU utilization while keeping interactive latency (time-to-first-token) acceptable. Design the request scheduler and batching strategy, explain why naive fixed-size batching fails here, and how you autoscale a fleet whose warmup takes minutes.