Feature flag checks across a food delivery API

Your food delivery company runs its API on about 600 servers. Every request checks 10 to 30 feature flags: which checkout flow to show, which payment providers are on, which cities are open. A small flag service holds all 2,000 flags. Each server fetches the full set every 30 seconds and keeps it in memory.

New servers start from a machine image built at each deploy, and there are 3 to 5 deploys a day. Some flags are kill switches: last winter one kept a payment provider off for 3 days while that provider had an outage.

Last month the flag service was down for 50 minutes. Servers already running kept their last copy and were fine. In the same 50 minutes, autoscaling started about 150 servers for the dinner peak, and a deploy replaced 200 more. Each new server failed its first flag fetch and refused to start. Orders dropped by a third.

Change the design so the flag service being down never stops a server from starting or serving. A flag change must still reach every server within a minute.