Feature flag service outage fallback

On-call incidentMedium30 min

Your task

Your services fetch feature flags and dynamic config from a managed flag service (LaunchDarkly-style) at startup and via a streaming connection. At 16:00 the flag service has an outage. Within minutes, several of your services start misbehaving, some new pods crash on boot, and a few features flip to their 'off/default' state, breaking flows that depended on a flag being ON. Dashboards: the flag SDK logs streaming-connection errors and 'falling back to defaults'; your own services are otherwise healthy. How do you triage and mitigate?

It covers

  • Config service
  • Feature flags
  • Fail closed