Question
Design the rotation orchestration for a single shared database credential used by a fleet of 12,000 service instances that all talk to one primary Postgres. The mandate after an incident: rotate this credential every 24 hours with ZERO failed connections — no instance may ever present a password that the database has already invalidated, and the rotation must complete even if a few hundred instances are slow, restarting, or briefly partitioned. You control the secrets store, the DB, and an agent on each host. Walk through the rotation protocol, the window where both old and new credentials are valid, how you confirm fleet-wide adoption before retiring the old one, and how you handle stragglers without blocking rotation forever.
Clarify scale and constraints first. Propose a clean component breakdown, then go deep on the hard parts — data model, bottlenecks, consistency, failure modes — and name the trade-offs you are making.