How to push broken code to production without anyone noticing.
Never deploy new code to 100% of your servers at once. If there's a fatal bug (like a missing environment variable or a bad config), the entire site goes down instantly.
Instead, use a Canary Rollout. You deploy the new version (v2) to just 5% of the servers. You monitor the error rate. If v2 starts throwing 500s, an automated system immediately routes traffic back to v1 (an Auto-Rollback). The blast radius is limited to 5%, and the outage is resolved in seconds, not hours.
# The Deployment Pipeline
1. Build & Test: Run unit/integration tests.
2. Bake (Canary): Deploy to 5% of servers. Wait 10 minutes.
- Watch the 'Golden Signals' (5xx Errors, Latency).
3. Evaluate:
if canary_error_rate > stable_error_rate:
# Immediate Auto-Rollback
route_traffic_to_stable(100%)
destroy_canary()
alert_oncall()
else:
# Proceed to full rollout
route_traffic_to_canary(100%)