Scale-in terminates in-flight requests
After an evening traffic peak passes, your autoscaler scales the fleet in to save cost. Immediately you get a wave of errors: in-flight requests fail with connection resets, a batch of long-running export jobs vanish and never complete, and customers see 'export failed'. The dashboards show instances being terminated within seconds of the scale-in decision, mid-request. The service handles requests up to 3 minutes long and runs background jobs in-process. CPU/memory are fine — the errors are purely from instances disappearing. How do you triage and make scale-in safe?
Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.