Credential stuffing evades login alarms

On-call incidentHard40 minPayPal and Capital One style

Your task

Overnight your login endpoint's success rate barely moved (down 2%) and total volume is only up ~15%, nothing tripped your failed-login or volume alarms. But fraud reports of unauthorized logins climb the next morning. Digging in: a credential-stuffing run used a known-good combo list, so most attempts succeeded on the first try (low failure rate by design), spread across ~80k residential-proxy IPs at <1 req/min each, mimicking real user-agents and pacing. The attempts cluster on accounts that have no 2FA. How do you confirm this is stuffing (vs. a normal night), contain it, and harden, while a Black-Friday-style real traffic surge is also happening so you can't just clamp logins?

It covers

  • Credential stuffing
  • Anomaly detection
  • Rate limiting
  • False positives