Logins fail as Redis hits its memory limit

On-call incidentEasy15 min

Your task

Since 13:50 about 30% of logins fail with HTTP 500. The auth service logs show OOM command not allowed when used memory > 'maxmemory' on SET session:<id>. People who are already logged in can keep browsing. Sessions live in a managed Redis node with 3 GB of memory and maxmemory-policy noeviction; the same node holds the API rate limiter's counters.

The memory graph has climbed in a straight line since Monday's release and reached the limit at 13:48. INFO keyspace shows db0:keys=31400000,expires=1100000. A SCAN sample shows most keys look like rl:203.0.113.7:202610171342 (client IP plus the minute), and TTL on them returns -1. Monday's release refactored the rate limiter. How do you triage and mitigate?

It covers

  • Redis
  • Maxmemory
  • Key ttl