Latency spikes with regex validation

On-call incidentHard40 min

Your task

An HTTP request-validation service (single-threaded event loop, Node.js) goes from p50 8ms / p99 30ms to p50 9ms / p99 9000ms at 14:10. Throughput collapses and the health check intermittently times out. CPU on the box pegs at 100% on one core; the others are idle. There was no deploy today, but marketing launched a campaign at 14:00 that drove a 3x traffic spike from a new partner whose user-supplied 'coupon code' field is validated with a regex. The event-loop-lag metric spiked from <1ms to multiple seconds. How do you triage and mitigate?

It covers

  • Regex redos
  • Tail latency
  • Flame graph