N+1 query explosion after deploy

On-call incidentMedium25 min

Your task

A REST endpoint listing a project's tasks-with-assignees went from 90 ms to 4 s in p95 after a deploy, only for projects with many tasks. The DB's slow query log shows NO single slow query, but pg_stat_statements shows a tiny SELECT ... WHERE user_id = $1 query whose calls count exploded ~30x since the deploy. DB CPU is up but no individual query is slow. App traces show hundreds of tiny DB round-trips per request. Triage and fix, and explain why the slow-query log was empty.

It covers

  • N plus one
  • Orm
  • Latency amplification