Order drop alert fires every evening at the peak

On-call incidentMedium25 min

Your task

It is 19:40 on Thursday. A business alert compares orders in the last 5 minutes with the same 5 minutes a week earlier. It counts order_created lines in the log search tool, which gets logs from Fluent Bit through Kafka and Logstash. This alert fires:

OrdersDropped: orders in the last 5 minutes below 50% of last week (current 212, last week 618)

What you see:

  • It also fired on Monday, Tuesday and Wednesday, each time between about 19:15 and 20:45, and cleared on its own. It never fired before Monday.
  • The orders table in the database gained 640 rows in the last 5 minutes. Payment success is normal.
  • On Monday the API team turned on request logging, which raised log volume by about 60%. The checkout web app also had a release on Monday.
  • Logstash's 4 nodes run near 100% CPU from about 19:00 to 21:00 every evening this week. Before Monday they peaked near 60%. Logstash reports no errors.

What do you do about tonight's alert, and what do you change before tomorrow evening?

It covers

  • Log based alerts
  • Log pipeline
  • Business alerts