Order drop alert fires every evening at the peak
Your task
It is 19:40 on Thursday. A business alert compares orders in the last 5 minutes with the same 5 minutes a week earlier. It counts order_created lines in the log search tool, which gets logs from Fluent Bit through Kafka and Logstash. This alert fires:
OrdersDropped: orders in the last 5 minutes below 50% of last week (current 212, last week 618)
What you see:
- It also fired on Monday, Tuesday and Wednesday, each time between about 19:15 and 20:45, and cleared on its own. It never fired before Monday.
- The
orderstable in the database gained 640 rows in the last 5 minutes. Payment success is normal. - On Monday the API team turned on request logging, which raised log volume by about 60%. The checkout web app also had a release on Monday.
- Logstash's 4 nodes run near 100% CPU from about 19:00 to 21:00 every evening this week. Before Monday they peaked near 60%. Logstash reports no errors.
What do you do about tonight's alert, and what do you change before tomorrow evening?
It covers
- Log based alerts
- Log pipeline
- Business alerts