Billing logs stop reaching search after midnight

On-call incidentEasy15 min

Your task

It is 09:40. The billing service, a Java app, runs on 3 VMs. It writes its log to /var/log/billing/app.log, and Fluent Bit on each VM tails that file and sends new lines to the log search tool. At 09:31 this alert fired, and your teammate Ana is working it:

BillingDeclineRate: card declines above 5% for 10 minutes (current 11%)

What you see:

  • Ana asks why the log search has no billing lines since last night. The last ones are from 00:00:04 on billing-1, 00:17:52 on billing-2 and 00:41:10 on billing-3.
  • On billing-1, ls -l /var/log/billing/ shows app.log at 0 bytes and app.log.1 at 9.4 GB. A minute later app.log.1 has grown by about 1 MB.
  • Yesterday afternoon a change added /etc/logrotate.d/billing for that file, with daily, rotate 7, compress, delaycompress and create 0640 billing billing.
  • Fluent Bit is running on all 3 VMs, and its own log has no errors.

How do you get Ana the billing lines she needs in the next few minutes, and what do you change before tonight?

It covers

  • Log shipping
  • Logrotate
  • Fluent bit