Billing logs stop reaching search after midnight
Your task
It is 09:40. The billing service, a Java app, runs on 3 VMs. It writes its log to /var/log/billing/app.log, and Fluent Bit on each VM tails that file and sends new lines to the log search tool. At 09:31 this alert fired, and your teammate Ana is working it:
BillingDeclineRate: card declines above 5% for 10 minutes (current 11%)
What you see:
- Ana asks why the log search has no billing lines since last night. The last ones are from 00:00:04 on billing-1, 00:17:52 on billing-2 and 00:41:10 on billing-3.
- On billing-1,
ls -l /var/log/billing/showsapp.logat 0 bytes andapp.log.1at 9.4 GB. A minute laterapp.log.1has grown by about 1 MB. - Yesterday afternoon a change added
/etc/logrotate.d/billingfor that file, withdaily,rotate 7,compress,delaycompressandcreate 0640 billing billing. - Fluent Bit is running on all 3 VMs, and its own log has no errors.
How do you get Ana the billing lines she needs in the next few minutes, and what do you change before tonight?
It covers
- Log shipping
- Logrotate
- Fluent bit