Logs missing from pods on one node
Your task
At 11:30 a developer pings you: they are debugging a failed payment, and the log search tool has no lines at all from two of the payments pods after 10:05. The other payments pods log normally, and error rates and latency are at baseline. Logs reach the search tool through a Fluent Bit DaemonSet, one collector pod per node.
Both silent pods run on node ip-10-4-2-17. kubectl get pods -n logging -o wide shows the Fluent Bit pod on that node in CrashLoopBackOff with 14 restarts, last state OOMKilled, memory limit 100Mi. A batch job pod that writes about 40,000 log lines a second was scheduled onto the same node at 10:00. The Fluent Bit tail input keeps its file offsets in a DB file and has no Mem_Buf_Limit set. How do you triage and restore logging?
It covers
- Log shipping
- Oomkilled
- Daemonset