Logs missing from pods on one node

On-call incidentEasy15 min

Your task

At 11:30 a developer pings you: they are debugging a failed payment, and the log search tool has no lines at all from two of the payments pods after 10:05. The other payments pods log normally, and error rates and latency are at baseline. Logs reach the search tool through a Fluent Bit DaemonSet, one collector pod per node.

Both silent pods run on node ip-10-4-2-17. kubectl get pods -n logging -o wide shows the Fluent Bit pod on that node in CrashLoopBackOff with 14 restarts, last state OOMKilled, memory limit 100Mi. A batch job pod that writes about 40,000 log lines a second was scheduled onto the same node at 10:00. The Fluent Bit tail input keeps its file offsets in a DB file and has no Mem_Buf_Limit set. How do you triage and restore logging?

It covers

  • Log shipping
  • Oomkilled
  • Daemonset