Card decline logs go missing on some days
Your task
It is 10:15 on Thursday. The payments service writes JSON log lines to stdout. Fluent Bit sends them through Kafka, which keeps 4 days of data, to Logstash, which writes them to Elasticsearch 8. Each UTC day gets its own index, such as payments-logs-2026.10.29, from an index template that sets shards, replicas and a 30-day delete policy. This alert has been firing since 01:00:
DeclineLogsLow: card-decline lines in search 90% below last week for 1 hour (current 0, last week 1,480)
What you see:
- The same alert fired all day on Monday and Wednesday, and not on Tuesday. The fraud team says Tuesday's decline lines are all in search.
- Logstash has logged
Could not index event to Elasticsearchwith status 400 every hour since payments 3.2 went out last Thursday at 15:10: about 1,500 an hour on Monday, Wednesday and today, and about 1,200 an hour on Tuesday. kubectl logson a payments pod shows lines like{"level":"warn","msg":"card declined","error":{"code":"card_declined","decline_code":"insufficient_funds"}}and, since 3.2,{"level":"error","msg":"charge failed","error":"card network timeout"}.- The payments team offers to roll back 3.2 now.
What do you do this morning to stop losing decline lines, and how do you get back the ones already missing?
It covers
- Elasticsearch
- Logstash
- Structured logs