Card decline logs go missing on some days

On-call incidentHard35 min

Your task

It is 10:15 on Thursday. The payments service writes JSON log lines to stdout. Fluent Bit sends them through Kafka, which keeps 4 days of data, to Logstash, which writes them to Elasticsearch 8. Each UTC day gets its own index, such as payments-logs-2026.10.29, from an index template that sets shards, replicas and a 30-day delete policy. This alert has been firing since 01:00:

DeclineLogsLow: card-decline lines in search 90% below last week for 1 hour (current 0, last week 1,480)

What you see:

  • The same alert fired all day on Monday and Wednesday, and not on Tuesday. The fraud team says Tuesday's decline lines are all in search.
  • Logstash has logged Could not index event to Elasticsearch with status 400 every hour since payments 3.2 went out last Thursday at 15:10: about 1,500 an hour on Monday, Wednesday and today, and about 1,200 an hour on Tuesday.
  • kubectl logs on a payments pod shows lines like {"level":"warn","msg":"card declined","error":{"code":"card_declined","decline_code":"insufficient_funds"}} and, since 3.2, {"level":"error","msg":"charge failed","error":"card network timeout"}.
  • The payments team offers to roll back 3.2 now.

What do you do this morning to stop losing decline lines, and how do you get back the ones already missing?

It covers

  • Elasticsearch
  • Logstash
  • Structured logs