Application logs for a payments company's servers

A payments company runs 400 application servers. An agent on each server reads the app's log files and sends them to a central log cluster, where engineers search them. Each server writes about 1 MB of logs a second at peak. The cluster can take in about 600 MB a second in total. Each server has a 100 GB disk, and the app itself uses 70 GB of it.

About a tenth of the log bytes come from the payment service, and auditors require every one of those lines to be kept. Request and debug logs may be lost.

Last month the log cluster was down for 3 hours. The agents held unsent logs in memory, and some ran out of memory and took the app down with them. When the cluster came back, every agent sent its backlog at once, and the cluster fell over again.

Where does the agent keep logs while the cluster is down, and how does the backlog get in once it is back?