Large customer uploads stay stuck in processing
Your task
It is 11:15 on Thursday. Customers upload ZIP files of documents to an S3 bucket. Each upload starts a Lambda function, unpack-upload, which S3 invokes asynchronously: Lambda queues the event and runs it in the background. The function downloads the ZIP to /tmp, unzips it there, scans each file and marks the upload done. It has 2,048 MB of memory, 512 MB of ephemeral storage for /tmp, and a 5-minute timeout. This alert fires:
UnpackErrors: 45 failed invocations in the last 15 minutes (threshold 5)
What you see:
- The failures log
OSError: [Errno 28] No space left on devicewhile unzipping. - Every failing ZIP is between 300 and 600 MB. All come from Northwind, an enterprise customer that signed up on Tuesday and began moving its archive at 09:00 today. ZIPs under 200 MB, from every customer, still work.
- Each failing upload appears in the logs 3 times, for example at 11:02, 11:03 and 11:05.
- The function has no on-failure destination and no dead-letter queue.
- Northwind's admin says 140 of their uploads still show "Processing", most of them from this morning.
How do you get Northwind's uploads processed today, the stuck ones included, and what do you change in the function?
It covers
- Lambda
- Ephemeral storage
- Async retries