Some photo uploads fail since this morning
Your task
You run the backend of a marketplace app where sellers upload photos of their items. At 10:15 you are paged:
UploadErrorRate: 18% of POST /api/listings/photos failed in the last 10m (threshold 2%)
What you see:
- Before 09:30 the upload error rate was 0.3%. At 09:30 the platform team moved this service from its old NGINX load balancer to the cluster's shared ingress controller. The change note says "no app changes".
- The mobile apps report
HTTP 413for the failed uploads. The response body starts with<html><head><title>413 Request Entity Too Large</title>. - The app's own access log has no entries for the failed uploads. The successful ones range from 180 KB to 980 KB.
- Most failures come from the iOS app, which uploads photos at full quality.
What do you do in the next 15 minutes, and what do you fix after?
It covers
- Ingress
- Request size
- Migration