Some photo uploads fail since this morning

On-call incidentEasy15 min

Your task

You run the backend of a marketplace app where sellers upload photos of their items. At 10:15 you are paged:

UploadErrorRate: 18% of POST /api/listings/photos failed in the last 10m (threshold 2%)

What you see:

  • Before 09:30 the upload error rate was 0.3%. At 09:30 the platform team moved this service from its old NGINX load balancer to the cluster's shared ingress controller. The change note says "no app changes".
  • The mobile apps report HTTP 413 for the failed uploads. The response body starts with <html><head><title>413 Request Entity Too Large</title>.
  • The app's own access log has no entries for the failed uploads. The successful ones range from 180 KB to 980 KB.
  • Most failures come from the iOS app, which uploads photos at full quality.

What do you do in the next 15 minutes, and what do you fix after?

It covers

  • Ingress
  • Request size
  • Migration