Trip updates fail as the morning rush builds

On-call incidentEasy15 min

Your task

It is 08:20. The ride app stores trips in a DynamoDB table, trips, in provisioned mode. Capacity is counted in write capacity units (WCU): 1 WCU is one write a second of an item up to 1 KB. The table's own write capacity is fixed at 6,000 WCU. Yesterday the team added a global secondary index, by_driver_day, for a new earnings page: a second copy of every trip, keyed by driver and day. It finished building at 01:30. This alert fires:

TripWriteErrors: 17% of trip writes failed in the last 5 minutes (threshold 1%)

What you see:

  • Failed writes log ProvisionedThroughputExceededException after the SDK's retries. Drivers see "Could not update trip" when they tap Arrived.
  • The app sends about 2,400 trip writes a second, every item under 1 KB, spread over about 40,000 active trips. The table consumes about 2,000 of its 6,000 WCU.
  • Since 08:05 the new index's consumed write capacity has been flat at 2,000 WCU. Before that it rose and fell with the table's.
  • Overnight, at about 300 writes a second, nothing failed.

What do you change to stop the failures before the rush peaks, and what does every new index need from now on?

It covers

  • Dynamodb
  • Global secondary index
  • Provisioned capacity