Trip updates fail as the morning rush builds
Your task
It is 08:20. The ride app stores trips in a DynamoDB table, trips, in provisioned mode. Capacity is counted in write capacity units (WCU): 1 WCU is one write a second of an item up to 1 KB. The table's own write capacity is fixed at 6,000 WCU. Yesterday the team added a global secondary index, by_driver_day, for a new earnings page: a second copy of every trip, keyed by driver and day. It finished building at 01:30. This alert fires:
TripWriteErrors: 17% of trip writes failed in the last 5 minutes (threshold 1%)
What you see:
- Failed writes log
ProvisionedThroughputExceededExceptionafter the SDK's retries. Drivers see "Could not update trip" when they tap Arrived. - The app sends about 2,400 trip writes a second, every item under 1 KB, spread over about 40,000 active trips. The table consumes about 2,000 of its 6,000 WCU.
- Since 08:05 the new index's consumed write capacity has been flat at 2,000 WCU. Before that it rose and fell with the table's.
- Overnight, at about 300 writes a second, nothing failed.
What do you change to stop the failures before the rush peaks, and what does every new index need from now on?
It covers
- Dynamodb
- Global secondary index
- Provisioned capacity