Date format library incompatibility

On-call incidentHard40 min

Your task

Service A is deployed with a routine upgrade of its JSON serialization library (major version). A's tests pass; A is healthy. But downstream service B (unchanged, not redeployed) starts failing to parse a small fraction of messages from A's queue with a date-parse error. Context: the serialization library's new major version changed its DEFAULT date format from RFC3339 with nanoseconds to plain seconds-precision RFC3339 (and timezone offset format). B's parser was written to the old format. Dashboards: A's outbound is fine; B's parse-error rate stepped up at A's deploy time; only messages with sub-second-precision timestamps are affected. Triage, explain, and mitigate.

It covers

  • Serialization default
  • Compatibility
  • Cross service