Checkout slows by half a second after a release

On-call incidentEasy15 min

Your task

It is 10:40. The checkout service is a Python app on 12 pods. At 10:05 a release turned on OpenTelemetry tracing for it, sending spans to a collector in the cluster. This alert has been firing since 10:20:

CheckoutLatencyP95: p95 above 500 ms for 15 minutes (current 640 ms)

What you see:

  • p95, the time 95% of requests finish within, was 180 ms before 10:05 and has been about 640 ms since. CPU per pod rose from 35% to 38%.
  • A checkout request creates about 50 spans: one for the request, the rest for its database and cache calls.
  • The release added this tracing setup:
provider = TracerProvider(resource=resource)
provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter(endpoint=COLLECTOR_URL)))
trace.set_tracer_provider(provider)
  • The collector answers each export in about 9 ms.

What do you do now, and what do you change before tracing goes back on?

It covers

  • Opentelemetry
  • Python
  • Request latency