Checkout slows by half a second after a release
Your task
It is 10:40. The checkout service is a Python app on 12 pods. At 10:05 a release turned on OpenTelemetry tracing for it, sending spans to a collector in the cluster. This alert has been firing since 10:20:
CheckoutLatencyP95: p95 above 500 ms for 15 minutes (current 640 ms)
What you see:
- p95, the time 95% of requests finish within, was 180 ms before 10:05 and has been about 640 ms since. CPU per pod rose from 35% to 38%.
- A checkout request creates about 50 spans: one for the request, the rest for its database and cache calls.
- The release added this tracing setup:
provider = TracerProvider(resource=resource) provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter(endpoint=COLLECTOR_URL))) trace.set_tracer_provider(provider)
- The collector answers each export in about 9 ms.
What do you do now, and what do you change before tracing goes back on?
It covers
- Opentelemetry
- Python
- Request latency