Error traces show only some of their services
Your task
It is 16:30. Your 30 services send spans to the OpenTelemetry Collector over gRPC. The collector runs as a Kubernetes Deployment behind a ClusterIP Service. It holds each trace's spans for 30 seconds after the first one arrives, then decides whether to keep the trace: every trace with an error, every trace slower than 2 s and 5% of the rest. A trace is all the spans of one request across services. This alert fires:
CheckoutErrors: 5xx above 2% for 10 minutes (current 4.1%)
What you see:
- The engineers working on checkout say failing traces are missing some of the 9 services a checkout passes through, a different set each time. Many spans have a parent the tool cannot find.
- Until Tuesday, failing checkout traces showed all 9 services.
- On Tuesday the single collector replica was killed twice for running out of memory at the evening peak, so it was scaled to 3 replicas. Each now uses about 30% of its 4 GiB limit.
- The collector's logs show no errors, and its exporter to the tracing backend reports no failed sends.
What do you change so the checkout team gets whole error traces today, and how should the collector grow from here?
It covers
- Opentelemetry collector
- Trace sampling
- Kubernetes