Distributed tracing pipeline
Your task
Design a distributed tracing pipeline for a microservice platform of ~800 services producing 5M traces/min, average 40 spans/trace (so ~200M spans/min at peak). The business wants to keep essentially all error traces and all traces slower than the p99 latency for that endpoint, but can afford to store only ~2% of the healthy traces. Storage budget allows ~30 days hot. Design ingest, the sampling decision, and trace storage/retrieval by trace ID and by service+latency+error filters.
It covers
- Sampling
- Tail sampling
- Span collection