Failing iPhone checkouts leave almost no traces

On-call incidentMedium25 min

Your task

It is 15:20. Your backend services use OpenTelemetry with OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.1. The tracing tool usually holds about 2,100 traces a minute from the iOS app. This alert fires:

IOSCheckoutErrors: 5xx above 3% for 10 minutes (current 6.2%)

What you see:

  • Web checkout is fine. In the tracing tool you find about 300 iOS traces a minute, and only 3 from failed iOS checkouts in the last 10 minutes.
  • iOS app 8.3 came out on Tuesday and now runs on 95% of iPhones. Its release notes say it adds OpenTelemetry to the app and traces 1% of requests on the device.
  • A request captured from iOS 8.3 carries traceparent: 00-6e0c63257de34c926f9efcd03927272e-a0d1e3c2b4f56789-00.
  • The API gateway logs every request with its route, upstream service, status and trace ID.

How do you find where iOS checkouts fail in the next 15 minutes, and what do you change so iOS requests are traced as before?

It covers

  • Opentelemetry
  • Mobile apps
  • Trace sampling