Sidecar v2 cannot resolve upstream

On-call incidentHard40 min

Your task

A platform team rolls out a new service-mesh sidecar (v2) to the inventory service via a 50% canary at 13:00. The 50% of inventory pods with the v2 sidecar start failing all outbound calls to the pricing service with 'no healthy upstream', while the 50% on v1 are fine. Pricing service itself is healthy and serving the v1-sidecar pods normally. The mesh control plane shows pricing's endpoints as healthy. The v2 sidecar release notes mention a change to how it resolves upstream service names. Triage and mitigate.

It covers

  • Dns
  • Service discovery
  • Partial outage