Question

Design the production-quality monitoring system for a customer-facing LLM agent (refunds, account changes) where there is no clean numeric label for 'was this answer correct' and a wrong action can cost money or break trust. Volume is millions of conversations a day, so you cannot human-review them all. Walk through what quality signals you can actually measure online, how you'd use automated graders/LLM-judges responsibly, and how you catch a quality regression caused by a prompt or model change before it spreads.