Plausible output false confidence
You ask an AI agent to write a complex SQL report that joins five tables to compute monthly revenue by cohort. It returns a 60-line query in seconds that runs without error and produces a plausible-looking number. You feel great about the time saved. Before you trust that number for a board deck, explain why this is a textbook false-confidence situation and how you'd convert the apparent speedup into a real one.
reconcile_joined_revenue(rows: list[str]) → list[int][["order_id=A1;amount_cents=1500","order_id=A1;amount_cents=1500","order_id=A2;amount_cents=2500"]]out[3,2,4000][["order_id=B1;amount_cents=1000","order_id=;amount_cents=9999","order_id=B2;amount_cents=-250"]]out[3,2,750]Treat the AI’s output as a draft to verify, not an answer to trust. Name the specific flaw and the input that triggers it, say how you’d catch it (tests, edge cases, reading critically), and how you’d re-prompt or decompose to get it right.
Vibe & agentic: describe the solution in plain language (or narrate it) and the coach grades your approach.