Measuring AI impact honestly
Your VP loved a vendor slide claiming '55% faster coding with AI' and wants you to prove the same for your 30-engineer org so they can justify the seat spend. You suspect the real picture is messier. Design the measurement: what do you actually track to know whether AI is helping or just moving work around, what vanity metrics will you refuse to report, and how do you avoid a measurement that makes the tool look good while the system gets worse?
summarize_ai_impact(metric_names: list[str], before_values: list[float], after_values: list[float]) → list[str][["lead_time_hours","lines_of_code"],[40,1000],[30,4000]]out["lead_time_hours:better"][["change_failure_rate","deploy_frequency_per_week"],[10,8],[16,12]]out["change_failure_rate:worse","deploy_frequency_per_week:better"][["prs_per_engineer","commit_count"],[5,100],[9,300]]out[]Treat the AI’s output as a draft to verify, not an answer to trust. Name the specific flaw and the input that triggers it, say how you’d catch it (tests, edge cases, reading critically), and how you’d re-prompt or decompose to get it right.
Vibe & agentic: describe the solution in plain language (or narrate it) and the coach grades your approach.