Measure whether AI tools help your engineering org

Judgment callHard25 minAnthropic and OpenAI style

Your task

Your VP saw a vendor slide claiming developers code 55% faster with AI. They want you to show the same for your 30-engineer org, to justify renewing the seats. You suspect the real picture is messier: some work is faster, and review queues have grown.

What do you measure, what will you refuse to report, and how do you keep the result honest if the answer is not 55%?

It covers

  • Measuring impact
  • Velocity vs defects
  • Metrics