Tell real agent speedup from a feeling of progress
Your task
2 engineers report opposite results from the same agent. Engineer A used it to convert 300 Mocha test files to Jest and says it saved a week.
Engineer B used it to design a cache invalidation scheme for a service running in 3 regions. It "felt like a week of progress in a day", but the design fell apart in review. You are their tech lead.
What explains the difference, and what rule do you give the team for when to lean on the agent?
It covers
- False confidence
- Speedup vs illusion