PR summary false claims
Your team uses an AI to auto-generate PR descriptions from the diff. On a PR that refactors a caching layer, the generated summary says 'This change improves cache hit rates by ~30% and fixes a memory leak.' Your actual diff just renames variables and extracts a helper — no benchmark, no leak fix. How do you handle AI-generated PR summaries so they help reviewers instead of misleading them?
Implement
filter_pr_claims(sentences: list[str]) → list[str]Examples
in
[["Renames cacheKey to cache_key across the module.","This change improves cache hit rates by ~30%.","Extracts a helper for eviction."]]out["Renames cacheKey to cache_key across the module.","Extracts a helper for eviction."]in
[["Fixes a memory leak in the eviction loop, see https://ci.example/bench/912."]]out["Fixes a memory leak in the eviction loop, see https://ci.example/bench/912."]What a strong answer looks like
Treat the AI’s output as a draft to verify, not an answer to trust. Name the specific flaw and the input that triggers it, say how you’d catch it (tests, edge cases, reading critically), and how you’d re-prompt or decompose to get it right.
0:00 of about 14 min
Vibe & agentic: describe the solution in plain language (or narrate it) and the coach grades your approach.
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.
Run or narrate your approach, then ask the coach.