Mocked test misses real behavior
An AI agent wrote a TypeScript test for a function that charges a customer via a payment gateway and records the charge in the DB. It reports 100% line coverage. Here's the test:
The test is green and coverage is high. What is wrong with this as a verification of chargeAndRecord, and how would you make it actually meaningful?
Implement
settle_charges(requests: list[str], charge_ids: list[str], insert_ok: list[bool]) → list[str]Examples
in
[["cus_1|500"],["ch_1"],[true]]out["charged:cus_1:500:ch_1","recorded:ch_1"]in
[["cus_1|500"],["ch_1"],[false]]out["charged:cus_1:500:ch_1","refunded:ch_1"]in
[["cus_1|5.00"],["ch_1"],[true]]out["invalid:cus_1|5.00"]What a strong answer looks like
Treat the AI’s output as a draft to verify, not an answer to trust. Name the specific flaw and the input that triggers it, say how you’d catch it (tests, edge cases, reading critically), and how you’d re-prompt or decompose to get it right.
0:00 of about 18 min
Vibe & agentic: describe the solution in plain language (or narrate it) and the coach grades your approach.
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.
Run or narrate your approach, then ask the coach.