AI in safety-critical medical firmware
You maintain firmware (in Rust) for a medical infusion pump that controls drug delivery rate. A new requirement says 'add a bolus dose mode' but the clinical safety constraints (max rate, lockout intervals, what happens on sensor fault) are only partially written down and live in domain experts' heads. A colleague wants to prototype the whole feature with an AI agent over the weekend. Make the call and justify it.
check_bolus_requests(request_minutes: list[int], lockout_minutes: int, max_doses_per_hour: int) → list[bool][[0,10,25,40],20,3]out[true,false,true,false][[0,15,30,45],15,3]out[true,true,true,false][[5,5,5],10,5]out[true,false,false]Treat the AI’s output as a draft to verify, not an answer to trust. Name the specific flaw and the input that triggers it, say how you’d catch it (tests, edge cases, reading critically), and how you’d re-prompt or decompose to get it right.
Vibe & agentic: describe the solution in plain language (or narrate it) and the coach grades your approach.