Decide if an agent may write pump dosing code

Judgment callHard18 minAnthropic style

Your task

You maintain the Rust firmware for a hospital infusion pump, the device that controls how fast a drug flows into a patient. A colleague wants an AI agent to prototype a new bolus mode, a one-off extra dose on demand, over the weekend.

The safety limits for bolus mode are only partly written down: the maximum rate, the lockout time after a dose, and what to do when a sensor fails. The rest lives in the heads of 2 clinical engineers.

What do you tell your colleague? Decide what, if anything, the agent may build, and defend the call.

It covers

  • Safety critical
  • Ambiguous requirements
  • When not to use