A recent commentary in the journal NEJM AI made an argument that deserves more attention than it perhaps received: the regulatory line we have drawn around “AI” is the wrong line. The risk in clinical decision-making does not come from whether a tool uses a neural network. It comes from whether a tool changes what happens to a patient without anyone being able to explain why.
By that standard, a logistic regression risk score embedded in an order set is every bit as consequential as a deep learning model, and far more common. We have spent two years debating the governance of generative models while thousands of unvalidated, unmonitored scoring rules quietly shape triage, admission, and discharge decisions in every hospital in the country. Most have never been audited against the population they are applied to.
In our cardiac program, the distinction between “AI” and “algorithm” is meaningless at the bedside. A perfusion threshold, an arrhythmia-burden cutoff, and a diffusion model trained on synthetic ECG data are all decision rules. Each one needs the same things: a defined intended use, a validation cohort that resembles the patients it will see, continuous monitoring for drift, and a named human who is accountable when it is wrong.
The regulatory conversation should move off the word “AI” entirely. The question regulators and hospital boards should ask is simpler and harder: which decision rules in this institution change patient care, who validated them, on whom, and who is watching them now? An institution that cannot answer that for its simplest sepsis score has a sophistication problem its generative AI strategy will not solve.
Tighter regulation framed around clinical consequence rather than technical category would catch the tools that actually hurt people, and it would stop punishing the careful work of validating the ones that help.
Dr. Khalpey is chief medical AI officer at Atari AI, chair of applied clinical AI at the Atari AI Foundation and director of Khalpey AI Lab.
At the Becker’s 32nd Annual Meeting: The Business and Operations of ASCs, taking place October 29-31 in Chicago, ASC leaders, surgeons and healthcare executives will explore strategies to drive growth, enhance operational performance, navigate reimbursement challenges and prepare for the future of ambulatory surgery. Apply for complimentary registration now.
