Health Care AI, Intended to Save Money, Turns Out to Require a Lot of Expensive Humans
Preparing Cancer Patients for Difficult Decisions
Preparing cancer patients for difficult decisions is an oncologist’s job. They don’t always remember to do it, however. At the University of Pennsylvania Health System, doctors are nudged to talk about a patient’s treatment and end-of-life preferences by an artificially intelligent algorithm that predicts the chances of death.
A Routine Tech Checkup Reveals Algorithm Decay
But it’s far from being a set-it-and-forget-it tool. A routine tech checkup revealed the algorithm decayed during the COVID-19 pandemic, getting 7 percentage points worse at predicting who would die, according to a 2022 study. There were likely real-life impacts. Ravi Parikh, an Emory University oncologist who was the study’s lead author, told KFF Health News the tool failed hundreds of times to prompt doctors to initiate that important discussion — possibly heading off unnecessary chemotherapy — with patients who needed it.
Algorithm Glitches and the Dilemma of AI Maintenance
Algorithm glitches are one facet of a dilemma that computer scientists and doctors have long acknowledged but that is starting to puzzle hospital executives and researchers: Artificial intelligence systems require consistent monitoring and staffing to put in place and to keep them working well.
More Machines, More People, and Higher Costs
In essence: You need people, and more machines, to make sure the new tools don’t mess up. “Everybody thinks that AI will help us with our access and capacity and improve care and so on,” said Nigam Shah, chief data scientist at Stanford Health Care. “All of that is nice and good, but if it increases the cost of care by 20%, is that viable?”
Evaluating AI Products and Standards
Government officials worry hospitals lack the resources to put these technologies through their paces. “I have looked far and wide,” FDA Commissioner Robert Califf said at a recent agency panel on AI. “I do not believe there’s a single health system, in the United States, that’s capable of validating an AI algorithm that’s put into place in a clinical care system.”
AI is Already Widespread in Healthcare
AI is already widespread in healthcare. Algorithms are used to predict patients’ risk of death or deterioration, to suggest diagnoses or triage patients, to record and summarize visits to save doctors’ work, and to approve insurance claims.
Challenges in Evaluating AI Performance
Evaluating whether these products work is challenging. Evaluating whether they continue to work — or have developed the software equivalent of a blown gasket or leaky engine — is even trickier. Take a recent study at Yale Medicine evaluating six “early warning systems,” which alert clinicians when patients are likely to deteriorate rapidly. A supercomputer ran the data for several days, said Dana Edelson, a doctor at the University of Chicago and co-founder of a company that provided one algorithm for the study.
No Standards for Comparing AI Output
It’s not easy for hospitals and providers to select the best algorithms for their needs. The average doctor doesn’t have a supercomputer sitting around, and there is no Consumer Reports for AI. “We have no standards,” said Jesse Ehrenfeld, immediate past president of the American Medical Association. “There is nothing I can point you to today that is a standard around how you evaluate, monitor, look at the performance of a model of an algorithm, AI-enabled or not, when it’s deployed.”
Conclusion
The challenges of AI maintenance and evaluation are multifaceted and complex. While AI has the potential to revolutionize healthcare, it requires significant resources and attention to ensure its effectiveness and reliability. As the healthcare industry continues to adopt AI technologies, it is essential to prioritize the development of standards and metrics to evaluate their performance and ensure that they are used to improve patient care.
FAQs
Q: What are some common AI products used in healthcare?
A: Some common AI products used in healthcare include algorithms that predict patients’ risk of death or deterioration, suggest diagnoses or triage patients, record and summarize visits to save doctors’ work, and approve insurance claims.
Q: Why do AI algorithms require consistent monitoring and staffing?
A: AI algorithms require consistent monitoring and staffing to ensure they continue to work effectively and accurately. Without regular maintenance, AI algorithms can decay and produce incorrect results.
Q: What are some of the challenges of evaluating AI performance?
A: Some of the challenges of evaluating AI performance include the lack of standards for comparing AI output, the need for significant resources and attention to ensure AI effectiveness and reliability, and the complexity of evaluating AI performance in real-world settings.
Q: What are some potential solutions to the challenges of AI maintenance and evaluation?
A: Some potential solutions to the challenges of AI maintenance and evaluation include the development of standards and metrics to evaluate AI performance, the use of artificial intelligence monitoring artificial intelligence, and the investment of significant resources and attention to ensure AI effectiveness and reliability.
