Health Care AI, Intended to Save Money, Turns Out to Require a Lot of Expensive Humans

Preparing Cancer Patients for Difficult Decisions

Preparing cancer patients for difficult decisions is an oncologist’s job. They don’t always remember to do it, however. At the University of Pennsylvania Health System, doctors are nudged to talk about a patient’s treatment and end-of-life preferences by an artificially intelligent algorithm that predicts the chances of death.

A Routine Tech Checkup Reveals the Algorithm’s Decay

But it’s far from being a set-it-and-forget-it tool. A routine tech checkup revealed the algorithm decayed during the COVID-19 pandemic, getting 7 percentage points worse at predicting who would die, according to a 2022 study.

Real-Life Impacts and the Need for Consistent Monitoring

There were likely real-life impacts. Ravi Parikh, an Emory University oncologist who was the study’s lead author, told KFF Health News the tool failed hundreds of times to prompt doctors to initiate that important discussion — possibly heading off unnecessary chemotherapy — with patients who needed it.

Algorithm Glitches and the Dilemma of Artificial Intelligence

Algorithm glitches are one facet of a dilemma that computer scientists and doctors have long acknowledged but that is starting to puzzle hospital executives and researchers: Artificial intelligence systems require consistent monitoring and staffing to put in place and to keep them working well.

Everybody Thinks AI Will Help Us, But at What Cost?

“In essence: You need people, and more machines, to make sure the new tools don’t mess up,” said Nigam Shah, chief data scientist at Stanford Health Care. “Everybody thinks that AI will help us with our access and capacity and improve care and so on,” said Shah. “All of that is nice and good, but if it increases the cost of care by 20%, is that viable?”

Evaluating AI Products and the Lack of Standards

Evaluating whether these products work is challenging. Evaluating whether they continue to work — or have developed the software equivalent of a blown gasket or leaky engine — is even trickier.

The Need for Standards and the Problem of Comparing Output

Take a recent study at Yale Medicine evaluating six “early warning systems,” which alert clinicians when patients are likely to deteriorate rapidly. A supercomputer ran the data for several days, said Dana Edelson, a doctor at the University of Chicago and co-founder of a company that provided one algorithm for the study.

The Challenges of Selecting the Best Algorithms

It’s not easy for hospitals and providers to select the best algorithms for their needs. The average doctor doesn’t have a supercomputer sitting around, and there is no Consumer Reports for AI.

The Problem of Errors and the Need for Human Oversight

Sometimes the reasons algorithms fail are fairly logical. For example, changes to underlying data can erode their effectiveness, like when hospitals switch lab providers. Sometimes, however, the pitfalls yawn open for no apparent reason.

The Need for Human Oversight and the High Cost of AI Monitoring

If metrics and standards are sparse and errors can crop up for strange reasons, what are institutions to do? Invest lots of resources. At Stanford, Shah said, it took eight to 10 months and 115 man-hours just to audit two models for fairness and reliability.

Conclusion

The integration of artificial intelligence into healthcare has the potential to revolutionize the way we deliver care, but it also requires a significant investment of resources and human oversight. As the technology continues to evolve, it is essential that we prioritize the development of standards and metrics to ensure that AI systems are reliable, accurate, and effective.

FAQs

Q: What is the purpose of AI in healthcare?
A: The purpose of AI in healthcare is to improve the delivery of care by automating tasks, analyzing data, and providing insights to clinicians.

Q: How do AI algorithms decay?
A: AI algorithms can decay due to changes in underlying data, lack of maintenance, or other factors.

Q: What are the challenges of evaluating AI products?
A: The challenges of evaluating AI products include the lack of standards, the need for complex technical expertise, and the difficulty of comparing output.

Q: How can institutions ensure the reliability and accuracy of AI systems?
A: Institutions can ensure the reliability and accuracy of AI systems by investing in human oversight, developing standards and metrics, and conducting regular audits and testing.

Q: What is the cost of AI monitoring?
A: The cost of AI monitoring can be high, requiring significant investments of resources and human expertise.