The misuse of artificial intelligence chatbots tops this year’s list of health technology hazards from ECRI, an independent non-profit dedicated to improving safety, quality, and cost-effectiveness in health care. Rob Schluth, ECRI’s Principal Project Officer for Device Safety, mentions during a webinar that this year’s #1 hazard isn’t even a medical device. In fact, ECRI intentionally tested only commonly available chatbots based on Large Language Models—such as ChatGPT, Claude, and Grok—used by patients as well as health care professionals. ECRI excluded chatbots designed as health applications, such as ChatGPT Health and Open Evidence, which is built on traditional medical evidence sources, such as The New England Journal of Medicine and JAMA, and is only accessible to physicians. 

AI chatbots are changing how we search for information and answers across all fields, including health care. In the same informational webinar, ECRI President and CEO Marcus Schabacker, M.D., Ph.D., notes that the use of LLM chatbots has become normalized, which poses a particular risk in health care. “When something seems helpful and definitive, people begin to rely on it without question…That’s the problem,” says Dr. Schabacker. ECRI is not claiming that AI chatbots are inherently dangerous, but it identifies potential risks arising from how they are understood and used. 

To mitigate the risk of harm, ECRI advises caution: recognize the limitations of AI chatbots, use them as educational resources more than decision-making aids, and keep a human “in the loop” by confirming important information with trusted sources and qualified professionals. 

ECRI’s annual Top 10 Health Technology Hazards list is designed to help health care organizations identify, prioritize and mitigate risks. Through expert, multidisciplinary analysis of safety reports and product evaluations, ECRI identifies hazards it deems most urgent, not necessarily those known to cause the most frequent or severe incidents.