ChatGPT misses emergencies when used for medical advice

A new study has revealed that the AI chatbot ChatGPT can miss “high-risk emergencies” when it is employed for medical purposes.

ChatGPT misses “high-risk emergencies” when it is used for medical advice.

Health questions are one of the most common uses for OpenAI’s chatbot and the company introduced the ChatGPT Health tool earlier this year, but a new study has found that the system could miss emergencies and can’t be relied upon to safely tell somebody that they need urgent medical care.

The need to check whether the AI tool was safe led to a fast-tracked study from the Icahn School of Medicine at Mount Sinai.

The research emerged from a recognition that ChatGPT was possibly being relied upon for life and death situations, despite there being limited analysis into whether it actually works.

Lead author and urologist Ashwin Ramaswamy said: “We wanted to answer a very basic but critical question: if someone is experiencing a real medical emergency and turns to ChatGPT Health for help, will it clearly tell them to go to the emergency room?”

The experts concluded that it did not, at least in enough cases to lead them to question how reliable it is.

For the study, doctors came up with 60 scenarios that covered 21 medical specialties. These ranged from low-risk situations that would only need at-home care to genuine medical emergencies.

It was discovered that ChatGPT generally handled clear emergencies correctly, but was not sufficiently concerned in over half the cases when doctors decided that the person needed emergency care.

The boffins said that the tool was good for “textbook emergencies” but was less proficient at spotting situations where the danger might be less obvious.

Harvard Medical School’s Isaac S Kohane, who was not involved in the research, said: “LLMs have become patients’ first stop for medical advice – but in 2026 they are least safe at the clinical extremes, where judgment separates missed emergencies from needless alarm.

“When millions of people are using an AI system to decide whether they need emergency care, the stakes are extraordinarily high. Independent evaluation should be routine, not optional.”