European Respiratory Society

ERS 2024: Google’s chatbot edged out trainee doctors in assessing complex pediatric respiratory illnesses

September 11, 2024

ChatGPT, as well as Google’s chatbot, Bard, were better able to assess complex cases of respiratory illness like cystic fibrosis and chest infections in children when compared to trainee doctors, according to research presented at the recent European Respiratory Society meeting in Vienna, Austria. Findings suggests that large language models could be used to support trainee doctors, nurses, and general practitioners to triage patients more quickly and ease pressure on health services.

• Researchers presented clinical scenarios involving conditions like cystic fibrosis, asthma, and sleep disordered breathing to ten pediatric trainees with less than four months of experience. The trainees had one hour to research solutions online without using chatbots. The same scenarios were also given to three chatbots. Six pediatric respiratory experts evaluated all responses for accuracy, thoroughness, practicality, believability, and clarity. They also guessed whether each response was human- or chatbot-generated and rated them on a scale of one to nine.
• ChatGPT version 3.5 received an average score of 7 out of 9 and mimicked human responses better than other chatbots. Google’s Bard earned an average of 6 out of 9 and was found to be more coherent than trainee doctors. Microsoft’s Bing, along with the trainee doctors, scored an average of 4 out of 9.
• Experts were consistently able to identify responses from Bing and Bard as non-human.

Source:

(2024, September 9). European Respiratory Society. ChatGPT outperformed trainee doctors in assessing complex respiratory illness in children. [News release]. https://www.ersnet.org/news-and-features/news/chatgpt-outperformed-trainee-doctors-in-assessing-complex-respiratory-illness-in-children/