JAMA Netw Open
AI-generated replies to patient queries receive higher satisfaction scores
This study is the first to evaluate satisfaction with AI-generated responses to patient medical questions in electronic health records (EHRs), finding higher satisfaction with AI responses compared with clinicians, though satisfaction didn't always align with clinician-rated information quality and empathy, and highlighting the need for patient involvement in AI development for optimal integration into practice.
In this cross-sectional study, researchers screened 1,089 clinical questions from 3,769,023 Patient Medical Advice Requests in EHRs, ultimately including 59 messages. Two generative AIs (ChatGPT-4 and Stanford GPT) generated responses with and without prompt engineering. Six licensed clinicians evaluated these AI and original clinician responses for information quality and empathy using a 5-point Likert scale. Additionally, 30 survey participants assessed satisfaction with AI and clinician responses. A total of 2,118 assessments were analyzed for AI response quality and 408 for satisfaction.
Findings
- Satisfaction scores were higher for AI responses (3.96) vs. clinicians’ responses (3.05), both overall (P < 0.001) and by specialty.
- AI responses to cardiology questions had the highest satisfaction (4.09), while endocrinology questions scored highest in information quality and empathy.
- Clinicians’ responses were shorter (mean 254.37 characters) than AI responses (mean 1,470.77 characters).
- Satisfaction was linked to the length of clinicians’ responses overall and in cardiovascular questions, but AI response length wasn't.
Source:
Kim J, et al. (2024, October 16). JAMA Netw Open. Perspectives on Artificial Intelligence–Generated Responses to Patient Messages. https://pubmed.ncbi.nlm.nih.gov/39412810/