We have remained at the forefront of medicine by fostering a culture of collaboration, pushing the boundaries of medical research, educating the brightest medical minds and maintaining an unwavering commitment to the diverse communities we serve.
We offer diagnostic and treatment options for common and complex medical conditions.
Search for condition information or for a specific treatment program.
We are committed to providing expert care—safely and effectively. Let us help you navigate your in-person or virtual visit to Mass General.
At Mass General, the brightest minds in medicine collaborate on behalf of our patients to bridge innovation science with state-of-the-art clinical medicine.
NewsMay | 17 | 2023
Braden Kuo, MD, a neurogastroenterologist and the director of the Center for Neurointestinal Health at MGH and an associate professor of Medicine at Harvard Medical School and Tsung-Chun Lee, MD, PhD, of Taipei Medical University Shuang Ho Hospital, in Taiwan are co-authors of a recent research letter published in Gastroenterology, ChatGPT Answers Common Patient Questions About Colonoscopy.
ChatGPT, a new language processing tool driven by artificial intelligence (AI), provides conversational text responses to questions and can generate valuable information for enquiring individuals, but the quality of ChatGPT-generated answers to medical questions is currently unclear.
We retrieved eight common questions and answers about colonoscopy from the publicly available webpages of three randomly-selected hospitals from the top-20 list of the US News & World Report Best Hospitals for Gastroenterology and Gastrointestinal Surgery.
We inputted these questions as prompts for ChatGPT for two times on the same day and recorded the ChatGPT-generated answers.
We then used a plagiarism detection software to compare the text similarity among all answers. Finally, to objectively interpret the quality of ChatGPT-generated answers, four gastroenterologists rated 36 random pairs of questions and answers for the following quality indicators on a 7-point scale:
Raters were also asked to interpret whether the answers were AI-generated or not.
ChatGPT answers had extremely low text similarity compared with answers on hospital webpages, while the text similarity ranged from 28% to 77% between the two ChatGPT answers.
ChatGPT answers were rated similarly by gastroenterologists to non-AI answers in terms of ease of understanding, but with the average AI scores higher than non-AI scores. Scores were also similar related to scientific adequacy and satisfaction with the answers. The raters were only 48% accurate in telling which answers were provided by ChatGPT.
This study is the first of its kind to demonstrate that a contemporary large language model–derived conversational AI program is able to provide easy to understand, scientifically adequate, and generally satisfactory answers to common questions about colonoscopy, as determined by gastroenterologists.
Such programs may help to optimize clinical communication to patients, especially for high volume procedures like colonoscopy. Conversational AI empowered by large language models like ChatGPT has the potential to transform and benefit shared decision-making by patients and physicians.
Future research should explore responses to a broader sample of patient questions and clinical conditions and include both patients and physicians as raters.
Lee, T. C., Staller, K., Botoman, V., Pathipati, M. P., Varma, S., & Kuo, B. (2023). ChatGPT Answers Common Patient Questions About Colonoscopy. Gastroenterology, S0016-5085(23)00704-7. Advance online publication. https://doi.org/10.1053/j.gastro.2023.04.033
Massachusetts General Hospital, founded in 1811, is the original and largest teaching hospital of Harvard Medical School. The Mass General Research Institute conducts the largest hospital-based research program in the nation, with annual research operations of more than $1 billion and comprises more than 9,500 researchers working across more than 30 institutes, centers and departments. In July 2022, Mass General was named #8 in the U.S. News & World Report list of “America’s Best Hospitals.” MGH is a founding member of the Mass General Brigham healthcare system.
Bench Press highlights the groundbreaking research and boundary-pushing scientists working to improve human health and fight disease.
Your gift helps fund groundbreaking research aimed at understanding, treating and preventing human disease.
Explore fellowships, residencies, internships and other educational opportunities.
Learn about career opportunities, search for positions and apply for a job.
Get the latest news, explore events and connect with Mass General.
Learn about the many ways you can get involved and support Mass General.
Copyright © 2007-2023. The General Hospital Corporation.
Leave a Reply
You must be logged in to post a comment.