Samuel (Sandy) Aronson, ALM, MA, executive director of IT and AI Solutions for Mass General Brigham Personalized Medicine and senior director of IT and AI Solutions for the Accelerator for Clinical Transformation, is the corresponding author of a paper published in NEJM AI that looked at whether generative AI could hold promise for improving scientific literature review of variants in clinical genetic testing. Their findings could have a wide impact beyond this use case. How would you summarize your study for a lay audience? We tested whether generative AI can be used to identify whether scientific articles contain information that can help geneticists determine whether genetic variants are harmful to patients.
While testing this work, we identified inconsistencies in generative AI that could present a risk for patients if not adequately addressed. We suggest forms of testing and monitoring that could improve safety. What question were you investigating? We investigated whether generative AI can be used to determine: 1) whether a scientific article contains evidence about a variant that could help a geneticist's assessment of a genetic variant and 2) whether any evidence found about the variant supports a benign, pathogenic, intermediate or inconclusive conclusion.
What methods or approach did you use? We tested a generative AI strategy based on GPT-4 using a labeled dataset of 72 articles and compared generative AI to assessments from expert geneticists. What did you find? G.