The AI will see you now as ChatGPT scores doctor’s exam pass

Share
ChatGPT was able to achieve passing scores for major medical exam, which usually requires years of training, laying groundwork for AI doctors and surgeons

OpenAI’s ChatGPT has met the 60% passing threshold for the United States Medical Licensing Exam taken by all medical students and physicians-in-training, offering a glimpse of the bot’s potential to work in medical education and clinical practice.

The chatbot provided responses that make coherent, internal sense and contain frequent insights, according to a study published yesterday in the open-access journal PLOS Digital Health by Tiffany Kung, Victor Tseng, and colleagues at AnsibleHealth.

Kung and her team tested the performance of ChatGPT on the USMLE, a set of three highly standardised and regulated exams required to practice medicine in the United States. The USMLE, which is taken by medical students and physicians-in-training, assesses knowledge across a range of medical disciplines, including biochemistry, diagnostic reasoning, and bioethics.

After removing image-based questions, the researchers tested ChatGPT on 350 of the 376 publicly available questions from the June 2022 USMLE release. After eliminating indeterminate responses, ChatGPT scored between 52.4% and 75.0% on the three USMLE exams, with the passing threshold each year being around 60%.

ChatGPT also showed 94.6% agreement in its responses and produced at least one new, non-obvious and clinically valid insight for 88.9% of its answers. It is worth mentioning that ChatGPT outperformed PubMedGPT's model, which was trained exclusively on biomedical domain literature and scored 50.8% on an older dataset of USMLE-style questions.

ChatGPT's future as clinical advisor

Despite the limited scope of analysis due to the small input size, the authors believe their findings offer a glimpse of ChatGPT's potential to improve medical education and, eventually, clinical practice. For instance, clinicians at AnsibleHealth already use ChatGPT to reword jargon-heavy reports for better patient understanding.

The authors say reaching the passing score for this notoriously tricky expert exam, and doing so without any human reinforcement, marks a notable milestone in clinical AI.

"ChatGPT contributed substantially to the writing of [our] manuscript, says Kung. “We interacted with ChatGPT much like a colleague, asking it to synthesise, simplify, and offer counterpoints to drafts in progress. All of the co-authors valued ChatGPT's input."

Share

Featured Articles

From $500m to $9bn: Charting Perplexity’s Soaring Valuation

Gen AI challenger is being tipped to receive US$500m in funding, pushing its valuation from US$500m to around US$9bn in just under a year

Workforce Bellwether? 1/4 of Google’s Code Generated by AI

CEO Sundar Pichai revealed a striking statistic at a recent earnings report: a quarter of Google's code is generated by AI before engineers validate it

Nokia Bell Labs & Cambridge Future Tech Bring AI to Health

Nokia Bell Labs partners with Cambridge Future Tech are partnering to bring OmniBuds, an AI-powered earbud system for monitoring vital health to market

Meta AI: The Social Media AI Tool That Reached 500m Users

AI Strategy

How Mondra's Azure-Powered AI Chatbot Helps Reduce Emissions

AI Applications

Why Toyota See NTT Solutions as Key to Making Cars Safer

AI Applications