Assessing the Accuracy of Diagnostic Capabilities of Large Language Models

Assessing the Accuracy of Diagnostic Capabilities of Large Language Models

<b>Background:</b> In recent years, numerous artificial intelligence applications, especially generative large language models, have evolved in the medical field. This study conducted a structured comparative analysis of four leading generative large language models (LLMs)—ChatGPT-4o (Op...

Full description

Saved in:
Bibliographic Details
Main Authors: Andrada Elena Urda-Cîmpean, Daniel-Corneliu Leucuța, Cristina Drugan, Alina-Gabriela Duțu, Tudor Călinici, Tudor Drugan
Format: Article
Language:English
Published: MDPI AG 2025-06-01
Series:Diagnostics
Subjects:
diagnostic accuracy
large language modes
artificial intelligence
medical education
Online Access:https://www.mdpi.com/2075-4418/15/13/1657
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:<b>Background:</b> In recent years, numerous artificial intelligence applications, especially generative large language models, have evolved in the medical field. This study conducted a structured comparative analysis of four leading generative large language models (LLMs)—ChatGPT-4o (OpenAI), Grok-3 (xAI), Gemini-2.0 Flash (Google), and DeepSeek-V3 (DeepSeek)—to evaluate their diagnostic performance in clinical case scenarios. <b>Methods:</b> We assessed medical knowledge recall and clinical reasoning capabilities through staged, progressively complex cases, with responses graded by expert raters using a 0–5 scale. <b>Results:</b> All models performed better on knowledge-based questions than on reasoning tasks, highlighting the ongoing limitations in contextual diagnostic synthesis. Overall, DeepSeek outperformed the other models, achieving significantly higher scores across all evaluation dimensions (<i>p</i> < 0.05), particularly in regards to medical reasoning tasks. <b>Conclusions:</b> While these findings support the feasibility of using LLMs for medical training and decision support, the study emphasizes the need for improved interpretability, prompt optimization, and rigorous benchmarking to ensure clinical reliability. This structured, comparative approach contributes to ongoing efforts to establish standardized evaluation frameworks for integrating LLMs into diagnostic workflows.
ISSN:2075-4418

Similar Items