Artificial Intelligence Model Aces US Medical Licensing Exam
TEHRAN (Tasnim) – An artificial intelligence model, developed by OpenAI, has shown impressive results by almost passing the United States Medical Licensing Exam (USMLE).
The USMLE is a crucial step in obtaining a medical license in the country and is known for its rigorous testing standards.
The AI model, known as "ChatGPT," was trained on a vast dataset of medical information and was able to demonstrate an understanding of complex medical concepts and procedures, Sputnik reported.
The research was published in the prestigious peer-reviewed scientific journal PLOS Digital Health.
As the authors of the study noted: "We evaluate(d) the performance of ChatGPT, a non-domain specific LLM, on its ability to perform clinical reasoning by testing its performance on questions from the United States Medical Licensing Examination (USMLE)."
The researchers indicated that ChatGPT passed the medical exam without specialized training. This means that the neural network was not trained on special medical papers, but simply used the information available on the internet. In doing so, researchers made sure that the answers could not simply be googled online. Artificial intelligence not only provided coherent responses but also scientifically grounded them.
ChatGPT scored between 52 and 75 percent across the three tests, while the pass mark is commonly 60 percent. In 88.9% of the responses, artificial intelligence gave answers that were perceived by the researchers as "new, non-obvious, and clinically valid". In other words, the neural network managed to produce medical insight.
The researchers added that the capabilities of the neural network should not be overestimated - it constructs plausible-sounding sentences based on the analyzed material, which means that ChatGPT can generate both extremely non-trivial ideas and complete nonsense.
The article highlighted that ChatGPT was much more effective in its responses than PubMedGPT, a bot specifically trained in medical literature.
Scholars believe that artificial intelligence will not be able to replace doctors in a foreseeable future. However, it may play an important role not only in training future medical students but also as an assistant to a real doctor, leading them to think about non-obvious diagnoses.