Performance Evaluation of Large Language Model Chatbots for Radiation Therapy Education
This study aimed to develop a large language model (LLM) chatbot for radiation therapy education and compare the performance of portable document format (PDF)- and webpage-based question-and-answer (Q&A) chatbots. An LLM chatbot was created using the EmbedChain framework, OpenAI GPT-3.5-Turbo AP...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-06-01
|
| Series: | Information |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2078-2489/16/7/521 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1839615879120158720 |
|---|---|
| author | Jae-Hong Jung Daegun Kim Kyung-Bae Lee Youngjin Lee |
| author_facet | Jae-Hong Jung Daegun Kim Kyung-Bae Lee Youngjin Lee |
| author_sort | Jae-Hong Jung |
| collection | DOAJ |
| description | This study aimed to develop a large language model (LLM) chatbot for radiation therapy education and compare the performance of portable document format (PDF)- and webpage-based question-and-answer (Q&A) chatbots. An LLM chatbot was created using the EmbedChain framework, OpenAI GPT-3.5-Turbo API, and Gradio UI. The performance of both chatbots was evaluated based on 10 questions and their corresponding answers, using the parameters of accuracy, semantic similarity, consistency, and response time. The accuracy scores were 0.672 and 0.675 for the PDF- and webpage-based Q&A chatbots, respectively. The semantic similarity between the two chatbots was 0.928 (92.8%). The consistency score was one for both chatbots. The average response time was 3.3 s and 2.38 s for the PDF- and webpage-based chatbots, respectively. The LLM chatbot developed in this study demonstrates the potential to provide reliable responses for radiation therapy education. However, its reliability and efficiency must be further optimized to be effectively utilized as an educational tool. |
| format | Article |
| id | doaj-art-0a0c8c2f5933468e9cd16bf6c2060d65 |
| institution | Matheson Library |
| issn | 2078-2489 |
| language | English |
| publishDate | 2025-06-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Information |
| spelling | doaj-art-0a0c8c2f5933468e9cd16bf6c2060d652025-07-25T13:25:00ZengMDPI AGInformation2078-24892025-06-0116752110.3390/info16070521Performance Evaluation of Large Language Model Chatbots for Radiation Therapy EducationJae-Hong Jung0Daegun Kim1Kyung-Bae Lee2Youngjin Lee3Department of Radiation Oncology, College of Medicine, Soonchunhyang University Hospital Bucheon, 170, Jomaru-ro, Wonmi-gu, Bucheon-si 14584, Republic of KoreaDepartment of Radiation Oncology, College of Medicine, Soonchunhyang University Hospital Bucheon, 170, Jomaru-ro, Wonmi-gu, Bucheon-si 14584, Republic of KoreaDepartment of Radiotechnology, Wonkwang Health Science University, 514, Iksan-daero, Iksan-si 54538, Republic of KoreaDepartment of Radiological Science, Gachon University, 191, Hambakmoe-ro, Yeonsu-gu, Incheon 21936, Republic of KoreaThis study aimed to develop a large language model (LLM) chatbot for radiation therapy education and compare the performance of portable document format (PDF)- and webpage-based question-and-answer (Q&A) chatbots. An LLM chatbot was created using the EmbedChain framework, OpenAI GPT-3.5-Turbo API, and Gradio UI. The performance of both chatbots was evaluated based on 10 questions and their corresponding answers, using the parameters of accuracy, semantic similarity, consistency, and response time. The accuracy scores were 0.672 and 0.675 for the PDF- and webpage-based Q&A chatbots, respectively. The semantic similarity between the two chatbots was 0.928 (92.8%). The consistency score was one for both chatbots. The average response time was 3.3 s and 2.38 s for the PDF- and webpage-based chatbots, respectively. The LLM chatbot developed in this study demonstrates the potential to provide reliable responses for radiation therapy education. However, its reliability and efficiency must be further optimized to be effectively utilized as an educational tool.https://www.mdpi.com/2078-2489/16/7/521radiation therapy educationprofessional educationlarge language modelchatbot |
| spellingShingle | Jae-Hong Jung Daegun Kim Kyung-Bae Lee Youngjin Lee Performance Evaluation of Large Language Model Chatbots for Radiation Therapy Education Information radiation therapy education professional education large language model chatbot |
| title | Performance Evaluation of Large Language Model Chatbots for Radiation Therapy Education |
| title_full | Performance Evaluation of Large Language Model Chatbots for Radiation Therapy Education |
| title_fullStr | Performance Evaluation of Large Language Model Chatbots for Radiation Therapy Education |
| title_full_unstemmed | Performance Evaluation of Large Language Model Chatbots for Radiation Therapy Education |
| title_short | Performance Evaluation of Large Language Model Chatbots for Radiation Therapy Education |
| title_sort | performance evaluation of large language model chatbots for radiation therapy education |
| topic | radiation therapy education professional education large language model chatbot |
| url | https://www.mdpi.com/2078-2489/16/7/521 |
| work_keys_str_mv | AT jaehongjung performanceevaluationoflargelanguagemodelchatbotsforradiationtherapyeducation AT daegunkim performanceevaluationoflargelanguagemodelchatbotsforradiationtherapyeducation AT kyungbaelee performanceevaluationoflargelanguagemodelchatbotsforradiationtherapyeducation AT youngjinlee performanceevaluationoflargelanguagemodelchatbotsforradiationtherapyeducation |