TY - JOUR
T1 - Non-technical Skills for Urology Trainees: A Double-Blinded Study of ChatGPT4 AI Benchmarking Against Consultant Interaction
AU - Pears, Matthew
AU - Wadhwa, Karan
AU - Payne, Stephen R.
AU - Hanchanale, Vishwanath
AU - Elmamoun, Mamoun Hamid
AU - Jain, Sunjay
AU - Konstantinidis, Stathis Th.
AU - Rochester, Mark
AU - Doherty, Ruth
AU - Spearpoint, Kenneth
AU - Ng, Oliver
AU - Dick, Lachlan
AU - Yule, Steven
AU - Biyani, Chandra Shekhar
N1 - © 2024 Springer Nature.
PY - 2024/11/14
Y1 - 2024/11/14
N2 - Non-technical skills (NTS) are crucial in healthcare, encompassing cognitive and social skills that support technical ability. Traditional NTS training is evolving with the emergence of artificial intelligence (AI) models that can intelligently converse with their users, known as large language models (LLMs). This study investigated the capabilities and limitations of a popular model named generative pre-trained transformer 4 (GPT-4) in NTS training, comparing its performance to that of human evaluators. Urology trainees identified NTS events in simulated scenarios and discussed them in blinded feedback sessions with AI and human consultants. Experts assessed the blinded interaction data, providing quantitative ratings and qualitative evaluations using annotated transcripts. Wilcoxon signed-rank tests compared pre- and post-intervention ratings, whilst Mann–Whitney U tests compared post-intervention ratings between AI and human feedback. Thematic analysis identified strengths, limitations, and differences between AI and human feedback approaches. The AI model demonstrated significant strengths in reinforcing knowledge gathering (p = 0.04), providing accurate and evidence-based feedback (p = 0.013), conveying empathy (p = 0.021), and tailoring explanations to complexity (p = 0.002). However, human feedback excelled in language terminology (p = 0.003), complexity (p = 0.020), and fact-based feedback (p = 0.025). The study highlights the potential for AI to augment assessment of NTS training in healthcare. A blended approach utilising AI and human expertise may boost training efficacy.
AB - Non-technical skills (NTS) are crucial in healthcare, encompassing cognitive and social skills that support technical ability. Traditional NTS training is evolving with the emergence of artificial intelligence (AI) models that can intelligently converse with their users, known as large language models (LLMs). This study investigated the capabilities and limitations of a popular model named generative pre-trained transformer 4 (GPT-4) in NTS training, comparing its performance to that of human evaluators. Urology trainees identified NTS events in simulated scenarios and discussed them in blinded feedback sessions with AI and human consultants. Experts assessed the blinded interaction data, providing quantitative ratings and qualitative evaluations using annotated transcripts. Wilcoxon signed-rank tests compared pre- and post-intervention ratings, whilst Mann–Whitney U tests compared post-intervention ratings between AI and human feedback. Thematic analysis identified strengths, limitations, and differences between AI and human feedback approaches. The AI model demonstrated significant strengths in reinforcing knowledge gathering (p = 0.04), providing accurate and evidence-based feedback (p = 0.013), conveying empathy (p = 0.021), and tailoring explanations to complexity (p = 0.002). However, human feedback excelled in language terminology (p = 0.003), complexity (p = 0.020), and fact-based feedback (p = 0.025). The study highlights the potential for AI to augment assessment of NTS training in healthcare. A blended approach utilising AI and human expertise may boost training efficacy.
KW - Artificial intelligence
KW - Healthcare education
KW - Non-technical skills
KW - Pedagogy
KW - Simulation
KW - Urology
UR - http://www.scopus.com/inward/record.url?scp=85209079416&partnerID=8YFLogxK
U2 - 10.1007/s41666-024-00180-7
DO - 10.1007/s41666-024-00180-7
M3 - Article
SN - 2509-498X
JO - Journal of Healthcare Informatics Research
JF - Journal of Healthcare Informatics Research
ER -