Exploring the potential of ChatGPT as a supplementary tool for providing orthopaedic information
Artikel i vetenskaplig tidskrift, 2023

Purpose: To investigate the potential use of large language models (LLMs) in orthopaedics by presenting queries pertinent to anterior cruciate ligament (ACL) surgery to generative pre-trained transformer (ChatGPT, specifically using its GPT-4 model of March 14th 2023). Additionally, this study aimed to evaluate the depth of the LLM’s knowledge and investigate its adaptability to different user groups. It was hypothesized that the ChatGPT would be able to adapt to different target groups due to its strong language understanding and processing capabilities. Methods: ChatGPT was presented with 20 questions and response was requested for two distinct target audiences: patients and non-orthopaedic medical doctors. Two board-certified orthopaedic sports medicine surgeons and two expert orthopaedic sports medicine surgeons independently evaluated the responses generated by ChatGPT. Mean correctness, completeness, and adaptability to the target audiences (patients and non-orthopaedic medical doctors) were determined. A three-point response scale facilitated nuanced assessment. Results: ChatGPT exhibited fair accuracy, with average correctness scores of 1.69 and 1.66 (on a scale from 0, incorrect, 1, partially correct, to 2, correct) for patients and medical doctors, respectively. Three of the 20 questions (15.0%) were deemed incorrect by any of the four orthopaedic sports medicine surgeon assessors. Moreover, overall completeness was calculated to be 1.51 and 1.64 for patients and medical doctors, respectively, while overall adaptiveness was determined to be 1.75 and 1.73 for patients and doctors, respectively. Conclusion: Overall, ChatGPT was successful in generating correct responses in approximately 65% of the cases related to ACL surgery. The findings of this study imply that LLMs offer potential as a supplementary tool for acquiring orthopaedic knowledge. However, although ChatGPT can provide guidance and effectively adapt to diverse target audiences, it cannot supplant the expertise of orthopaedic sports medicine surgeons in diagnostic and treatment planning endeavours due to its limited understanding of orthopaedic domains and its potential for erroneous responses. Level of evidence: V.

ChatGPT

Large language models

Anterior cruciate ligament

Correctness

ACL

Artificial intelligence

Författare

Janina Kaarre

Göteborgs universitet

University of Pittsburgh

Robert Feldt

Chalmers, Data- och informationsteknik, Software Engineering

Laura E. Keeling

University of Pittsburgh

Sahil Dadoo

University of Pittsburgh

Bálint Zsidai

Göteborgs universitet

Jonathan D. Hughes

University of Pittsburgh

Kristian Samuelsson

Göteborgs universitet

Sahlgrenska universitetssjukhuset

Volker Musahl

University of Pittsburgh

Knee Surgery, Sports Traumatology, Arthroscopy

0942-2056 (ISSN) 1433-7347 (eISSN)

Vol. 31 11 5190-5198

Ämneskategorier

Kirurgi

DOI

10.1007/s00167-023-07529-2

Mer information

Senast uppdaterat

2024-03-07