Session: Abstracts: Imaging of Rheumatic Diseases (0745–0750)
0745: Artificial Intelligence Models for Computer-Assisted Joint Detection and Sharp-van Der Heijde Score Prediction in Hand Radiographs from Patients with Rheumatoid Arthritis
Carol Hitchon1, Saqib Al Islam1, Daryl LX Fung1, Qian Liu1, Leann Lac1, Susan Bartlett2, Louis Bessette3, Gilles Boire4, Vivian Bykerk5, Glen Hazlewood6, Edward Keystone7, Janet Pope8, Orit Schieir9, Carter Thorne10, Diane Tin11, Marie-France Valois12, Désirée van der Heijde13, Canadian Early Arthritis Cohort (CATCH) Investigators14, Liam O'Neil1 and Pingzhao Hu15, 1University of Manitoba, Winnipeg, MB, Canada, 2McGill University, Montreal, QC, Canada, 3Centre de l'Ostéoporose et de Rhumatologie de Québec, Quebec City, QC, Canada, 4Université de Sherbrooke, Sherbrooke, QC, Canada, 5Hospital for Special Surgery, New York, NY, 6University of Calgary, Calgary, AB, Canada, 7Keystone Consulting Enterprises Inc., Toronto, ON, Canada, 8University of Western Ontario, London, ON, Canada, 9McGill University, Montréal, QC, Canada, 10Southlake Regional Health Centre, Newmarket, ON, Canada, 11The Arthritis Program Research Group, Newmarket, ON, Canada, 12McGill University, Pointe-Claire, QC, Canada, 13Department of Rheumatology, Leiden University Medical Center, Leiden, Netherlands, 14CATCH, Winnipeg, MB, Canada, 15Western University, London, ON, Canada
Background/Purpose: Radiographs are used to detect and monitor joint damage due to rheumatoid arthritis (RA).Scoring methods such as the Sharp van der Heijde tool (SVH) quantify joint space narrowing (JSN) and erosions. Obtaining SVH scores is time-consuming and requires expertise not always available. We aimed to develop and validate a deep learning system for automated detection and prediction of SVH scores in hand radiographs of patients with RA.
Methods: We used a convolutional neural network (CNN) based algorithm (You Only Look Once (YOLO)v5l6) trained on an object detection model (COCO-Common Objects in Context) to detect joints in 240 training and 89 test pediatric hand radiographs from the Radiologic Society of North America database. Radiographs were annotated by boxing and labeling the joints of interest: proximal interphalangeal, metacarpophalangeal, wrist, distal radius, distal ulna. Images were augmented using mix-up (mixing features of different images into one), mosaic (combining 4 training images into 1 image), rotation, translation, scaling, and shearing. The joint detection model was validated with 54 clinician-annotated radiographs from 4 adult RA patients followed for 9-13 years (10-12 images per patient) (joint detection gold standard). We applied a supervised vision transformer model (VTM) to predict each joint's SVH erosion and JSN score. The VTM was validated using 2249 hand radiographs with clinician-assigned SVH scores from 381 RA patients from the Canadian Early Arthritis Cohort (SVH score gold standard). We applied techniques used to train highly class-imbalanced datasets including weighted sampling, stratified data-split, data augmentation and transfer learning (TL). As the joint detection model was trained to detect the whole wrist and we had clinician-assigned SVH scores for individual wrist joints, we trained a separate multi-task model to predict wrist joint scores from whole wrist images. The performance of the VTM to predict joint scores was compared to CNN-based EfficientNetV2 and MobileNetV3. Model accuracy for joint detection is reported as the F1-score (reflecting model precision and model recall) and mean absolute precision (mAP) for a range of Intersection-over-Union (IoU) measures reflecting the overlap between clinician and model assigned bounding boxes of detected joints. Accuracy for SVH score prediction is reported as root mean squared error (RMSE) and balanced accuracy.
Results: The joint detection model accurately identified target joints (pediatric data FI-score = 0.991, map0.1 = 0.993 with an IoU threshold of 0.1 ; adult data F1-score = 0.812, map0.1 = 0.871 (n=54) (Table 1). Applying TL improved the joint detection model’s mean precision by 0.05 over the COCO model. The VTM predicted JSN and erosion SVH scores with high accuracy (RMSE JSN 0.91, erosion 0.93). The multi-task models predicted SVH erosion and JSN scores of wrist joints with moderately high accuracy (0.6-0.91). EfficientNetV3 performed better for wrist joints (VTM vs EfficientNetV3 average difference 0.10) (Figure 1).
Conclusion: Automated deep learning systems accurately identify and predict joint damage in hand radiographs from patients with rheumatoid arthritis and may aid in monitoring joint damage.
Table: Accuracy of model for joint detection
Figure: Joint damage score prediction models. A) Models trained on all joints except wrist. The metrics reported were balanced accuracy, F1-Score and Root Mean Square Error (RMSE) predicting Joint Space Narrowing (JSN) and Erosion scores. B) Models were trained on wrist joints. Separate models trained to detect the erosion score and JSN scores. PIP=proximal interphalangeal, MCP=metacarpophalangeal, JSN= joint space narrowing, Vit= Vision Transformer, mc = metacarpal, nav = navicular, cmc = carpometacarpal
C. Hitchon: Astra Zeneca, 1, Pfizer, 5; S. Al Islam: None; D. Fung: None; Q. Liu: None; L. Lac: None; S. Bartlett: Janssen, 6, Merck/MSD, 2, 6, Novartis, 2, Organon, 1, 6, PROMIS Health Organization, 4, Sandoz, 2, 6; L. Bessette: AbbVie, 2, 5, 6, Amgen, 2, 5, 6, Bristol Myers Squibb, 2, 5, 6, Eli Lilly, 2, 5, 6, Fresenius Kabi, 2, 6, Gilead, 2, 5, 6, JAMP Pharma, 2, 5, 6, Janssen, 2, 5, 6, Novartis, 2, 5, 6, Organon, 2, 6, Pfizer, 2, 5, 6, Sandoz, 2, 6, Sanofi, 2, 5, 6, Teva, 2, 6, UCB, 2, 5, 6, UCBA, 5; G. Boire: Eli Lilly, 1, Janssen, 6, Organon, 1, Orimed Pharma, 1, 6, Otsuka, 1, Pfizer, 1, 5, Sandoz, 1, Teva, 1, Viatris, 1, 6; V. Bykerk: Abbvie, 2, BMS, 2, Pfizer, 2; G. Hazlewood: None; E. Keystone: AbbVie/Abbott, 2, 6, Amgen, 2, 6, celltrion, 2, 6, Eli Lilly, 2, 6, Fresenius Kabi, 2, 6, Pfizer, 2, 6, Samsung Bioepsis, 2, sandoz, 2, 6; J. Pope: AbbVie, 1, 2; O. Schieir: None; C. Thorne: Abbvie, 1, Biogen, 2, Nordic Pharma, 1, Pfizer, 1, 5, Roche, 1, Sandoz, 1, 2; D. Tin: None; M. Valois: None; D. van der Heijde: AbbVie, 2, Bayer, 2, BMS, 2, Eli Lilly, 2, Galapagos, 2, Gilead, 2, GSK, 2, Imaging Rheumatology BV, 12, Director, Janssen, 2, Novartis, 2, Pfizer, 2, Takeda, 2, UCB Pharma, 2; C. (CATCH) Investigators: AbbVie Corporation, 5, Amgen Canada, 5, Hoffman La Roche Limited, 5, Medexus Pharmaceuticals, 5, Organon Canada, 5, Pfizer Canada, 5, Sandoz Biopharmaceuticals Canada, 5; L. O'Neil: None; P. Hu: None.