2022 Annual Meeting and Alumni Reunion
Category : Clinical Research Candidate : Cris Jacoba Poster #: C6
Automated Machine Learning Models for Diabetic Retinopathy Screening Using Handheld Fundus Cameras in a Low-resource Community Screening Program
Cris Martin P . Jacoba , Duy Doan , Lizzie A . Aquino , Joseph Paolo Y . Silva , Claude M . Salva , Recivall P . Salongccay , Glenn P . Alog , Kexin Zhang , Kaye Locaylocay , Aileen V . Saunar , Jennifer K . Sun , Tunde Peto , Lloyd P . Aiello , Paolo S . Silva
Purpose : Evaluating diabetic retinopathy ( DR ) severity from retinal images is one of the most common uses of artificial intelligence ( AI ) in medicine . However , commercially available algorithms are expensive , not universally generalizable , and are limited to specific retinal cameras . Automated machine learning ( AutoML ) allows the development of code-free models that decrease the barrier to using AI for clinical needs . This is useful in low resource settings where the financial incentive of commercial ventures is low , but the clinical need is high . This study used Google AutoML Vision to train models based on locally sourced data from community programs in the Philippines to detect referable DR .
Methods : AutoML Vision ( Google Cloud ) models were generated based on previously acquired 5-field ( macula centered , disc centered , superior , inferior , temporal macula ) retinal images ( 17,237 images , 3447 eyes ) from the Philippine DR screening program ( DRSP ). Each individual image was labelled based on the International DR and diabetic macular edema ( DME ) classification and performed by four certified graders from a centralized reading center ( RC ), with secondary adjudication done by a senior retina specialist . Images for the initial model were split 8-1-1 for training , optimization and testing to detect referable DR [( refDR ), defined as moderate nonproliferative DR or worse or any level of DME ]. The AutoML platform provided model evaluation metrics by showing the area under the precision-recall curve ( AUPRC ), and the confusion matrix . Internal validation of the autoML model was performed using a holdout image set that used a 2-field imaging protocol ( disc and macula-centered , 225 eyes ) using the same device in the same population , evaluated by the same RC . External validation was performed using a high-quality published image set ( macula-centered , 207 eyes ) using tabletop retinal cameras , which showed a different patient population . This dataset is publicly available from the Eye Picture Archive Communication System ( EyePACS ) Kaggle , USA . Sensitivity , specificity , positive predictive value , negative predictive ( SN , SP , PPV , NPV ), accuracy and F1 score for refDR were calculated .
Results : In the training set , refDR was present in 17.3 %, non-refDR 82.7 %. The model ’ s AUPRC was 0.995 with a precision and recall of 97 % using a score threshold of 0.5 . In the internal validation set , refDR was present in 39.1 %, non-refDR 60.9 %. The SN , SP , PPV , NPV , accuracy and F1 scores were 0.96 ( 95 % CI : 0.88-0.99 ), 0.98 ( 95 % CI : 0.94-0.99 ), 0.96 ( 95 % CI : 0.88-0.99 ), 0.98 ( 95 % CI : 0.94-0.99 ), 0.97 and 0.96 , respectively . Some false positive eyes had other pathologies that warranted referral . In the external validation set , refDR was present in 42.5 %, non-refDR 57.5 %. The model performed well with the high-quality external validation set , with SN , SP , PPV , NPV , accuracy and F1 scores at 1.0 .
Conclusions : This study demonstrates the accuracy and feasibility of low-cost autoML models for identifying refDR developed for a DRSP using handheld retinal imaging in a low-resource setting community program . The performance approaches and potentially outperforms published diagnostic accuracy metrics of commercial models used for DRSP . These data emphasize the use of local data in the development and optimization of machine learning models to potentially improve performance in the populations that they will be used . Furthermore , the use of autoML may increase access to machine learning models adapted for specific programs that are guided by clinicians to rapidly address disparities in patient care .