0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Original Investigation |

Validity of a Telemedicine System for the Evaluation of Acute-Phase Retinopathy of Prematurity FREE

Graham E. Quinn, MD, MSCE1; Gui-shuang Ying, PhD2; Ebenezer Daniel, MBBS, MS, PhD2; P. Lloyd Hildebrand, MD3; Anna Ells, MD, FRCS4; Agnieshka Baumritter, MS1; Alex R. Kemper, MD, MPH, MS5; Eleanor B. Schron, PhD, RN6; Kelly Wade, MD, PhD, MSCE7 ; for the e-ROP Cooperative Group
[+] Author Affiliations
1Division of Pediatric Ophthalmology, The Children’s Hospital of Philadelphia, University of Pennsylvania Perelman School of Medicine, Philadelphia
2Department of Ophthalmology, Scheie Eye Institute, Perelman School of Medicine, University of Pennsylvania, Philadelphia
3Department of Ophthalmology, University of Oklahoma Health Sciences Center, Oklahoma City
4University of Calgary, Calgary, Alberta, Canada
5Program on Pediatric Health Services Research, Duke Clinical Research Institute, Duke University, Durham, North Carolina
6Division of Extramural Research, National Institutes of Health, Bethesda, Maryland
7Department of Pediatrics, Division of Neonatology, The Children’s Hospital of Philadelphia, University of Pennsylvania Perelman School of Medicine, Philadelphia
JAMA Ophthalmol. 2014;132(10):1178-1184. doi:10.1001/jamaophthalmol.2014.1604.
Text Size: A A A
Published online

Importance  The present strategy to identify infants needing treatment for retinopathy of prematurity (ROP) requires repeated examinations of at-risk infants by physicians. However, less than 10% ultimately require treatment. Retinal imaging by nonphysicians with remote image interpretation by nonphysicians may provide a more efficient strategy.

Objective  To evaluate the validity of a telemedicine system to identify infants who have sufficiently severe ROP to require evaluation by an ophthalmologist.

Design, Setting, and Participants  An observational study of premature infants starting at 32 weeks’ postmenstrual age was conducted. This study involved 1257 infants with birth weight less than 1251 g in neonatal intensive care units in 13 North American centers enrolled from May 25, 2011, through October 31, 2013.

Interventions  Infants underwent regularly scheduled diagnostic examinations by an ophthalmologist and digital imaging by nonphysician staff using a wide-field digital camera. Ophthalmologists documented findings consistent with referral-warranted (RW) ROP (ie, zone I ROP, stage 3 ROP or worse, or plus disease). A standard 6-image set per eye was sent to a central server and graded by 2 trained, masked, nonphysician readers. A reading supervisor adjudicated disagreements.

Main Outcomes and Measures  The validity of grading retinal image sets was based on the sensitivity and specificity for detecting RW-ROP compared with the criterion standard diagnostic examination.

Results  A total of 1257 infants (mean birth weight, 864 g; mean gestational age, 27 weeks) underwent a median of 3 sessions of examinations and imaging. Diagnostic examination identified characteristics of RW-ROP in 18.2% of eyes (19.4% of infants). Remote grading of images of an eye at a single session had sensitivity of 81.9% (95% CI, 77.4-85.6) and specificity of 90.1% (95% CI, 87.9-91.8). When both eyes were considered for the presence of RW-ROP, as would routinely be done in a screening, the sensitivity was 90.0% (95% CI, 85.4-93.5), with specificity of 87.0% (95% CI, 84.0-89.5), negative predictive value of 97.3%, and positive predictive value of 62.5% at the observed RW-ROP rate of 19.4%.

Conclusions and Relevance  When compared with the criterion standard diagnostic examination, these results provide strong support for the validity of remote evaluation by trained nonphysician readers of digital retinal images taken by trained nonphysician imagers from infants at risk for RW-ROP.

Trial Registration  clinicaltrials.gov Identifier:NCT01264276

Retinopathy of prematurity (ROP) is a leading cause of avoidable blindness in children worldwide1,2 and an increasing problem in underserved areas of the United States and Canada.3,4 In most settings, ophthalmologists travel regularly to neonatal units to perform serial eye examinations of infants at risk; however, less than 10% of infants examined require treatment.5 The interpretation of the ROP findings varies across examiners.6 A potential solution is to develop a telemedicine system using digital retinal imaging to detect sight-threatening disease. The solution must deal with key screening principles. Such evaluations differ fundamentally from clinical examinations by physicians and must provide compelling evidence for a high likelihood of altering the natural history of the disease in a significant proportion of individuals evaluated.79

In recent years, studies have evaluated the validity of retinal imaging to detect moderate to severe ROP.1021 However, the sensitivities varied widely (33%11 to 100%10,12,14,1821), as did the number of infants included (range, 10-122210,18) and the primary outcome (plus disease10 to suspect treatment requiring18). Images for these studies were largely obtained by an ophthalmologist and were also graded by ophthalmologists. Given the current status of retinal imaging in ROP, the American Academy of Ophthalmology prioritized the need for “understanding …the place of digital wide-angle photography in the evaluation of at-risk infants.”22

Ells et al12 introduced the term referral-warranted ROP (RW-ROP) in 2003 for use in telemedicine to describe eyes with ROP that had high-risk characteristics defined as plus disease, ROP in zone I, or stage 3 ROP or greater. Eyes with RW-ROP require careful ophthalmoscopic examination and many require treatment.

This large multicenter, National Eye Institute–funded clinical study evaluated the validity of an ROP telemedicine system to detect eyes with RW-ROP.23 We compared remote evaluations of digital images to the findings of a criterion-standard indirect ophthalmoscopic examination performed by experienced ophthalmologists.

Eligibility

Infants with birth weight (BW) less than 1251 g meeting current ROP screening guidelines in 12 US centers and 1 Canadian center were included in the study. Exclusion criteria were postmenstrual age greater than 39 weeks at first opportunity for imaging unless transferred in for ROP treatment, admission to a neonatal intensive care unit (NICU) with regressing or treated ROP, significant media opacity precluding visualization of the retina, or major ocular or systemic congenital abnormality. The protocol and informed consent processes were approved by the institutional review boards of the participating study centers. Written informed consent was obtained for all participants.

Procedures

Infants underwent serial ROP imaging in both eyes using the RetCam Shuttle (Clarity Medical Systems), in addition to a standard diagnostic examination by study-certified ophthalmologists experienced in diagnosing ROP. The diagnostic examination results were classified as having clinical findings consistent with RW-ROP: zone I ROP, stage 3 ROP, or plus disease. The imagers were masked to the results of the examination and the physicians were masked to images and subsequent grading. The timing of diagnostic examinations was determined by local clinical center criteria for usual clinical care and imaging sessions began at 32 weeks’ postmenstrual age. To prevent bias in terms of adverse events, the order of imaging and examinations alternated.

Imaging was conducted by 25 nonphysician study-certified personnel including NICU nurses (44%), neonatal nurse practitioners (24%), ophthalmic photographers (8%), an ocular coherence tomography technician (4%), an ophthalmic medical technologist (4%), and individuals with nonclinical backgrounds (16%). All participated in classroom and hands-on instruction in taking retinal images in infants and selecting and uploading images to the Inoveon ROP Data Center server in Oklahoma City, Oklahoma. Imagers were certified after submission of quality prestudy image sets and passing a knowledge assessment test.

Images were obtained for each eye using the video mode with a 130° wide-field imaging system. The imager selected still frames for a standard 6-image set consisting of the pupil and 5 retinal fields, with optic disc central, nasal, temporal, superior, and inferior.

Demographic data and the medical status of the infant before, during, and after each session were collected. Surveillance for ocular and systemic complications and other adverse events was conducted.

The paired diagnostic examinations and imaging sessions continued as clinically indicated until the examining ophthalmologist noted any of the following: mature retinal vessels, immature zone III on 2 occasions at least 7 days apart, ROP regressed or regressing on 2 occasions at least 7 days apart, treatment for severe ROP, or the infant reached 40 weeks’ postmenstrual age with no ROP or only stage 1 or 2 ROP.

Image Grading

Image sets were uploaded as unmodified uncompressed files in .png format from the RetCam Shuttle to the Inoveon ROP Data Center server and used for remote grading by nonphysician trained readers and by expert readers (ophthalmologists with ROP expertise). All readers participated in joint didactic and image grading training sessions, underwent a certification process, and practiced with training image sets. All readers, including 3 trained readers and 3 expert readers, used the same software to access the images, used standardized workstations to grade images, and recorded findings on the same web-based data collection form. Each image set was graded independently by 2 trained readers, with discrepancies adjudicated by a reading supervisor. All readers were masked to the results of diagnostic examinations, previous gradings for either eye of the infant, and demographic data.

All readers determined the quality of each of the 5 retinal images in a set as good, acceptable, poor, or missing. They also determined, by quadrant, whether the posterior pole vessels were normal or sufficiently abnormal to be plus disease and determined zone of vascularization or the zone in which morphologic features consistent with ROP were present. Retinopathy of prematurity was determined by the presence of a demarcation line (stage 1), a ridge (stage 2), extraretinal neovascularization (stage 3), or retinal detachment (stage 4).

Image Selection for Grading

Trained readers graded all images from 242 infants who developed RW-ROP based on the diagnostic examination. Approximately 80% of infants were not expected to develop RW-ROP; therefore, a random sample of 613 infants (60.5%) who never developed RW-ROP was selected a priori. All image sets from this selected subsample of infants (n = 242 + 613 = 855) were graded by the trained readers. A random sample of 200 of these 855 infants was also graded by expert readers (eFigure in the Supplement).

Statistical Considerations
Sample Size

The sample size was determined by the need to have half width of the 95% CI of sensitivity within 5%. Assuming sensitivity between 80% and 95%, approximately 250 infants with RW-ROP were needed for the study.

Sensitivity/Specificity Analysis

At the same session, we compared the RW-ROP finding (positive, negative, or indeterminate) from evaluation of an image set to findings of the diagnostic examination consistent with RW-ROP (presence, absence, or indeterminate). Sensitivity and specificity of image grading of detecting RW-ROP were calculated by using the results of the diagnostic examination as the criterion standard. Eyes with indeterminate status in the diagnostic examination were excluded from the sensitivity/specificity analysis. When the image set did not provide sufficient information for determining RW-ROP status, the eye was scored as RW-ROP positive in sensitivity/specificity analysis because the primary aim of this study was to determine whether referral for a diagnostic examination was warranted.

We included only 1 session of digital image/diagnostic examinations from each eye in the primary analysis (single-session-per-eye analysis). For sensitivity calculation, the session when the diagnostic examination first identified RW-ROP was used, while a random session was chosen for each eye that did not develop RW-ROP for the specificity calculation.

Sensitivity was calculated as the proportion of RW-ROP–positive image gradings when examination indicated RW-ROP presence, and specificity was calculated as the proportion of RW-ROP–negative image gradings when examination indicated RW-ROP absence. The 95% CIs for sensitivity and specificity were calculated, with intereye correlation adjusted by generalized estimating equations24 using the sandwich robust estimate of variance.25 The prespecified subgroup analyses of sensitivity/specificity were performed by birth weight, gestational age (GA), and image quality.

Secondary analyses of sensitivity/specificity were conducted at the infant level by comparing the presence or absence of RW-ROP on examination vs RW-ROP positive/negative status from image grading at 1 selected session (single-session-per-infant analysis) and any sessions (any-session-per-infant analysis). The sensitivity and specificity for detecting the infants who underwent ROP treatment were also calculated based on the last session before treatment (last-session-per-infant analysis).

For per-infant analysis, the negative predictive value (NPV) and the positive predictive value (PPV) were calculated based on their corresponding sensitivity, specificity, and the observed rate of RW-ROP. All the statistical analyses were performed using SAS version 9.3 (SAS Institute Inc).

Population and ROP Status

A total of 1284 infants with BW less than 1251 g were enrolled from May 25, 2011, through October 31, 2013 (eFigure in the Supplement). Twenty seven (2.1%) were discharged prior to a diagnostic examination. Table 1 provides the characteristics of 1257 study infants who completed at least 1 diagnostic examination. The mean BW was 864 g and mean GA of 27 weeks. Most infants had BW less than or equal to 1000 g including 34.1% of infants with BW of 750 g or less and 36.2% with BW from 751 to 1000 g. About half had GA of less than 27 weeks (51.9%) and few infants had GA of 31 weeks or greater (5.9%). More than half were non-Hispanic white, 29.3% black, and 49.2% female; 63.0% were born at the enrolling clinical center and 29.8% were multiple births, of which 84.3% were twin births.

Table Graphic Jump LocationTable 1.  Characteristics of 1257 Study Infants at Enrollment

Among 1257 infants who had eye examinations, the mean (SD) number of examinations per infant was 3.4 (2.1) (median, 3; range, 1-12) and 40% had 4 or more examinations. The median interval between examinations was 9 days (range, 1-54 days), with 11% of examinations within 1 week and 88% within 2 weeks.

At least 1 imaging session was conducted in 1241 infants. The mean (SD) number of imaging sessions per infant was 3.2 (2.0) (median, 3; range, 1-12) and 36% had 4 or more imaging sessions.

Among the 5520 image sets selected for trained reader grading, a total of 27 600 possible individual retinal images were evaluated for availability and image quality. Among these images, 3% were missing, 91% were adequate quality, and 6% were poor quality.

Retinopathy of prematurity was noted on examination in 801 infants (63.7%) (Table 1). Clinical findings consistent with RW-ROP were noted in 1 or both eyes of 244 infants (19.4%) and were bilateral in 87.8% of infants. The presence of RW-ROP could not be determined for only 2.4% of infants. Clinical findings consistent with RW-ROP were detected in 458 eyes (18.2%), most frequently owing to stage 3 or worse alone (48.5%) or in combination with zone I ROP and/or plus disease (33.6%; Table 2). All 3 components (plus disease, zone I ROP, and stage 3 ROP) were rarely present (3.3%).

Table Graphic Jump LocationTable 2.  Combination of RW-ROP Components Among 458 Eyes With RW-ROP Findings From Diagnostic Examinations
Comparison of Results From Examination and Imaging Grading

Among the 5520 pairs of diagnostic examinations and image gradings, their RW-ROP status was in agreement in 78.6% (Table 3). In the 813 pairs (14.7%) with RW-ROP findings on diagnostic examination, image grading by trained readers detected 1 or more of the components of RW-ROP in 77.7% of image sets, while in 19.8%, the image grading did not detect RW-ROP and 2.5% were indeterminate. In the 4648 pairs without RW-ROP on diagnostic examination, trained readers agreed in 3703 image gradings (79.7%), while in 854 pairs (18.4%), trained readers detected findings consistent with RW-ROP. In 91 pairs, RW-ROP status was indeterminate.

Table Graphic Jump LocationTable 3.  Cross-Tabulation of RW-ROP Findings From Diagnostic Examination vs Image Grading of All Sessions per Eyea
Sensitivity and Specificity for RW-ROP

When image sets for a single session were graded by the trained readers and compared with the results of diagnostic examinations (1709 image sets in which the first session when RW-ROP findings were diagnosed were used and a random session for infants without RW-ROP), the sensitivity for detection of RW-ROP was 81.9% (95% CI, 77.4-85.6), with a specificity of 90.1% (95% CI, 87.9-91.8) (Table 4).

Table Graphic Jump LocationTable 4.  Sensitivity and Specificity of Image Grading for Detecting RW-ROPa

As shown in Table 4, when both eyes of the infant were considered, as would routinely be done in a clinical setting, the sensitivity for RW-ROP was 90.0% (95% CI, 85.4-93.5), with specificity of 87.0% (95% CI, 84.0-89.5), NPV of 97.3%, and PPV of 62.5% at the RW-ROP prevalence rate of the study (19.4%).

When RW-ROP results from any session of image grading and diagnostic examination were paired for the infant, the sensitivity increased to 97.1% (95% CI, 94.0-98.6). Specificity for this comparison was 75.9% (95% CI, 72.2-79.1), NPV was 99.1%, and PPV was 49.2%.

We also calculated the sensitivity and specificity of this telemedicine system to detect RW-ROP in infants who underwent treatment in 1 or both eyes. When the last session before treatment was analyzed, sensitivity was 98.2% (95% CI, 94.4-99.4), with specificity of 80.2% (95% CI, 77.0-83.0), NPV of 99.6%, and PPV of 44.3% at a 13.8% treatment-requiring ROP rate. Only 3 of 162 infants treated by clinical center ophthalmologists did not have RW-ROP detected on the last image grading before treatment. On diagnostic examination, 1 infant had zone I stage 3 disease in both eyes and another 2 infants had plus disease in both eyes.

Expert and trained readers independently graded a random sample of 1312 image sets from 100 infants with RW-ROP and 100 without RW-ROP. Using all 1312, expert readers had a lower sensitivity of 85.9% (95% CI, 80.8-89.8) compared with 91.4% (95% CI, 86.1-94.8) for trained readers and a lower specificity of 56.5% (95% CI, 51.9-61.0) vs 73.3% (95% CI, 67.6-78.3) for detecting an eye with RW-ROP.

Timing of RW-ROP Detection on Image Grading

The timing of the detection of RW-ROP was compared between the image grading and diagnostic examination. In 87.5% of the cases (391 of 447), RW-ROP was detected on image grading before or at the same examination that documented RW-ROP findings. Referral-warrantedROP was detected on image grading an average (SD) of 15 (11) days earlier than the examination in 44.7% of cases (200 of 447). Image grading did not detect RW-ROP at any image grading in 7.2% of cases (32 of 447). In 5.4% of cases (24 of 447), RW-ROP was detected an average (SD) of 15 (13) days after the examination documented RW-ROP. The results were very similar when analyzed by infant (data not shown).

Sensitivity and Specificity by BW, GA, and Image Quality

In the single-session-per-eye analysis, sensitivity of RW-ROP detection decreased with increasing BW (Table 5). This pattern was not observed for GA.

Table Graphic Jump LocationTable 5.  Subgroup Analyses of Sensitivity and Specificity of RW-ROP From Trained Readers Grading Single-Session-per-Eye Analysisa,b

The quality of the images submitted was important. When all 5 retinal images were judged to be of good or acceptable image quality by trained readers, the sensitivity was 84.7% compared with 68.0% when 4 or fewer images were of good or acceptable quality. Specificity for this comparison was similar.

The e-ROP Study results provide strong support for the validity of using a telemedicine system consisting of trained nonphysician imagers and readers to detect RW-ROP in infants at risk. When image-set grading of an eye was compared with the results of its paired diagnostic examination, the sensitivity for detection of RW-ROP was 81.9%, with a specificity of 90.1%. When both eyes of an infant were considered, the sensitivity increased to 90.0%, with specificity of 87.0%, NPV of 97.3%, and PPV of 62.5% at the observed RW-ROP rate of 19.4%. Importantly, among infants treated for ROP, the sensitivity of image grading increased to 98.2%.

Among the strengths of the e-ROP Study is the successful use of nonphysician imagers. The imagers were trained to capture standard image sets for grading by nonphysician trained readers using a standard grading protocol. The diagnostic examinations were performed by ophthalmologists with extensive ROP experience. Furthermore, we elected to have the results of nonphysician trained readers as the primary outcome measure because, as in other retinal disorders,26,27 we confirmed in this study that dedicated and trained readers would perform at least as well as ophthalmologist experts and their availability was likely to be higher and sustained. Standardized imaging and grading protocols improve the generalizability of this ROP telemedicine system.

This study was limited by the difficulty of comparing image grading results with a criterion standard with known inherent variability.28,29 Among eyes judged at treatment level by the first examiner in the Cryotherapy for ROP Trial, there was disagreement on a second confirming examination in 12% of the cases.6 Other studies have highlighted the variability in diagnosing plus disease and disease zone among ophthalmologist examiners.3033 This variability across examiners may well help explain some of the false-positive and false-negative cases documented on image grading in the current study. In addition, the e-ROP Study limited enrollment to infants at high risk (ie, BW < 1251 g) and may not be generalizable to all infants eligible for ROP examinations. Furthermore, the e-ROP Study was not designed to assess the time to reporting results to the clinical center or to provide rapid feedback requesting additional images when image quality was poor. Sensitivity improved substantially with better image quality so efforts to improve image quality are warranted.

One of the most important factors to consider in evaluating telemedicine systems for ROP is how many infants with serious ROP might be missed. In this study with a RW-ROP rate of 19.4%, we found the NPV was high (97.3%). Despite this high NPV, if implementing such a system for clinical care, there will be a small number of infants in whom image grading does not detect RW-ROP and who are at risk for progression to blindness. What safeguards need to be in place for such infants? One might consider a more frequent imaging schedule than used in this study that mimicked current clinical care or also consider adding other algorithms, such as early weight patterns in predicting ROP risk,34,35 to determine those infants at highest risk.

Adopting a telemedicine system approach in ROP is influenced by factors other than validity including the availability of ROP specialists, the number of examinations required per week, and the prevalence of RW-ROP in a NICU. Licensing and liability issues must be dealt with, as well as establishing a consistent and reliable reading center. Procedures to address poor image quality are needed. Parental and NICU staff acceptance of a screening program that replaces a physician diagnostic examination with nonphysician imaging needs further assessment.

The results of this study provide important information about the validity of the e-ROP telemedicine system in managing ROP as we move forward to address the broader use of digital retinal imaging in NICUs in the United States and other regions in the world.

Corresponding Author: Graham E. Quinn, MD, MSCE, Division of Ophthalmology, The Children’s Hospital of Philadelphia, Wood Center, 1st Floor, Philadelphia, PA 19104 (quinn@email.chop.edu).

Submitted for Publication: February 27, 2014; final revision received April 8, 2014; accepted April 9, 2014.

Published Online: June 26, 2014. doi:10.1001/jamaophthalmol.2014.1604.

Author Contributions: Drs Quinn and Ying had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Quinn, Ying, Hildebrand, Ells, Baumritter, Kemper, Wade.

Acquisition, analysis, or interpretation of data: Quinn, Ying, Daniel, Hildebrand, Ells, Baumritter, Schron, Wade.

Drafting of the manuscript: Quinn.

Critical revision of the manuscript for important intellectual content: Ying, Daniel, Hildebrand, Ells, Baumritter, Kemper, Schron, Wade.

Statistical analysis: Quinn, Ying, Kemper, Wade.

Obtained funding: Quinn.

Administrative, technical, or material support: Quinn, Ying, Daniel, Hildebrand, Ells, Baumritter, Schron, Wade.

Study supervision: Quinn, Ying, Daniel, Hildebrand, Baumritter.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Hildebrand reported receiving support from Inoveon Corp and has a patent, Digital Disease Management System, with royalties paid to Inoveon Corp. Dr Ells has received personal fees from the scientific advisory board of Clarity Medical Systems. No other disclosures were reported.

Funding/Support: This study was funded by a cooperative agreement grant (U10 EY017014) from the National Eye Institute of the National Institutes of Health, Department of Health and Human Services. Dr Daniel has received grants from the National Institutes of Health and Dr Hildebrand has received grants from the National Eye Institute; this funding was not related to the current research.

Role of the Sponsor: The study was conducted under a cooperative agreement from the National Eye Institute and, as such, Dr Schron, the National Eye Institute project officer for this study, had input in the collection, management, analysis, and interpretation of the data; preparation, review, and approval of the manuscript; and decision to submit the manuscript for publication, but not the design of the study.

Group Information: The e-ROP Study Group investigators include the following: Office of Study Chair: The Children’s Hospital of Philadelphia: Graham E. Quinn, MD, MSCE (principal investigator [PI]), Kelly Wade, MD, PhD, MSCE, Agnieshka Baumritter, MS, Trang B. Duros, Lisa Erbring; Johns Hopkins University: Michael X. Repka, MD (PI), Jennifer A. Shepard, CRNP, Pamela Donohue, ScD, David Emmert, BA, C. Mark Herring; Boston Children’s Hospital: Deborah VanderVeen, MD, Suzanne Johnston, MD, Carolyn Wu, MD, Jason Mantagos, MD, Danielle Ledoux, MD, Tamar Winter, RN, BSN, IBCLC, Frank Weng, Theresa Mansfield; Nationwide Children’s Hospital and Ohio State University Hospital: Don L. Bremer, MD (PI), Richard Golden, MD, Mary Lou McGregor, MD, Catherine Olson Jordan, MD, David L. Rogers, MD, Rae R. Fellows, MEd, CCRC, Suzanne Brandt, RNC, BSN, Brenda Mann, RNC, BSN; Duke University: David Wallace, MD (PI), Sharon Freedman, MD, Sarah K. Jones, Du Tran-Viet, Rhonda “Michelle” Young; University of Louisville: Charles C. Barr, MD (PI), Rahul Bhola, MD, Craig Douglas, MD, Peggy Fishman, MD, Michelle Bottorff, Brandi Hubbuch, RN, MSN, NNP-BC, Rachel Keith, PhD; University of Minnesota: Erick D. Bothun, MD (PI), Inge DeBecker, MD, Jill Anderson, MD, Ann Marie Holleschau, BA, CCRP, Nichole E. Miller, MA, RN, NNP, Darla N. Nyquist, MA, RN, NNP; University of Oklahoma: R. Michael Siatkowski, MD (PI), Lucas Trigler, MD, Marilyn Escobedo, MD, Karen Corff, MS, ARNP, NNP-BC, Michelle Huynh-Blunt, MS, ARNP, Kelli Satnes, MS, ARNP, NNP-BC; Children’s Hospital of Philadelphia: Monte D. Mills, MD, Will Anninger, MD, Gil Binenbaum, MD, MSCE, Graham E. Quinn, MD, MSCE, Karen A. Karp, BSN, Denise Pearson, COMT; University of Texas Health Science Center at San Antonio: Alice Gong, MD (PI), John Stokes, MD, Clio Armitage Harper, MD, Laurie Weaver, Carmen McHenry, BSN, Kathryn Conner, Rosalind Heemer, Elnora Cokley, RNC, Robin Tragus, MSN, RN, CCRC; University of Utah: Robert Hoffman, MD (PI), David Dries, MD, Katie Jo Farnsworth, Deborah Harrison, MS, Bonnie Carlstrom, Cyrie Ann Frye, CRA, OCT-C; Vanderbilt University: David Morrison, MD (PI), Sean Donahue, MD, Nancy Benegas, MD, Sandy Owings, COA, CCRP, Sandra Phillips, COT, CRI, Scott Ruark; Hospital of the Foothills Medical Center: Anna Ells, MD, FRCS (PI), Patrick Mitchell, MD, April Ingram, Rosie Sorbie, RN. Data Coordinating Center: University of Pennsylvania Perelman School of Medicine: Gui-shuang Ying, PhD (PI), Maureen Maguire, PhD, Mary Brightwell-Arnold, BA, SCP, Max Pistilli, MS, Kathleen McWilliams, CCRP, Sandra Harris, Claressa Whearry; Image Reading Center: University of Pennsylvania Perelman School of Medicine: Ebenezer Daniel, MBBS, MS, MPH (PI), E. Revell Martin, BA, Candace R. Parker Ostroff, Krista Sepielli, Eli Smith. Expert Readers: Antonio Capone, MD, The Vision Research Foundation; G. Baker Hubbard, MD, Emory University School of Medicine; Anna Ells, MD, FRCS, University of Calgary Medical Center. Image Data Management Center: University of Oklahoma Health Sciences Center/Inoveon Corp: P. Lloyd Hildebrand, MD (PI), Kerry Davis, G. Carl Gibson, Regina Hansen. Cost-Effectiveness Component: Alex R. Kemper, MD, MPH, MS (PI), Lisa Prosser, PhD. Data Management and Oversight Committee (DMOC): David C. Musch, PhD, MPH (chair), Stephen P. Christiansen, MD, Ditte J. Hess, CRA, Steven M. Kymes, PhD, SriniVas R. Sadda, MD, Ryan Spaulding, PhD. National Eye Institute: Eleanor B. Schron, PhD, RN.

Additional Contributions: We would like to acknowledge gratefully the contributions of Clare Gilbert, FRCOphth, MD, MSc (London School of Hygiene and Tropical Medicine); Alistair Fielder, FRCOphth (professor emeritus at City University London); and Judith Alexander, BA (former director of the Fundus Photograph Reading Center, Scheie Eye Institute, University of Pennsylvania Perelman School of Medicine), for their contributions to the early development of the e-ROP Study. They did not receive compensation from a funding sponsor for their contributions.

Gilbert  C, Fielder  A, Gordillo  L,  et al; International NO-ROP Group.  Characteristics of infants with severe retinopathy of prematurity in countries with low, moderate, and high levels of development: implications for screening programs. Pediatrics. 2005;115(5):e518-e525.
PubMed   |  Link to Article
Gilbert  C.  Retinopathy of prematurity: a global perspective of the epidemics, population of babies at risk and implications for control. Early Hum Dev. 2008;84(2):77-82.
PubMed   |  Link to Article
Kemper  AR, Freedman  SF, Wallace  DK.  Retinopathy of prematurity care: patterns of care and workforce analysis. J AAPOS. 2008;12(4):344-348.
PubMed   |  Link to Article
Kemper  AR, Wallace  DK.  Neonatologists’ practices and experiences in arranging retinopathy of prematurity screening services. Pediatrics. 2007;120(3):527-531.
PubMed   |  Link to Article
Early Treatment For Retinopathy of Prematurity Cooperative Group.  Revised indications for the treatment of retinopathy of prematurity: results of the early treatment for retinopathy of prematurity randomized trial. Arch Ophthalmol. 2003;121(12):1684-1694.
PubMed   |  Link to Article
Reynolds  JD, Dobson  V, Quinn  GE,  et al; CRYO-ROP and LIGHT-ROP Cooperative Study Groups.  Evidence-based screening criteria for retinopathy of prematurity: natural history data from the CRYO-ROP and LIGHT-ROP studies. Arch Ophthalmol. 2002;120(11):1470-1476.
PubMed   |  Link to Article
Lee  P.  Telemedicine: opportunities and challenges for the remote care of diabetic retinopathy. Arch Ophthalmol. 1999;117(12):1639-1640.
PubMed   |  Link to Article
Adriaensen  WJ, Matheï  C, Buntinx  FJ, Arbyn  M.  A framework provided an outline toward the proper evaluation of potential screening strategies. J Clin Epidemiol. 2013;66(6):639-647.
PubMed   |  Link to Article
Holland  W.  Screening for disease: considerations for policy. Health Policy Bull Eur Observator Health Syst Policies.2006;8(3):1-4.
Schwartz  SD, Harrison  SA, Ferrone  PJ, Trese  MT.  Telemedical evaluation and management of retinopathy of prematurity using a fiberoptic digital fundus camera. Ophthalmology. 2000;107(1):25-28.
PubMed   |  Link to Article
Yen  KG, Hess  D, Burke  B, Johnson  RA, Feuer  WJ, Flynn  JT.  Telephotoscreening to detect retinopathy of prematurity: preliminary study of the optimum time to employ digital fundus camera imaging to detect ROP. J AAPOS. 2002;6(2):64-70.
PubMed   |  Link to Article
Ells  AL, Holmes  JM, Astle  WF,  et al.  Telemedicine approach to screening for severe retinopathy of prematurity: a pilot study. Ophthalmology. 2003;110(11):2113-2117.
PubMed   |  Link to Article
Chiang  MF, Keenan  JD, Starren  J,  et al.  Accuracy and reliability of remote retinopathy of prematurity diagnosis. Arch Ophthalmol. 2006;124(3):322-327.
PubMed   |  Link to Article
Wu  C, Petersen  RA, VanderVeen  DK.  RetCam imaging for retinopathy of prematurity screening. J AAPOS. 2006;10(2):107-111.
PubMed   |  Link to Article
Chiang  MF, Wang  L, Busuioc  M,  et al.  Telemedical retinopathy of prematurity diagnosis: accuracy, reliability, and image quality. Arch Ophthalmol. 2007;125(11):1531-1538.
PubMed   |  Link to Article
Photographic Screening for Retinopathy of Prematurity (Photo-ROP) Cooperative Group.  The Photographic Screening for Retinopathy of Prematurity Study (Photo-ROP): primary outcomes. Retina. 2008;28(3)(suppl):s47-s54.
PubMed
Dhaliwal  C, Wright  E, Graham  C, McIntosh  N, Fleck  BW.  Wide-field digital retinal imaging versus binocular indirect ophthalmoscopy for retinopathy of prematurity screening: a two-observer prospective, randomised comparison. Br J Ophthalmol. 2009;93(3):355-359.
PubMed   |  Link to Article
Lorenz  B, Spasovska  K, Elflein  H, Schneider  N.  Wide-field digital imaging based telemedicine for screening for acute retinopathy of prematurity (ROP): six-year results of a multicentre field study. Graefes Arch Clin Exp Ophthalmol. 2009;247(9):1251-1262.
PubMed   |  Link to Article
Silva  RA, Murakami  Y, Lad  EM, Moshfeghi  DM.  Stanford University network for diagnosis of retinopathy of prematurity (SUNDROP): 36-month experience with telemedicine screening. Ophthalmic Surg Lasers Imaging. 2011;42(1):12-19.
PubMed   |  Link to Article
Dai  S, Chow  K, Vincent  A.  Efficacy of wide-field digital retinal imaging for retinopathy of prematurity screening. Clin Experiment Ophthalmol. 2011;39(1):23-29.
PubMed
Weaver  DT, Murdock  TJ.  Telemedicine detection of type 1 ROP in a distant neonatal intensive care unit. J AAPOS. 2012;16(3):229-233.
PubMed   |  Link to Article
Chiang  MF, Melia  M, Buffenn  AN,  et al.  Detection of clinically significant retinopathy of prematurity using wide-angle digital retinal photography: a report by the American Academy of Ophthalmology. Ophthalmology. 2012;119(6):1272-1280.
PubMed   |  Link to Article
Quinn  GE, Ying  G-S, Daniel  E,  et al; e-ROP Cooperative Group.  Telemedicine approaches to evaluating acute-phase retinopathy of prematurity: study design. Ophthalmic Epidemiol. In press.
Liang  K, Zeger  S.  Longitudinal data analysis using generalized linear models. Biometrika. 1986;73(1):13-22. doi:10.1093/biomet/73.1.13.
Link to Article
Smith  PJ, Hadgu  A.  Sensitivity and specificity for correlated observations. Stat Med. 1992;11(11):1503-1509.
PubMed   |  Link to Article
Feman  SS, Leonard-Martin  TC, Andrews  JS,  et al.  A quantitative system to evaluate diabetic retinopathy from fundus photographs. Invest Ophthalmol Vis Sci. 1995;36(1):174-181.
PubMed
Patra  S, Gomm  EMW, Macipe  M, Bailey  C.  Interobserver agreement between primary graders and an expert grader in the Bristol and Weston diabetic retinopathy screening programme: a quality assurance audit. Diabet Med. 2009;26(8):820-823.
PubMed   |  Link to Article
Trese  MT.  What is the real gold standard for ROP screening? Retina. 2008;28(3)(suppl):s1-s2.
PubMed   |  Link to Article
Phelps  DL.  It’s plus disease, isn’t it? Arch Ophthalmol. 2007;125(7):963-964.
PubMed   |  Link to Article
Chiang  MF, Jiang  L, Gelman  R, Du  YE, Flynn  JT.  Interexpert agreement of plus disease diagnosis in retinopathy of prematurity. Arch Ophthalmol. 2007;125(7):875-880.
PubMed   |  Link to Article
Chiang  MF, Gelman  R, Martinez-Perez  ME,  et al.  Image analysis for retinopathy of prematurity diagnosis. J AAPOS. 2009;13(5):438-445.
PubMed   |  Link to Article
Darlow  BA, Elder  MJ, Horwood  LJ, Donoghue  DA, Henderson-Smart  DJ; Australian and New Zealand Neonatal Network.  Does observer bias contribute to variations in the rate of retinopathy of prematurity between centres? Clin Experiment Ophthalmol. 2008;36(1):43-46.
PubMed   |  Link to Article
Wallace  DK, Quinn  GE, Freedman  SF, Chiang  MF.  Agreement among pediatric ophthalmologists in diagnosing plus and pre-plus disease in retinopathy of prematurity. J AAPOS. 2008;12(4):352-356.
PubMed   |  Link to Article
Löfqvist  C, Hansen-Pupp  I, Andersson  E,  et al.  Validation of a new retinopathy of prematurity screening method monitoring longitudinal postnatal weight and insulinlike growth factor I. Arch Ophthalmol. 2009;127(5):622-627.
PubMed   |  Link to Article
Binenbaum  G, Ying  GS, Quinn  GE,  et al; Premature Infants in Need of Transfusion Study Group.  A clinical prediction model to stratify retinopathy of prematurity risk using postnatal weight gain. Pediatrics. 2011;127(3):e607-e614.
PubMed   |  Link to Article

Figures

Tables

Table Graphic Jump LocationTable 1.  Characteristics of 1257 Study Infants at Enrollment
Table Graphic Jump LocationTable 2.  Combination of RW-ROP Components Among 458 Eyes With RW-ROP Findings From Diagnostic Examinations
Table Graphic Jump LocationTable 3.  Cross-Tabulation of RW-ROP Findings From Diagnostic Examination vs Image Grading of All Sessions per Eyea
Table Graphic Jump LocationTable 4.  Sensitivity and Specificity of Image Grading for Detecting RW-ROPa
Table Graphic Jump LocationTable 5.  Subgroup Analyses of Sensitivity and Specificity of RW-ROP From Trained Readers Grading Single-Session-per-Eye Analysisa,b

References

Gilbert  C, Fielder  A, Gordillo  L,  et al; International NO-ROP Group.  Characteristics of infants with severe retinopathy of prematurity in countries with low, moderate, and high levels of development: implications for screening programs. Pediatrics. 2005;115(5):e518-e525.
PubMed   |  Link to Article
Gilbert  C.  Retinopathy of prematurity: a global perspective of the epidemics, population of babies at risk and implications for control. Early Hum Dev. 2008;84(2):77-82.
PubMed   |  Link to Article
Kemper  AR, Freedman  SF, Wallace  DK.  Retinopathy of prematurity care: patterns of care and workforce analysis. J AAPOS. 2008;12(4):344-348.
PubMed   |  Link to Article
Kemper  AR, Wallace  DK.  Neonatologists’ practices and experiences in arranging retinopathy of prematurity screening services. Pediatrics. 2007;120(3):527-531.
PubMed   |  Link to Article
Early Treatment For Retinopathy of Prematurity Cooperative Group.  Revised indications for the treatment of retinopathy of prematurity: results of the early treatment for retinopathy of prematurity randomized trial. Arch Ophthalmol. 2003;121(12):1684-1694.
PubMed   |  Link to Article
Reynolds  JD, Dobson  V, Quinn  GE,  et al; CRYO-ROP and LIGHT-ROP Cooperative Study Groups.  Evidence-based screening criteria for retinopathy of prematurity: natural history data from the CRYO-ROP and LIGHT-ROP studies. Arch Ophthalmol. 2002;120(11):1470-1476.
PubMed   |  Link to Article
Lee  P.  Telemedicine: opportunities and challenges for the remote care of diabetic retinopathy. Arch Ophthalmol. 1999;117(12):1639-1640.
PubMed   |  Link to Article
Adriaensen  WJ, Matheï  C, Buntinx  FJ, Arbyn  M.  A framework provided an outline toward the proper evaluation of potential screening strategies. J Clin Epidemiol. 2013;66(6):639-647.
PubMed   |  Link to Article
Holland  W.  Screening for disease: considerations for policy. Health Policy Bull Eur Observator Health Syst Policies.2006;8(3):1-4.
Schwartz  SD, Harrison  SA, Ferrone  PJ, Trese  MT.  Telemedical evaluation and management of retinopathy of prematurity using a fiberoptic digital fundus camera. Ophthalmology. 2000;107(1):25-28.
PubMed   |  Link to Article
Yen  KG, Hess  D, Burke  B, Johnson  RA, Feuer  WJ, Flynn  JT.  Telephotoscreening to detect retinopathy of prematurity: preliminary study of the optimum time to employ digital fundus camera imaging to detect ROP. J AAPOS. 2002;6(2):64-70.
PubMed   |  Link to Article
Ells  AL, Holmes  JM, Astle  WF,  et al.  Telemedicine approach to screening for severe retinopathy of prematurity: a pilot study. Ophthalmology. 2003;110(11):2113-2117.
PubMed   |  Link to Article
Chiang  MF, Keenan  JD, Starren  J,  et al.  Accuracy and reliability of remote retinopathy of prematurity diagnosis. Arch Ophthalmol. 2006;124(3):322-327.
PubMed   |  Link to Article
Wu  C, Petersen  RA, VanderVeen  DK.  RetCam imaging for retinopathy of prematurity screening. J AAPOS. 2006;10(2):107-111.
PubMed   |  Link to Article
Chiang  MF, Wang  L, Busuioc  M,  et al.  Telemedical retinopathy of prematurity diagnosis: accuracy, reliability, and image quality. Arch Ophthalmol. 2007;125(11):1531-1538.
PubMed   |  Link to Article
Photographic Screening for Retinopathy of Prematurity (Photo-ROP) Cooperative Group.  The Photographic Screening for Retinopathy of Prematurity Study (Photo-ROP): primary outcomes. Retina. 2008;28(3)(suppl):s47-s54.
PubMed
Dhaliwal  C, Wright  E, Graham  C, McIntosh  N, Fleck  BW.  Wide-field digital retinal imaging versus binocular indirect ophthalmoscopy for retinopathy of prematurity screening: a two-observer prospective, randomised comparison. Br J Ophthalmol. 2009;93(3):355-359.
PubMed   |  Link to Article
Lorenz  B, Spasovska  K, Elflein  H, Schneider  N.  Wide-field digital imaging based telemedicine for screening for acute retinopathy of prematurity (ROP): six-year results of a multicentre field study. Graefes Arch Clin Exp Ophthalmol. 2009;247(9):1251-1262.
PubMed   |  Link to Article
Silva  RA, Murakami  Y, Lad  EM, Moshfeghi  DM.  Stanford University network for diagnosis of retinopathy of prematurity (SUNDROP): 36-month experience with telemedicine screening. Ophthalmic Surg Lasers Imaging. 2011;42(1):12-19.
PubMed   |  Link to Article
Dai  S, Chow  K, Vincent  A.  Efficacy of wide-field digital retinal imaging for retinopathy of prematurity screening. Clin Experiment Ophthalmol. 2011;39(1):23-29.
PubMed
Weaver  DT, Murdock  TJ.  Telemedicine detection of type 1 ROP in a distant neonatal intensive care unit. J AAPOS. 2012;16(3):229-233.
PubMed   |  Link to Article
Chiang  MF, Melia  M, Buffenn  AN,  et al.  Detection of clinically significant retinopathy of prematurity using wide-angle digital retinal photography: a report by the American Academy of Ophthalmology. Ophthalmology. 2012;119(6):1272-1280.
PubMed   |  Link to Article
Quinn  GE, Ying  G-S, Daniel  E,  et al; e-ROP Cooperative Group.  Telemedicine approaches to evaluating acute-phase retinopathy of prematurity: study design. Ophthalmic Epidemiol. In press.
Liang  K, Zeger  S.  Longitudinal data analysis using generalized linear models. Biometrika. 1986;73(1):13-22. doi:10.1093/biomet/73.1.13.
Link to Article
Smith  PJ, Hadgu  A.  Sensitivity and specificity for correlated observations. Stat Med. 1992;11(11):1503-1509.
PubMed   |  Link to Article
Feman  SS, Leonard-Martin  TC, Andrews  JS,  et al.  A quantitative system to evaluate diabetic retinopathy from fundus photographs. Invest Ophthalmol Vis Sci. 1995;36(1):174-181.
PubMed
Patra  S, Gomm  EMW, Macipe  M, Bailey  C.  Interobserver agreement between primary graders and an expert grader in the Bristol and Weston diabetic retinopathy screening programme: a quality assurance audit. Diabet Med. 2009;26(8):820-823.
PubMed   |  Link to Article
Trese  MT.  What is the real gold standard for ROP screening? Retina. 2008;28(3)(suppl):s1-s2.
PubMed   |  Link to Article
Phelps  DL.  It’s plus disease, isn’t it? Arch Ophthalmol. 2007;125(7):963-964.
PubMed   |  Link to Article
Chiang  MF, Jiang  L, Gelman  R, Du  YE, Flynn  JT.  Interexpert agreement of plus disease diagnosis in retinopathy of prematurity. Arch Ophthalmol. 2007;125(7):875-880.
PubMed   |  Link to Article
Chiang  MF, Gelman  R, Martinez-Perez  ME,  et al.  Image analysis for retinopathy of prematurity diagnosis. J AAPOS. 2009;13(5):438-445.
PubMed   |  Link to Article
Darlow  BA, Elder  MJ, Horwood  LJ, Donoghue  DA, Henderson-Smart  DJ; Australian and New Zealand Neonatal Network.  Does observer bias contribute to variations in the rate of retinopathy of prematurity between centres? Clin Experiment Ophthalmol. 2008;36(1):43-46.
PubMed   |  Link to Article
Wallace  DK, Quinn  GE, Freedman  SF, Chiang  MF.  Agreement among pediatric ophthalmologists in diagnosing plus and pre-plus disease in retinopathy of prematurity. J AAPOS. 2008;12(4):352-356.
PubMed   |  Link to Article
Löfqvist  C, Hansen-Pupp  I, Andersson  E,  et al.  Validation of a new retinopathy of prematurity screening method monitoring longitudinal postnatal weight and insulinlike growth factor I. Arch Ophthalmol. 2009;127(5):622-627.
PubMed   |  Link to Article
Binenbaum  G, Ying  GS, Quinn  GE,  et al; Premature Infants in Need of Transfusion Study Group.  A clinical prediction model to stratify retinopathy of prematurity risk using postnatal weight gain. Pediatrics. 2011;127(3):e607-e614.
PubMed   |  Link to Article

Correspondence

CME


You need to register in order to view this quiz.
Submit a Comment

Multimedia

Supplement.

eFigure. Flowchart for Enrollment, Diagnostic Examinations, Imaging, and Image Grading

Supplemental Content

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Articles Related By Topic
Related Collections
PubMed Articles