Flawed Study Design in Report on Interexpert Agreement of Plus Disease Diagnosis in Retinopathy of Prematurity

We are concerned that the article by Chiang and colleagues¹ and accompanying editorial by Phelps² could lead to mismanagement and inappropriate care for infants with retinopathy of prematurity (ROP). Based on their study, Chiang and colleagues concluded: “Interexpert agreement of plus disease diagnosis is imperfect. This may have important implications for clinical ROP management, continued refinement of the international ROP classification system, development of computer-based diagnostic algorithms, and implementation of ROP telemedicine systems.”^{1 (p875)} These comments imply more than the words stipulate. We believe that many readers will interpret this conclusion to mean that the diagnosis of plus disease is so faulty that it is useless in the management of ROP.

The conclusions drawn by Chiang et al are inappropriate for the design of their study. Selected wide-angle photographs were sent to experts for their review. Noticing considerable disagreement in the diagnosis of plus disease among these experts, Chiang et al made far-reaching conclusions about the inability of experts to agree on the proper diagnosis of plus disease. Such a conclusion would require masked paired examinations of the same infant rather than a review of wide-angle photographs.

Additional flaws in the study design are apparent. The camera used in the experiment offers a minified view compared with the standard reference photograph of plus disease.³ The camera itself places weight on the eye and can change the appearance of plus disease or stage 3 ROP. Experts were shown photos that were devoid of any other anatomical information (eg, presence or absence of stage 3 disease). These additional factors are helpful during indirect ophthalmoscopy in real life. Pictures shown to experts were very ambiguous. Thus, Chiang et al probed interexpert agreement in a gray area, exactly where disagreement is expected.

Given the design of the study, the only appropriate conclusion is that the use of wide-angle photographs leads to considerable variability in the assessment of plus disease. This study calls into question the use of the wide-angle photographs for the diagnosis of plus disease, rather than the diagnosis of plus disease per se.

If readers interpret these articles to indicate that plus disease is an unreliable feature of ROP diagnosis, then their clinical practice behavior may change in a dangerous way. Indeed, it is hard to imagine any resulting change in practice that would benefit the collective group of infants who have advanced ROP. Plus disease is difficult to diagnose clinically in borderline cases; however, when the full spectrum of plus disease is considered, there are few characteristics of ROP that have demonstrated such a strong association with the clinical sequelae of ROP. A change in practice that leaves out plus disease or causes physicians to underestimate its importance, based on this study, could place many children at increased risk of blindness.

We urge clinicians to continue to emphasize the importance of plus disease in clinical decision making. Plus disease has been extensively and clinically tested as one of the most important and alarming findings in ROP. Meanwhile, Chiang and colleagues have shown us that wide-angle photography is not useful for the diagnosis of plus disease. For this observation, we can be grateful.

Correspondence: Dr Good, Smith-Kettlewell Eye Research Institute, 2318 Fillmore St, San Francisco, CA 94115 (good@ski.org).

Financial Disclosure: None reported.