Evaluation of Visual Acuity Measurements After Autorefraction vs Manual Refraction in Eyes With and Without Diabetic Macular EdemaAutorefraction vs Manual Refraction

Visual acuity (VA) is a common outcome measure in clinical research for diabetic eye disease, and as a measure of visual function, it is one of a handful of well-accepted primary end points for new drug registration with the US Food and Drug Administration.¹ For many years, clinical research studies have used the Early Treatment Diabetic Retinopathy Study (ETDRS) testing method for standardizing refraction and subsequent measurement of VA.² However, this method requires substantial investment in training and certification of refractionists, and the procedure itself can be time-consuming, with substantial costs accrued over the course of a large phase 2 or 3 study owing to the efforts of all the associated personnel. Thus, an acceptable, less time-consuming alternative to the rigorous ETDRS refraction procedure might result in substantial savings of cost and time for clinical trials in diabetic retinopathy. It might also improve the recruitment and retention of participants in clinical trials because of shorter and less technically burdensome clinic visits.

One potential alternative to the ETDRS technique of manual refraction is the technique of autorefraction,³ which utilizes a computer-controlled device to provide an objective measure of an individual's refractive error without the need for a skilled refractionist. Since first being described and validated against manual refraction in the early 1970s,^{4 - 7} autorefraction has come into widespread clinical use because of the ease and speed of the semiautomated autorefractors, the lack of need for a trained refractionist, and its commercial availability.

In clinical trials, the role of autorefraction has been limited to providing starting information for subsequent manual refraction. However, results from a recent single-site study⁸ sponsored by the Diabetic Retinopathy Clinical Research Network (DRCR.net) suggest that, with certain devices, autorefraction may be an acceptable substitute for manual refraction in obtaining best-corrected VA in eyes of patients with diabetes mellitus. The study⁸ was performed at a tertiary referral center for diabetes care and enrolled 216 eyes with varying degrees of diabetic retinopathy and VA (20/16 to 20/800 measured with the Electronic Visual Acuity [EVA] test [electronic ETDRS]) using a single autorefractor type. Refractive errors measured by autorefraction and manual refraction were relatively similar, with a median vector dioptric difference (VDD) of 0.71 diopters (D) and a VDD of less than 1.00 D in 70% of eyes. On average, the EVA letter scores after manual refraction (MR-EVA scores) were slightly better than the EVA letter scores after autorefraction (AR-EVA scores), with a median difference (AR-EVA score − MR-EVA score) of −1 letter (25th-75th percentile range, −4 to 2 letters). Furthermore, the variability between the AR-EVA scores and the MR-EVA scores was similar to the test-retest variability of the MR-EVA score itself, and this similarity was present for all VA subgroups.

Although results from this pilot study⁸ suggested that autorefraction may be a feasible alternative to manual refraction, the study was relatively small, was performed only at a single center with a single autorefractor type, and did not focus specifically on participants with diabetic macular edema (DME), which is a common inclusion criterion for trials in diabetic retinopathy. Our study reports results from a substantially larger DRCR.net-sponsored multicenter study designed to compare VA scores obtained after autorefraction with those obtained after manual refraction in patients with and without center-involved DME across a diverse range of clinical sites, autorefractors, and certified refractionists with varying levels of experience.

Our study was conducted at 26 sites participating in the DRCR.net. The study protocol was approved by the institutional review board of each site, and each study participant gave verbal or written informed consent for participation in the study.

To be eligible for participation, a study participant was required to be at least 18 years old with type 1 or type 2 diabetes and have at least 1 eye with (1) an optical coherence tomography central subfield of 250 μm or greater, (2) a Snellen equivalent VA of 20/400 or better, and (3) DME as the primary cause of any decreased vision.

Each participant provided his or her medical history, which was also extracted from available medical records. Data collected included age, gender, race/ethnicity, diabetes type (1 or 2), and concomitant ocular conditions that might contribute to decreased VA.

All study procedures were performed during a single study visit by an experienced examiner certified by the DRCR.net for VA testing and/or refraction, and data were recorded on standardized forms. All study participants had 3 EVA tests performed on each eye: 1 EVA measurement performed using the refraction from an autorefractor and 2 EVA measurements performed using the DRCR.net manual refraction. Each EVA test was performed first on the right eye and then on the left eye. The VA technician was masked to the source of refraction in 92% and 93% of cases when the source was DRCR.net refraction and autorefraction, respectively (the technician was occasionally unmasked when only 1 technician was available to perform the testing). Tests were performed prior to dilation and prior to measurement of intraocular pressure. There was approximately a 5-minute rest period between each test.

For 112 study participants who participated in another DRCR.net protocol, the DRCR.net manual refraction and EVA measurement using the manual refraction (MR-EVAsuppl score) were performed according to the original protocol. A second refraction was then performed using an autorefractor. Two additional EVA measurements were then completed by a VA technician: (1) a second EVA measurement using the DRCR.net manual refraction (MR-EVA score) and (2) an EVA measurement using the autorefraction (AR-EVA score). The order of the 2 VA tests was randomized (Figure 1A), and study participants were masked during each VA test as to which refraction (autorefraction or manual refraction) was being used. The VA examiner was masked to the source of the refraction in 81% and 85% when the source was DRCR.net refraction and autorefraction, respectively.

For 370 study participants who did not participate in another DRCR.net protocol or who participated in another DRCR.net protocol without a protocol refraction or EVA test scheduled as part of their current visit, a DRCR.net manual refraction was performed in each eye in addition to a refraction using an autorefractor. Visual acuity was measured once in each eye using the DRCR.net manual refraction (MR-EVA score) and once in each eye using the autorefraction (AR-EVA score). The order of the 2 EVA tests was randomized. The VA examiner was masked to the source of the refraction in 95% and 95% when the source was DRCR.net refraction and autorefraction, respectively. An second, unmasked EVA measurement using the DRCR.net manual refraction (MR-EVAsuppl score) was completed last (Figure 1B).

Twenty-two different autorefractor models were used in our study, based on the availability at participating sites. The autorefractor models included in our study are listed in Table 1. The 3 most common types of autorefractors included those manufactured by Marco Nidek, Nidek, and Topcon. Of the Topcon models, the 8000 series autorefractors differed from the other autorefractor models included in our study in that they utilize a rotary prism technology that theoretically enables measurements of a wider retinal area through a smaller diameter pupil.

Eyes evaluated using portable autorefractors (eg, Nikon Retinomax), outdated and not commercially available technology (eg, Xinyuan fa-6000 and Topcon KR3000), or missing either of the MR-EVA scores were not included in the analyses (59 of 964 eyes [6%]). The refractive error transformation method originally described by Long⁹ and later modified by Harris¹⁰ and Thibos¹¹ was used for the comparison between autorefraction and manual refraction. Refractive error data were transformed into a spherical equivalent and 2 Jackson-Cross cylinder powers with axes at 180° and 45° using the Thibos method, and the VDD was calculated from these transformations with a modified formula that scales the unit vector to match that of the Harris method. In addition, autorefraction and manual refraction were compared and were considered the “same” if the spherical equivalent difference was 0.25 D or less and if the cylinder difference was 0.25 D or less; “similar” if the spherical equivalent difference was greater than 0.25 D but less than or equal to 0.5 D, with a cylinder difference of less than or equal to 0.5 D or greater than or equal to 0.5 D and an axis difference of 10° or less; “very different” if the spherical equivalent difference was 2 D or greater, if the cylinder difference was greater than or equal to 2 D or greater than 1.5 D, and if the axis difference was 20° or greater; and “moderately different” if none of the these criteria were met.

A relationship of differences (MR-EVAsuppl score − MR-EVA score and AR-EVA score − MR-EVA score) with VA was explored using Bland-Altman plots.¹² Distributions of differences in VA and refractive error according to the refraction method are described using percentiles, rather than limits of agreement, because the differences were not normally distributed. To compute the Bland-Altman coefficient of repeatability, the standard method was used, but differences greater than 3 times the initial Bland-Altman coefficient of repeatability were truncated at this value, and the coefficient was recalculated. A total of 5 MR-EVAsuppl score − MR-EVA score differences and 5 AR-EVA score − MR-EVA score differences were truncated. Because 2 measures of VA based on the (gold standard) manual refraction were available, the estimation of the underlying VA was based on averaging the 2 MR-EVA scores, rather than basing it on an average of the MR-EVA score and the AR-EVA score. Comparisons of differences among subgroups were made using regression models with generalized estimating equations to account for the correlation of data from 2 eyes of the same participant, adjusting for underlying VA and autorefractor as potential confounders. Absolute differences between the AR-EVA score and the MR-EVA score were compared with absolute test-retest differences between the MR-EVA score and the MR-EVAsuppl score, using a paired t test with generalized estimating equations to account for correlation between eyes. A rank-based transformation for normality (van der Waerden scores) was applied to the differences prior to these generalized estimating equation analyses. With subgroup comparisons of MR-EVAsuppl score − MR-EVA score differences and AR-EVA score − MR-EVA score differences, we used the t test with generalized estimating equations and also adjusted for underlying VA and autorefractor as potential confounders. All reported P values are 2-sided and unadjusted for multiple testing. In view of the large number of variables evaluated in the subgroup analyses, only associations with P < .01 were considered to be unlikely due to chance. SAS version 9.1 (SAS Institute) was used for all analyses.

A total of 905 eligible eyes from 458 individuals with diabetes who enrolled in our study were included in the analyses. Of these eyes, 27 eyes (3%) underwent autorefraction that resulted in “no target” readings, leaving 878 eyes from 456 study participants included in the final analyses. The inability to obtain an autorefraction appeared to be associated with worse VA (0%, 1%, 4%, and 8%, respectively, when the manual refraction VA was 20/20 or better, 20/25 to 20/32, 20/40 to 20/80, and 20/100 or worse; P < .001). No associations were detected between age, pupil size, lens status, or autorefractor type and the ability to autorefract successfully.

The mean (SD) age of study participants was 63 (11) years, and 57% were men. The median MR-EVA Snellen equivalent was 20/32, being 20/20 or better in 158 of 878 eyes (18%), 20/25 to 20/32 in 279 of 878 eyes (32%), 20/40 to 20/80 in 307 of 878 eyes (35%), and 20/100 or less in 134 of 878 eyes (15%). The spherical equivalent of refractive error from the manual refraction ranged from −9.38 to 6.88 D. Additional study participant characteristics and ocular characteristics are presented in Table 2.

Refractive errors measured by autorefraction and those measured by manual refraction were similar (median VDD, 0.79 D; 5th-95th percentile range, 0.25-3.55 D), with the VDD differing by 1.00 D or greater in 337 eyes (38%) and by 2.00 D or greater in 122 eyes (14%) (Table 3). The VDD was larger in eyes with worse VA (P < .001) or with a larger refractive error (P < .001). The autorefraction spherical equivalent was similar compared with the manual refraction spherical equivalent (median difference, 0.00; 5th-95th percentile range, −1.75 to 1.13 D), although agreement between the spherical equivalent from autorefraction vs the spherical equivalent from manual refraction was worse in eyes with worse VA and in eyes with a higher refractive error (P < .001 for both; eTable 1).

Of all the autorefractors, the Topcon 8000 series (Topcon RM 8000, KR8000, KR 8800, and KR 8900 models) generated autorefractions most similar to manual refractions. “Same” or “similar” autorefractions were generated for 55% of eyes with a Topcon 8000 series machine compared with 42% of eyes with the Marco Nidek autorefractor, 27% of eyes with the Nidek autorefractor, and 39% of eyes with other autorefractors (eTable 2).

The median difference between the 2 MR-EVA scores (the MR-EVAsuppl score − the MR-EVA score) for all 878 eyes was 0 letters (5th-95th percentile range, −5 to 7 letters), and the median absolute difference was 2 letters (5th-95th percentile range, 0-9 letters) (Figure 2 and Table 4). Thirteen percent of test-retest scores differed by 5 or more letters but less than 10 letters, 2% differed by 10 or more letters but less than 15 letters, and 2% differed by 15 or more letters (Table 4). The overall Bland-Altman coefficient of repeatability (the half-width of the interval containing 95% of test-retest differences) was 9 letters and was larger in eyes with worse VA (5-13 letters; Table 4). Greater absolute differences in test-retest scores were associated with worse VA (P < .001), although the median letter difference between MR-EVA scores (ie, MR-EVA and MR-EVAsuppl) was not statistically different among VA subgroups (P = .18).

Overall, the EVA scores obtained with manual refraction were slightly better than those obtained with autorefraction. The median difference between the 2 EVA measurements (AR-EVA scores − MR-EVA scores) was −1 letter (5th-95th percentile range, −17 to 7 letters) (Figure 3 and Table 4). Of the 878 eyes included in the analysis, 545 (62%) were within −4 to 4 letters, 247 (28%) had an EVA score that was better by 5 or more letters after manual refraction, and 86 (10%) had an EVA score that was better by 5 or more letters after autorefraction. The median absolute difference between AR-EVA scores and MR-EVA scores was 4 letters (5th-95th percentile range, 0-20 letters), with 23% of measurements differing by 5 or more letters but by less than 10 letters, 7% differing by 10 or more letters but by less than 15 letters, and 8% differing by 15 or more letters (Table 4). Greater differences and greater absolute differences between the 2 EVA measurements (ie, AR-EVA scores and MR-EVA scores) were associated with worse VAs (P < .001 and P = .02, respectively).

The absolute differences between the AR-EVA scores and the MR-EVA scores (Table 4) were slightly larger than the differences between the MR-EVA scores and the MR-EVAsuppl scores (P < .001), with the median absolute difference between the AR-EVA scores and the MR-EVA scores ranging from 3 to 4 letters compared with the median absolute difference between the MR-EVA scores ranging from 1 to 2 letters according to VA level (Figure 3 and Table 4). A greater absolute difference between the AR-EVA scores and the MR-EVA scores compared with the absolute difference between the 2 MR-EVA scores was present for all VA subgroups.

Of the 122 eyes that had markedly different refractions, 76 (62%) had an MR-EVA score of 5 or more letters better than the AR-EVA score, and 13 (11%) had an AR-EVA score of 5 or more letters better than the MR-EVA score, whereas 58 (48%) had an MR-EVA score of 10 or more letters better than the AR-EVA score, and 6 (5%) had an AR-EVA score of 10 or more letters better than the MR-EVA score.

Differences and absolute differences between AR-EVA scores and MR-EVA scores were not associated with study participant age, gender, race, DME severity, primary cause of vision loss, pupil size, lens status, or refractive error (eTable 3). However, both differences and absolute differences between AR-EVA scores and MR-EVA scores were highly associated with the type of autorefractor used (P < .001) (Table 5). Of all the autorefractor models included in our study, autorefractors from the Topcon 8000 series generated refractions that yielded VA results most similar to those found using manual refraction. In general, the results with the Topcon 8000 series models were better than the results with other autorefractor models but still worse than the test-retest variability of the MR-EVA scores alone. The median difference between the AR-EVA scores and the MR-EVA scores in the Topcon 8000 series group was 0 letters (5th-95th percentile range, −10 to 7 letters), and the median absolute difference between the AR-EVA scores and the MR-EVA scores in this group was 3 letters (5th-95th percentile range, 0-13 letters) (Figure 4). Test-retest absolute differences for MR-EVA scores of 10 or more letters but less than 15 letters and of 15 letters or more were present in 2% and 2% of eyes, respectively. In the Topcon group, absolute differences of 10 or more letters but less than 15 letters and of 15 or more letters between the AR-EVA scores and the MR-EVA scores still were present for 5% and 4% of eyes, respectively, compared with 8% and 10% of eyes tested with other autorefractor models. The results obtained with the Topcon 8000 series machines were generally consistent between the 8 clinical sites that used these models, although one of these sites appeared to have a greater variability than the others between AR-EVA scores and MR-EVA scores, accounting for 4 of the 5 observations with an AR-EVA and MR-EVA absolute difference of 30 letters or more.

The training and certification of examiners to accurately refract study participants for the determination of best-corrected VA is both time-consuming and expensive. The rigorous manual refraction protocol currently in use also is time-consuming to perform and lengthens the duration of visits for study participants. Thus, the ability to substitute an automated refraction for manual refraction could streamline study visits and result in substantial savings of time and cost for clinical staff. However, our results, obtained across a broad range of sites with a large variety of autorefractors, do not support the general substitution of autorefraction for a standardized manual refraction for most clinical trial protocols at this time.

Although differences in performance were noted between autorefractor models, with the Topcon 8000 series autorefractors performing on average most similarly to manual refraction, there still was wide variability among results generated by the certified VA examiners using these models. Thus, even with this model, the discrepancies between the MR-EVA scores and the AR-EVA scores do not support using the AR-EVA score in lieu of the MR-EVA score routinely for clinical trial purposes. Topcon 8000 series autorefractors differ from the other autorefractor models included in our study in that they utilize a rotary prism technology enabling evaluation of a wider retinal area through a smaller diameter pupil. It is conceivable that the differences in autorefractor performance seen in this study are tied to this difference in autorefractor hardware and that further improvements in autorefraction are possible. It is unlikely that differences between autorefractors were due to differences in performance across clinical sites because, overall, there was no substantial difference in variability of VA measurements among the individual sites that used the Topcon 8000 series machines.

If substitution of autorefraction for manual refraction is desired, these results suggest that some types of clinical trials with specific outcomes or treatment algorithms may lend themselves more readily than others to this approach. To some extent, increased variability of VA measurements associated with autorefraction could be accounted for in study design by increasing sample size (eTable 4). However, the wide range of differences seen between vision tested after autorefraction vs vision tested after manual refraction would make autorefraction a poor substitute for manual refraction in clinical trials in which treatment decisions for individual patients are driven by relatively small changes in VA (eg, 5 letters). Autorefraction might be a more feasible substitute for manual refraction in studies in which VA is evaluated solely for determination of outcomes across a large study population or in which VA is a secondary or tertiary outcome variable. However, in these studies, careful consideration of the value of VA for assessing adverse events would also have to be considered.

The results from our study also confirm that there is a high level of test-retest reliability of EVA measurements in eyes with DME. The EVA test was performed twice with the same manual refraction in our study to serve as the benchmark for comparison of AR-EVA scores and MR-EVA scores. The test-retest variability of the MR-EVA scores for eyes with DME in our study was similar to that for eyes without DME. Furthermore, in our study, the results for the 81% of eyes with DME (half-width of 2-sided CIs: 9 letters, 83% and 96% within 5 and 10 letters, respectively) were comparable to a previous study¹³ in which only a small proportion (10%) of participants had DME (half-width of 2-sided CIs: 8 letters, 89% and 98% within 5 and 10 letters, respectively). The limitations of our study include the fact that a small percentage of VA examiners were not masked to the refraction source. However, the use of the electronic ETDRS testing protocol to measure VA minimizes the potential bias that could be attributed to an examiner, and no statistically significant difference was seen in absolute differences between VA based on autorefraction and VA based on manual refraction for the results from the more than 90% of VA examinations that were performed in a masked fashion vs those that were unmasked. In addition, although a relatively low percentage of eyes in our study had decreased VA, we were still able to determine a significant difference between AR-EVA scores and MR-EVA scores as measured in different VA subgroups. We were able to enroll substantial numbers of eyes to assess 3 most commonly available autorefraction models (Marco Nidek, Nidek, and Topcon), but for the other, generally older autorefractor models, we were limited by their relatively small numbers. Finally, although our study evaluated the test-retest reliability of MR-EVA scores, it did not assess the test-retest reliability of AR-EVA scores. It is possible that autorefraction, if highly repeatable, might be useful for following changes in VA over time.

In summary, these results demonstrate that VA after autorefraction tends to be slightly worse than VA after manual refraction. Variability between VA after autorefraction and VA after manual refraction is substantially higher than the test-retest variability of manual refraction. We also observed important differences among autorefractor models. Although, in general, autorefraction may not be an acceptable substitute for manual refraction, specific elements of the study design, including increased sample size and nonreliance of treatment algorithm on small differences in VA, may allow limited substitution of autorefraction for manual refraction in some studies.