CORNEAL STAINING

A Statistical Analysis of The Staining Grid

Researchers consider clinical and statistical relevance in the data from Andrasko and Ryen's Staining Grid study.

By Sally M. Dillehay, OD, EdD; Bill Long, BS, MBA; & Gary Cutter, PhD

Dr. Dillehay is the Director of Medical Marketing & Clinical Claims for CIBA Vision. She is a Diplomate in the Cornea and Contact Lens Section of the American Academy of Optometry and has conducted several hundred research trials. Mr. Long is the Senior Project Manager in Medical Marketing & Clinical Claims at CIBA Vision. He is a Fellow of the American Academy of Optometry and the British Contact Lens Association. Dr. Cutter is a Professor of Biostatistics and Head, Section on Research Methods and Clinical Trials, Department of Biostatistics at the School of Public Health, University of Alabama at Birmingham.

Recent reports about corneal staining, specifically about the corneal "Staining Grid" developed by Andrasko and Ryen, have raised very important and valid questions about the actual impact of corneal staining. Although the Staining Grid has been available for about a year at www.staininggrid.com, only very recently have Andrasko and Ryen published an article detailing their methodology and results. However, these recent articles haven't answered a fundamental question about the Staining Grid — does the data have any statistical or clinical relevance?

In the world of clinical research, the goal is to show if one product performs differently from another. But the reality of clinical research is that each patient reacts differently to a treatment, and even the same patient may react differently over time. It's only through using appropriate study designs and statistical analyses that we can make reasonable predictions about the possible impact of a product when individual responses vary.

Andrasko and Ryen have in essence published a data set. They provided no statistical testing to demonstrate that any of the numbers on their Staining Grid are in fact different from any of the other numbers for corneal staining on their grid, or if those values occurred because of random chance distribution in their patient sample. Yet Andrasko and Ryen state that you can use their Staining Grid to "choose a solution that is likely to be biocompatible" with a lens material and that it "provides a starting point for making that choice."

The Staining Grid can serve as such a starting point only if we first can answer that most fundamental question of clinical research — whether the data represent anything different from the normal, expected population of patients exhibiting corneal staining.

The Importance of Statistics in Clinical Trials

Let's illustrate the need for statistical testing with an example familiar to most people — the colors of M&M candies. In a small bag with 25 pieces, about 30 percent of the candies will be brown. Simple statistics allows us to understand that 95 percent of the time the percentage of brown candies would vary from 12 percent to 48 percent (a 95 percent confidence interval). So if you got a bag with only 12 percent brown and then you got a bag with 48 percent brown, you might be tempted to say those are "different." But in reality, these are both likely to have arisen from the same batch of candies. The difference in values results from normal variation in the "population" of M&Ms.

So what do M&Ms have to do with clinical research and specifically with corneal staining? The Staining Grid dataset by Andrasko and Ryen shows areas of corneal staining that range from 1 percent to 76 percent. It's by using statistics that we determine whether those two values likely happened because of random chance or if there really was a difference in the various lens/lens care combinations.

Performing the Statistical Analysis

Andrasko and Ryen published sufficient descriptive data to perform statistical testing with some assumptions. Admittedly, many different approaches could be taken for statistical testing of their data, and our ability to conduct statistical testing is limited to the data that they actually published. We conducted a series of statistical analyses ranging from very basic to quite complex. Table 1 shows a comparison of these different statistical approaches to provide a meaningful context in which to examine the Andrasko and Ryen corneal staining results.

Table 1 - Comparison of Results of Different Level of Statistical Analysis of Andrasko & Ryen's "Staining Grid" Data Set (p ≤ 0.05)

Our first analysis created a set of basic descriptive statistics for their overall study results. Their study documented an average corneal staining area of 14.3 percent with a pooled standard deviation of 11.8 percent. These two numbers tell us a lot about their data and corneal staining after two hours of wear. First, of about 2,500 observations, soft lens patients — regardless of the lens/lens care product combination they used during the study — showed on average about 14.3 percent area of corneal staining. Second, the standard deviation of the data shows that staining was a highly variable measurement even when taken repeatedly by the same observers using the same grading scale. Because many of the observations were on the same patients, this variability is underestimated. The standard deviation of their data is 83 percent of the mean, an indication that a larger sample size is likely required when evaluating something with the aim of assessing product differences.

Next, like the M&M example we discussed previously, we calculated the 95 percent confidence interval (CI) for their dataset using the mean of 14.3 percent and the pooled standard deviation of ±11.8 percent. The 95 percent CI for their data is 0 percent to 37.4 percent corneal staining area. The 95 percent CI tells us that if a normal distribution were assumed (a strong assumption for this data in which a large number of zeros occurred) and the exact study were repeated with about 2,500 observations, then 95 percent of patients — regardless of what lens/lens care product combination they used — would likely show somewhere between 0 percent and 37.4 percent area of corneal staining.

And just like the M&M example, we can't automatically assume that values within the CI range are different. They may be just normal variations for a patient within the sample and the differences in values may result from random chance. Further analysis is still required for individual lens/lens care combination values.

To do this, we computed the CI for the mean for each lens/lens care combination evaluated (n=29 or 30 depending on the data) and superimposed the 95 percent CI for the data observed (Figures 1 and 2). We may liberally consider lens/lens care combinations whose CI bar (shown as the vertical bars) is completely above and does not overlap at all with the 95 percent CI range of the overall population distribution (shown as a dashed red horizontal lens) as statistically different from each other. A lens/lens care combination that falls outside the overall 95 percent CI for the data indicates a combination that exhibits corneal staining area that exceeds what is expected just by chance.

We were able to confirm only 11 out of 85 (12.9 percent) lens/lens care combinations evaluated as significantly different than the average observed: ReNu MultiPlus (Bausch & Lomb) with Proclear (CooperVision), SofLens 66 (B&L) and PureVision (B&L); Target generic version of ReNu MultiPlus with Proclear, SofLens 66 and PureVision; Wal-Mart generic version of ReNu MultiPlus with Proclear, SofLens 66 and PureVision; Complete MoisturePlus (Advanced Medical Optics) with SofLens 66 and PureVision.

Next, we determined if corneal staining observed resulted from the lens or lens care product of each combination. We used a general linear model using weighted least squares to adjust for the unequal variances observed over all the cells. This method held the potential impact of each lens constant while we explored the impact of each lens care product on corneal staining, and then held the impact of each lens care product constant while we explored the impact of each lens, assuming no interactions between the two. This further analysis showed that the only lens care products that demonstrated significantly higher amounts of corneal staining were ReNu MultiPlus and both its Target and Wal-Mart generic versions, ReNu with MoistureLoc (B&L) and Complete MoisturePlus. The only lenses that demonstrated significantly higher amounts of corneal staining were Proclear, SofLens 66 and PureVision.

Next we performed a head-to-head comparison of the 85 lens/lens care combinations shown on the staining grid to the other 84 combinations. Because this analysis involved 3,570 multiple comparisons, we made an adjustment to ensure that we weren't picking up differences due to random chance. We used a conservative Bonferroni correction, which adjusts the critical value by the number of comparisons. The lens care products that demonstrated no statistically significantly different amounts of corneal staining with all of the lenses tested included Unisol 4 Saline (Alcon), Clear Care (CIBA Vision), Opti-Free Express (Alcon), Opti-Free Replenish (Alcon) and Aquify MPS (CIBA).

ReNu with MoistureLoc, ReNu MultiPlus (branded and both Wal-Mart and Target generic versions) and Complete MoisturePlus demonstrated significantly varying amounts of corneal staining depending on lens brand evaluated. ReNu with MoistureLoc demonstrated statistically significantly lower amounts of corneal staining when used with Acuvue Oasys (Vistakon), O₂Optix (CIBA) and PureVision than when used with Acuvue 2 (Vistakon). ReNu MultiPlus demonstrated significantly higher amounts of corneal staining with Proclear, PureVision and SofLens 66 as compared to Acuvue 2, Acuvue Advance, Acuvue Oasys, Biofinity (CooperVision), Night & Day (CIBA) and O₂Optix. The results of both Wal-Mart and Target generic versions of ReNu MultiPlus were very similar to those of ReNu MultiPlus.

Complete MoisturePlus demonstrated significantly higher amounts of corneal staining with PureVision and SofLens 66 as compared to Acuvue 2, Acuvue Advance, Acuvue Oasys, Biofinity, Night & Day, O₂Optix and Proclear. Thus, this detailed analysis supported the findings of the other statistical analyses. The only lenses that demonstrated consistently significantly higher levels of corneal staining with any of the lens care products were Proclear, PureVision and SofLens 66. The only lens care products that demonstrated consistently significantly higher levels of corneal staining were ReNu Multi-Plus (branded or generic versions) and Complete MoisturePlus.

We then examined those lens care products from the paired-wise analysis that did not demonstrate significantly different amounts of staining with any of the lenses tested (shown in Table 1 as the Level 4 analysis). We then directly compared these products to the corneal staining observed with Unisol 4 Saline because Andrasko and Ryen used it as the control product. Unpreserved saline such as Unisol 4 should indicate the minimal amount of corneal staining expected from any lens care product. We again made a Bonferroni adjustment to account for multiple comparisons. Only Clear Care and Opti-Free Express demonstrated no differences to Unisol 4 for any of the lens brands tested. Opti-Free Replenish demonstrated significantly higher corneal staining with Acuvue 2, Acuvue Oasys and PureVision; Aquify MPS had significantly higher corneal staining with Proclear, PureVision and SofLens 66.

A different picture results when we apply objective objective statistical analysis to the Staining Grid versus using seemingly arbitrarily defined color coding. Of the 19 lens/lens care combinations coded as red by Andrasko and Ryen, we found only 11 (57.8 percent) were statistically different from the other combinations on the staining grid. Those lens/lens care combinations that consistently demonstrated higher amounts of corneal staining were Proclear with ReNu MultiPlus (branded/generic); SofLens 66 with ReNu MultiPlus (branded/generic) and Complete MoisturePlus; and PureVision with ReNu MultiPlus (branded/generic) and Complete MoisturePlus. For the 10 lens/lens care product combinations color coded yellow by Andrasko and Ryen, we found none (0 percent) to have significantly different amounts of corneal staining as compared to the other combinations. Therefore, this analysis doesn't seem to support the color coding of red to indicate "excessive staining" and yellow to indicate "marginal staining, proceed with caution" used by Andrasko and Ryen. Table 2 offers a revision of their color-coded staining grid, but its color codes are based on scientifically sound principles of statistics to indicate differences in product performances.

Figure 1: Mean and 95% Cnfidence Interval for Andrasko & Ryen Lens-Lens Care Combinations A verticle bar that is above and does not overlap the dashed, red horizontal line indicates corneal staining exceeding that expected by chance.

Andrasko and Ryen used green color coding for all lens brands for Unisol 4 Saline, Clear Care, Opti-Free Express and Opti-Free Replenish. This green color coding suggests that all of these products performed similarly to one another, but statistical analysis demonstrates that this isn't an accurate representation. Additionally, the sample size used by Andrasko and Ryen isn't large enough (N=30) to allow testing for equivalence of products. Equivalence, which the common coloring implies, is a more stringent comparison and often requires much larger sample sizes. Using the ranges of the Andrasko and Ryen dataset, establishing equivalence could require that a sample of 2,500 subjects demonstrate equivalency of the products they color-coded as green. Therefore, a larger sample size is required to demonstrate whether there were performance differences between Clear Care and Opti-Free Express. The current analysis based on this limited data seems to indicate that Clear Care demonstrated a trend to produce less staining than Opti-Free Express for Acuvue Oasys, O₂Optix and PureVision.

Figure 2: Mean and 95% Cnfidence Interval for Andrasko & Ryen Lens-Lens Care Combinations A verticle bar that is above and does not overlap the dashed, red horizontal line indicates corneal staining exceeding that expected by chance.

Staining Grid Study Design

One of several confounding factors in the Andrasko and Ryen study is that some patients participated in more than one product comparison. If someone were a high or low responder, the impact of using that subject repeatedly is a confounding factor that remained unaccounted for in their data. In addition, their study results examined only the corneal staining obtained in the worst eye. Use of the worst eyes accentuates the errors in their measurements and overrepresents the staining. At least 50 percent of the eyes that Andrasko and Ryen evaluated would have shown the same or lesser amounts of corneal staining than what they reported.

Andrasko and Ryen also report on their Web site that: "Lens cases were precycled with the test solutions to eliminate any potential interactions between the solution and lens case. The case is filled with the test solution and allowed to soak for at least eight hours. Then the solution is discarded and the process is repeated a total of 6 times." The utility of a special lens case precycling step in this study is unclear because it's inconsistent with labeled instructions for use of the products, with recognized industry standards for efficacy testing (ISO and ANSI) and with applicable preclinical testing requirements of lens disinfection products by regulatory agencies.

According to other published research by Alcon, which has been the primary sponsor of Andrasko and Ryen's Staining Grid work, precycling of lens cases prior to corneal staining studies is performed for this reason: "Because new plastic lens case materials readily absorb and adsorb lens care ingredients, presoaking lens cases saturates the lens case polymer and allows the lens to be exposed to the nominal formulation concentrations. This procedure is clinically relevant because it mimics circumstances in which new lenses are exposed to old lens cases."

However, Alcon's own scientists published specifically that their preservative is taken up by plastic whereas PHMB used by competitive products was taken up selectively by glass: "Specifically it is known that the active antimicrobial ingredient contained in Alcon's competitors' products, polyhexamethylene biguanide (PHMB, e.g. polyhexanide and Dymed), adheres to glass surfaces….However, the active antimicrobial ingredient in Opti-Free express [sic] may adhere to some types of plastic." This indicates that the preservative in Opti-Free Express and Opti-Free Replenish may be taken up selectively by certain plastics as compared to PHMB. It's unclear whether Andrasko and Ryen used the same type of lens case for all products or the lens cases specific to the lens care products tested. This lens case precycling may impact the corneal staining results if it differentially affected the various lens care products evaluated.

Table 2: Statistically-based Corneal Staining Grid Analysis of the Dataset of Andrasko and Ryen Data Accessed May 27, 2007 from www.staininggrid.com

Additionally, longer term studies of corneal staining do not concur with Andrasko and Ryen's results, especially regarding the amount of corneal staining observed with Opti-Free Express and Opti-Free Replenish. In a three-month evaluation of corneal staining with various silicone hydrogel lenses, Carnt et al (2007) determined that both Opti-Free Express and Opti-Free Replenish demonstrated much higher levels of corneal staining after three months of real world use than what's reported by Andrasko and Ryen after only two hours of non-real world usage. Andrasko and Ryen have also published four-hour data from the same study, which shows significant declines in corneal staining for virtually every lens/lens care combination evaluated. Even studies by Alcon researchers who observed corneal staining over a course of 10 days documented much higher levels of corneal staining with Opti-Free Express. Garofalo et al (2005) reported that in patients using Acuvue Advance lenses examined over a period of 10 days, 25 percent of subjects using Opti-Free Express at two hours, 7 percent at four hours and 13 percent at six hours exhibited what they labeled as "clinically significant" corneal staining.

Why Statistics Matter in Clinical Patient Care

Clinical research is designed to take the questions individual practitioners ask beyond the level of just observation and into interpretation of what different data actually mean. Statistical analysis in clinical research is fundamentally required to determine what are likely real differences in product performance. The application of statistics to well-designed clinical research is the only method for ensuring that recorded differences are not just random occurrences.

As noted by Portney and Watkins (2000), the goal of clinical research is "to contribute to a scientific understanding of clinical phenomena, to predict outcomes and strengthen the theoretical foundations of treatment and evaluation." We commend Andrasko and Ryen for undertaking a large-scale, important clinical question. But it's not enough to just publish a set of data, color code it based on seemingly arbitrary designations and make recommendations on the use of the observations without careful consideration to the meaning of the underlying data. Many of the color coded recommendations by Andrasko and Ryen appear unbased on any analysis, especially given the small sample size of each combination tested and the large variability of the attribute evaluated. Andrasko reported that they are working on publishing the statistical analysis of their data. Our statistical analysis of their data found that the color coding of their staining grid is generally not supported.

We believe that you should consider the colorcoded versions of the Andrasko and Ryen Staining Grid with great caution because they provided no statistical testing to justify any of the product differences suggested. Table 2 in this article uses a statistically based approach to identify which lens/lens care combinations demonstrated higher amounts of corneal staining. This statistically based approach allows practitioners to know that these results are based on scientifically sound principles and not on random chance or arbitrary criteria.

We believe the Andrasko and Ryen color-coded staining grid has blinded many practitioners to the reality of the actual implications of the results of their work. The color coding has led many practitioners to believe that there are substantial product differences in cases where none exist, and that there aren't any substantial differences where some do exist. This analysis of their data supports only consistent, non-random differences in corneal staining observed with ReNu MultiPlus (branded/generic) when used with PureVision, SofLens 66 and Proclear, and for Complete MoisturePlus when used with PureVision and SofLens 66. But it's important to note that this data represents only two hours of wear and that the underlying study design has many issues that may make this data not reflective of realworld wearing conditions.

For eyecare practitioners who are concerned about corneal staining with different lens/lens care combinations, the Andrasko and Ryen data showed that Unisol 4 Saline, Clear Care, Opti-Free Express, Opti-free Replenish and Aquify MPS demonstrated no significant differences in the amount of corneal staining regardless of the lens brand, including with three HEMA and six different silicone hydrogel lens brands evaluated. If you wish to minimize the amount of corneal staining regardless of what lens brand a patient is wearing, then their data demonstrated that only Clear Care and Opti-Free Express demonstrated no significantly higher amount of corneal staining as compared to unpreserved saline (Unisol 4), for all the lens brands evaluated.

Andrasko and Ryen suggested some silicone hydrogel materials are more biocompatible with certain multipurpose solutions and that the introduction of new silicone hydrogel materials and lens care solutions were making the task of selecting a lens/lens care combination that will minimize corneal staining more difficult. They've offered their Staining Grid as one method to help simplify this task. Our objective statistical analysis consistently demonstrated higher corneal staining for two non-silicone hydrogel lenses (Proclear and SofLens 66), only one silicone hydrogel lens (PureVision) and only one currently available lens care solution (ReNu MultiPlus branded and generic). Therefore, these suggestions by Andrasko and Ryen do not appear to be supported by objective analysis of their data. Their Staining Grid may be adding confusion instead of simplifying the realities of what their data supports about differences in product performance for the evaluated lens/lens care combinations. CLS

For references, please visit http://www.clspectrum.com/references.asp and click on document #144.