Effect of training and different measurement strategies on the reproducibility of brain MRI lesion load measurements in multiple sclerosis

Quantitative assessment of lesion load on brain MRIs from patients with MS is increasingly being used to monitor disease evolution, either natural or modified by treatment.^1,2 The expected yearly change of lesion load in patients with MS has been estimated from the placebo arm of previous clinical trials to be around 10%.³ It has already been demonstrated that several factors markedly influence the reproducibility of lesion load measurements of brain MRI abnormalities from patients with MS, including intrapatient biological variations,⁴ the use of multiple MR scanners,⁵ different acquisition variables,^6-8 segmentation techniques,^9,10 and accuracy of repositioning.^11,12 Because the magnitude of the variability introduced by all these factors may make it impossible to detect any reliable lesion load change over time, several strategies to reduce the effect of these sources of variations have also been suggested and developed.^4,5,12

There are, however, at least two other factors, operator training and measurement strategy, that are likely to be major sources of variation when measuring lesion load present on MRIs from MS patients. These aspects have not yet been systematically investigated. Techniques to segment MS lesions are based on two steps: lesion identification and lesion delineation.¹ However, no efforts have been made to establish common rules for measurements or to evaluate the effect on reproducibility of measurements due to the two steps.

In this study, we established some common rules for lesion load measurements of brain MRI abnormalities in patients with MS and evaluated the effect of training and different measurement strategies on intra- and interobserver reproducibility of such measurements using both manual outlining and a semiautomated local thresholding technique (LTT).

Methods. Effect of training. Five patients with clinically definite MS¹³ in a long-term natural history study in Milan were selected; they had a wide and representative range of brain MRI abnormalities. For each patient, three slices of the proton-density scans (TR/TE = 2000/50; 24 contiguous interleaved axial slices with a slice thickness of 5 mm; field of vision [FOV] = 250 mm; matrix size = 256 × 256; 1 excitation) obtained using a magnet operating at 1.5 T were chosen, one showing the posterior fossa structures, one the supratentorial perivetricular areas, and one the cortex and the subcortical white matter. Five observers (MF, MGC, CG, JHvW, and JG) in five different centers measured the abnormalities present on these slices on two occasions (separated by an interval of 1 month), using manual outlining and the LTT. These observers had considerable experience using both techniques to segment MS lesions but had not developed any consensus regarding their application. Thus, for the first set of measurements, each observer used a different approach based on personal experience. Then, all observers met for a full-day session to discuss any discrepancies among the measurements. Rules to reduce measurement variability were discussed and formalized by one of them (MF) (Appendix). These rules, which included the definition of a common strategy for measurements and guidelines indicating standard procedures to be followed for each step of the measurement process, were circulated among the five observers, who again repeated the two sets of measurements on the same material with the same time schedule and segmentation techniques used in the untrained situation.

Effect of different measurement strategies. Five patients with clinically definite MS,¹³ who were part of the placebo arm of a clinical trial carried out in London and Amsterdam, were selected with the same criteria used to select the first group of patients. For each patient, three slices (located approximately at the same anatomic levels of the first set) of the proton-density scans (TR/TE = 2000/34; 24 contiguous interleaved axial slices with a slice thickness of 5 mm; FOV= 250 mm; matrix size - 256 × 256; 1 excitation), obtained using a machine operating at 1.5 T, were measured. These slices were distributed to the same five observers together with the corresponding T2-weighted images(TR/TE = 2000/90; 24 contiguous interleaved axial slices with a slice thickness of 5 mm; FOV = 250 mm; matrix size = 256 × 256; 1 excitation) and the two adjacent proton-density and T2-weighted slices to increase confidence in lesion identification. The five observers were asked to measure the abnormalities present on these slices on two occasions (separated by an interval of 1 month), using the same two segmentation techniques. Three months later, the lesions present on such images were identified by a radiologist(MGC) using a small square region of interest (ROI) placed in the center of each lesion and the same five observers were asked to repeat again the entire measurement procedure.

Statistical analysis. The effect of training and different measurement strategies on measured lesion loads, defined as the total lesion volumes present on the scans from each patient, were evaluated separately using the two different sets of images. Mean lesion loads (averaged from the measurements of the five observers) obtained before and after training and with different measurement strategies were compared with the Student's t-test for paired data. The analysis of the intraobserver variability of each condition was performed using a two-way ANOVA correcting for observer and patient effects. To evaluate the interobserver variability of each condition, the two measurements obtained by each observer in each condition were averaged. Then, the interobserver variability was estimated by a one-way ANOVA correcting for patient effect. All estimated variances were compared using the F-test. The ANOVA was applied to logarithmically transformed data to account for heteroscedasticity. Indicating as "s" the antilog of the SDs of the transformed volumes, we obtained our measure of variability defined as proportional variation (PV) = s - 1. Differences were considered statistically significant when p < 0.05.

Results. Effect of training. As expected (seeAppendix), the mean lesion load volume was significantly lower when the rules for lesion load measurements were used, both for manual outlining (p < 0.001) and the LTT (p < 0.001)(table 1).

In table 2, the intraobserver variabilities obtained before and after the use of the rules for lesion load measurements are reported. The differences found were not significant both for manual outlining (F_25,25 = 1.53, p > 0.10) and for LTT(F_25,25 = 1.10, p < 0.25).

On the other hand, the interobserver variabilities obtained with manual outlining or the LTT were reduced when rules for lesion load measurements were used: the PV for manual outlining was 34% without rules and 23% with rules (F_20,20 = 2.12, p = 0.05) and the PV for the LTT was 24% without rules and 13% with rules (F_20,20 = 2.05, p = 0.05) (see table 2).

Effect of different measurement strategies. The mean lesion volume obtained in the second set of scans when measurements were performed with the use of adjacent slices and the corresponding T2-weighted images was not significantly different from that obtained when measurements were performed with the use of radiologic marking (table 3).

For manual outlining, the intraobserver variability was similar for the two measurement strategies (F_25,25 = 1.07, p > 0.25), whereas for the LTT, the intraobserver variability was significantly reduced by the use of the radiologic marking (F_25,25 = 2.33, p< 0.05) (table 4).

The interobserver variabilities for both manual outlining (F_20,20= 1.12, p > 0.10) and the LTT (F_20,20 = 1.24, p > 0.10) were not different when adjacent slices and the corresponding T2-weighted images or the radiologic marking were used (seetable 4). However, the PV for both measurement strategies were lower than those obtained in the untrained situation.

Discussion. In this study, we evaluated the intra- and interobserver variabilities in measuring brain MRI lesion loads in MS using manual outlining and and LTT. We also evaluated whether training and different measurement strategies reduced these variables.

Effect of training. The intraobserver variability we found in the free condition was slightly higher than expected from previous studies.^2,9,10 There are three possible explanations for this finding. First, the first set of slices provided to the observers was particularly difficult to measure because heavily T2-weighted images and contiguous slices were not available. Both are available in the daily-life situation and will undoubtedly increase the degree of certainty in lesion identification for equivocal areas of increased signal. Second, because only three slices per patient were measured, the total lesion loads were much lower than those normally detected in MS. This is another possible source that can artificially increase the variability of the measurements, because for high lesion loads, although not yet proven, it is likely that overestimates and underestimates of lesion sizes can counteract each other. Third, four of five observers were asked to measure MRI abnormalities on images from scanners different from those they were used to. This situation is realistic, because all phase III trials in MS are multicenter and thus using scanners from different manufacturers. As already suggested,⁵ it is possible that the degree of experience in measuring images from a single manufacturer is associated with better measurement reproducibility.

Nevertheless, the interobserver variability for both segmentation techniques was much higher than expected.^2,9,10 Because the observers who took part in this study all have considerable experience in measuring MRI lesion load in MS, it is conceivable that, in addition to the abovementioned factors the use of different measurement strategies should be considered an important source of variation.

During the training session, the main source of variation was lesion identification in patients with lower lesion loads and both lesion identification and outlining in those with higher lesion loads. Differences in lesion identification were especially noted in the posterior fossa (some observers missed "low intensity" lesions, whereas other observers included some areas of hyperintensity that reasonably could be considered as flow artifacts) and in cortical/subcortical areas (some observers interpreted possible lesions as partial volume from adjacent gray matter or enlarged sulci, whereas other included in the measurements areas that could reasonably be considered as Virchow-Robin spaces) (figure 1, a and b). On the basis of these observations, we identified possible ways to increase reproducibility (seeAppendix). We are aware that some of these rules are arbitrary. However, the ultimate goal of such rules is to ameliorate the reliability of measurements in settings, like clinical trials, where reproducibility is more important than accuracy if changes of lesion loads over time are to be detected.¹⁴

For both the segmentation techniques, the use of rules for lesion load measurements in MS and the training performed had a clear effect on the interobserver variability but not on the intraobserver variability.

For reasons explained in the Appendix, the approach used to design the rules was "conservative" (i.e., false-negative ROIs were preferred to false-positive ROIs). Our results indicate that the observers applied the rules in the right way because the lesion volumes measured after training were significantly lower than those obtained in the free condition. This is also the most likely explanation for the dramatic reduction of interobserver variability.

The lack of a positive effect on intraobserver variability might be secondary to the fact that even if the rules and the training have an overall positive effect on interrater variability, they might also be confounding for some of the observers for several reasons: difficulty in changing operator strategies when operators are experienced, a too short training period, and a need for visual examples of the rules. As a consequence, one could argue that the use of these rules and a longer and more structured training period might have a stronger positive effect on less experienced observers. Another possible explanation for the absence of any effect on intrarater reproductibility is, for obvious reasons, that the rules pay more attention to lesion outlining rather than lesion identification. Therefore, our results confirm the conventional wisdom that both the steps used in lesion segmentation are crucial in determining the interobserver variability. Assuming that rules were more effective in reducing variability from lesion outlining, other strategies to reduce variability from lesion identification were tested.

Effect of different measurement strategies. For the second set of measurements, both radiologic marking and adjacent slices and corresponding T2-weighted images were aimed at reducing measurement variability secondary to lesion identification. The use of radiologic marking was particularly helpful in reducing intraobserver variability when the LTT was used. This is due to the fact that the outlining of lesions is less operator dependent in the LTT(although in a few cases more than one outline can be obtained by clicking the pointer within the lesions at different distances from the edges[figure 2]; seeAppendix) and any improvement in lesion identification should result in better reproducibility. The lack of a similar effect when adjacent slices and corresponding T2-weighted images were used indicates that lesion identification strategies, at least in well-experienced observers, are not affected by such an approach. Again, we do not know whether this might be helpful in less-trained situations.

However, both strategies were helpful in reducing interobserver variability compared with the free condition, but no further positive effects were observed after radiologic marking. Because radiologic marking is time consuming, it might be argued it is not necessary. However, the interobserver PV when this approach was used were slightly better than those obtained for both the segmentation techniques when adjacent and T2-weighted images were used. In addition, again, we cannot rule out a possibly greater effect of such an approach in less-experienced observers, where a personal decision about what has to be considered as an MS lesion might result in a very poor interobserver reproducibilities. Also, in serial scans when the same observer measures all scans, the marking system should improve the reproducibility.

In conclusion, several of our findings demonstrate that lesion identification and outlining are important sources of variation for MRI lesion load measurements in MS and that the magnitude of their effect is similar. In this study, we identified some possible strategies to reduce such variation and demonstrated that they are effective even when experienced observers are involved, thus suggesting greater benefits to less-trained individuals. In this respect, we are currently making available on an internet site for public consultation our training guidelines and associated visual aids showing what we believe are good or poor examples of lesion identification and outlining. The set of rules presented here have already been adopted by all the groups participating in the European Magnetic Resonance Network in Multiple Sclerosis. However, if an operator-dependent segmentation technique is used in large multicenter trials in MS, it is still desirable to use a single observer to measure scans from the same individual patients. It is also advisable that an experienced radiologist identifies MS lesions before quantitative assessment is performed and specific rules and training for lesion outlining are used.

Acknowledgments

We thank Dr. Mark A. Horsfield (Department of Medical Physics, University of Leicester, Leicester, UK) and Dr. Liqun Wang (NMR Research Unit, Institute of Neurology, London, UK) for their helpful contribution during the performance of the study and Mr. David Plummer (Department of Medical Physics, University College, London, UK) for providing the software for image display.

Appendix