Evidence-based measurement

Participants and treatment.

Participants were recruited at two clinical sites: the Neurorehabilitation Unit (NRU) at the National Hospital for Neurology and Neurosurgery in London, and the Rehabilitation Research Unit (RRU) at Southampton General Hospital. Both units provide intensive, multidisciplinary, goal-oriented, inpatient rehabilitation that is tailored to individual patients according to the neurologic diagnosis, disabilities, handicaps, and needs. The recruitment strategy differed at each unit because of the varying rates of patient turnover and was predetermined to limit selection bias. The NRU recruited the first two admissions each week for 18 months, whereas the RRU recruited consecutive admission for 12 months. Ethical approval and informed consent were obtained.

Outcome measures.

The BI, FIM, and FIM+FAM are observer-rated, multi-item, summed rating scales to evaluate disability in terms of dependency (table 1). As recommended,12 the version of the BI by Collin et al. was used in this study. It includes items with 2-point (two items), 3-point (six items), and 4-point (two items) response options that are summed to generate a total score. In contrast, FIM and FIM+FAM items have 7-point response options and three summed scores can be generated for each measure: total, motor, and cognitive subscales. For all measures, low scores indicate greater disability. Reliability, validity, and responsiveness have been demonstrated for the BI and FIM (for reviews, see McDowell and Newell12), but the FIM+FAM has had limited psychometric evaluation. Staff raters received formal training in rating the FIM and FIM+FAM.

Other validated measures were collected to evaluate disability (Office of Population Censuses and Surveys Disability Scales, OPCS4); handicap (London Handicap Scale, LHS13); physical and mental health status (Medical Outcomes Study 36-Item Short-Form Health Survey [SF-36]14); psychological well-being (General Health Questionnaire [GHQ-28]12); global cognitive function (four verbal subtests of the revised Wechsler Adult Intelligence Test [WAIS-R Verbal IQ]15); reasoning (Verbal and Spatial Reasoning Test [VESPAR]15); and verbal memory (California Verbal Learning Test [CVLT]15)

Data collection.

The following data were collected within 4 days of admission: demographic and diagnostic information; the BI, FIM, and FIM+FAM, rated by consensus opinion of the treating multidisciplinary team; the LHS, OPCS, SF-36, and GHQ-28, administered by the study coordinators; and WAIS-R Verbal IQ, VESPAR, and CVLT, administered by a neuropsychologist (D.W.L.) at the NRU. Within 2 days of discharge, the multidisciplinary team rated the BI, FIM, and FIM+FAM again.

Analyses.

Standard traditional psychometric analyses16-18⇓⇓ were undertaken to evaluate whether the items of the BI, FIM, and FIM+FAM could be summed to generate total scores, and whether these resulting total scores satisfied recommended criteria for acceptability, reliability, validity, and responsiveness. Three comparisons were made. The total scores of the FIM and FIM+FAM were compared as measures of global disability, the BI and the motor subscale scores of the FIM and FIM+FAM were compared as measures of physical disability, and the cognitive subscale scores of the FIM and FIM+FAM were compared as measures of cognitive disability.

The basic assumption underlying summated rating scales is that items can be summed without weighting or standardization if they are internally consistent.1,19⇓ Three indicators of internal consistency were examined: corrected item-total correlations, mean interitem correlations, and alpha coefficients. Corrected item-total correlations are product-moment correlations between each item and the sum of the remaining items in the scale; they indicate the extent to which each item relates to the construct measured by the total score. “Correcting” the total score by removing the item of interest prevents spuriously high values due to item overlap.20 Recommended minimum values include 0.20,17 0.30,16 and 0.40.21 We adopted the most stringent of these criteria. Interitem correlations indicate the extent to which the individual items of a rating scale are related. It is recommended that the mean interitem correlation should exceed 0.30.22 Cronbach’s alpha coefficients23 estimate the internal consistency of an item group. Alpha coefficients exceeding 0.80 are considered acceptable for scales used to make group comparisons.16

Acceptability, the extent to which the score distributions of a rating scale adequately represent the distribution of health in a sample, was determined by examining the distributions of admission total scores.18 It is recommended that scores span the full scale range, mean scores be near the scale midpoint,22 and floor and ceiling effects (percentage of sample scoring minimum and maximum scores, respectively) do not exceed 20%.24

Reliability is the extent to which a rating scale is free from random error.16 Two types of reliability were examined. Internal consistency was determined for admission scores as described previously. Intrarater reproducibility was estimated in a subsample of patients by determining the agreement, reported as an intraclass correlation coefficient (ICC), between ratings made by the same multidisciplinary team for the same patients on two different occasions. There are numerous versions of the ICC.25 Because the members of the rating multidisciplinary teams varied, the estimates of variance were obtained from repeated-measures analysis of variance under a random effects model.26 Values should exceed 0.80.16 The reproducibility of the BI was not examined because this has been shown to be high in several previous studies (for review, see McDowell and Newell12).

Two types of validity were examined. Concurrent validity, the extent to which rating scales measure the same construct, was quantified using Pearson’s product-moment correlations and an ICC.27 ICCs exceeding 0.75 have been suggested to indicate “excellent” agreement.28 However, because the magnitude of both Pearson’s correlations and ICCs is influenced by the range of scores and presence of extreme values,29 scatterplots of scores also were examined. Because each scale has a different scale range, they were transformed to a 0 to 100 range using the formula suggested by Stewart and Ware2:equation

Convergent and discriminant construct validity, the extent to which a scale’s correlations with measures of similar and dissimilar constructs conform with a priori hypotheses,30 was determined by examining the magnitude and pattern of product-moment correlations with OPCS, LHS, SF-36, GHQ-28, and WAIS-R.

Responsiveness was determined by calculating effect sizes31 from admission and discharge total scores. There are many effect size calculations.32 Standardized response means,33 the mean change score divided by SD of change scores, were chosen because these are the most relevant to clinical studies. A multiple of this statistic is used to determine the statistical significance of within-group change.

Internal consistency.

Corrected item-total correlations exceed 0.40, mean interitem correlations exceed 0.30, and alphas exceed 0.80 (table 3). These findings support the generation of summed scores for the BI, FIM, and FIM+FAM.

Acceptability.

All scales demonstrate good score variability, have mean scores near the scale midpoints, and small floor and ceiling effects (see table 3). Therefore, all scales satisfy recommended criteria for acceptability. There appear to be no clear advantages for any one scale over corresponding measures of global, motor, or cognitive disability.

Reliability.

All scales satisfy recommended criteria for internal consistency reliability (see earlier) and intrarater reproducibility (see table 3). Estimates for competing scales are similar, suggesting that there appear to be no clear advantages in reliability of one scale over corresponding scales.

Validity.

Scales purporting to measuring the same aspect of disability were highly related (Pearson’s r = 0.96 to 0.996) and had high levels of agreement (ICC = 0.95 to 0.995). The scatterplots (not shown) demonstrate that the strong relationships between competing scales hold throughout the full range of scores, and that the results are not biased by extreme values. These results suggest that the FIM and FIM+FAM total scores; BI, FIM, and FIM+FAM motor scores; and the FIM and FIM+FAM cognitive scores measure very similar constructs.

Table 3 shows correlations between the BI, FIM, and FIM+FAM and six measures of similar and different constructs. For each of the scales and subscales, the direction, magnitude, and pattern of correlations provide evidence for their convergent and discriminant validity. For example, the BI correlates highly with the OPCS disability measure (evidence for convergent validity) and has low to moderate correlations with measures of handicap, health status, psychological distress, and neuropsychological functioning (evidence for discriminant validity). More important, the magnitudes of correlation between corresponding scales and subscales of the BI, FIM, and FIM+FAM and the six measures of similar and dissimilar constructs are very comparable, suggesting that they have similar convergent and discriminant construct validity.

Responsiveness.

Standardized response means for BI, FIM, and FIM+FAM scales measuring global, motor, and cognitive disability are similar (see table 3), suggesting that there is no advantage in responsiveness of one measure over another.