Validation of the Functional Assessment of Multiple Sclerosis quality of life instrument

Phase 1: item generation.

In the item-generation phase, we compiled an 88-item inventory of quality of life concerns that have either general relevance for chronic illness or specific relevance to the symptoms and problems associated with MS. The goal was to produce a testable measure of quality of life for MS general enough to allow comparison with other chronic conditions yet specific enough to capture the diverse complications of MS. We began with a ``base'' generic set of questions originally developed for people with cancer, the Functional Assessment of Cancer Therapy, General version (FACT-G). ^[11] To this we added a set of questions about symptoms and problems associated with MS.

MS-specific items were generated using semistructured interview input from 20 MS patients with varying degrees of symptomatology and from five MS specialists (MD or RN). After meeting with their physician, 20 patients (13 women and 7 men) were asked to participate in a brief interview ``to help develop a measure of quality of life for people with MS.'' All agreed to participate. Their median age was 52 years (range 30 to 67 years).

After gathering demographic and treatment information, patients were asked to describe their concept of quality of life and to identify issues relevant to the quality of life of MS patients in the areas of physical, social, emotional, and functional well-being. Additional concerns not covered by these areas were also explored. The patients were then asked to rate the relevance of the 28 FACT-G questions to their quality of life. No question was deemed irrelevant by most patients. Therefore, to allow for comparison of quality of life across cancer and MS, all 28 FACT-G items were retained for further testing.

One hundred twenty new items were generated by patients and experts. In addition, 11 questions were drawn from the Fatigue and Spirituality subscales of the FACT Measurement System and the Fatigue Severity Scale developed by the Department of Neurology at the University of Chicago. Four other items were drawn from a review of the MS literature. Thus, 135 new items were generated.

Phase 2: item reduction/scale construction.

The 135 new items were reviewed by two of the original five experts for redundancy, relevance, and relative priority, reducing the list to 60 items. These 60 questions were then reviewed by the study team for clarity and reworded when appropriate. Adding these questions to the 28-item FACT-G produced the 88-item Version 1 of the Functional Assessment of Multiple Sclerosis (FAMS). These items can be found through NAPS. To maintain compatibility with the FACT-G, a five-point (0 to 4) Likert-type response format was selected.

Phase 3: validation of the FAMS.

A large mail survey was conducted to empirically configure the primary subscales of the fams; to obtain a stable estimate of test structure; and to test for internal consistency, reliability, and concurrent validity. review of the 60 additional concerns (see appendix 1 Table 7, Table 8, Table 9) suggested that a factor analysis approach would produce a set of subscales distinct from those of the original fact-g. a second smaller study of patients seen in the clinic was also conducted to obtain data comparing neurologic assessment with patient responses to questionnaires (criterion-related validity) and to retest the same patients within 1 week to obtain coefficients of test stability (test-retest reliability).

The sample is therefore divided into two groups: those who received a mail survey with paper-and-pen assessment instructions supplemented by telephone assistance when needed (hereafter the survey sample) and those studied during a clinic visit (hereafter the clinical sample). The clinical sample was administered the same packet of questionnaires as the survey sample. At the time of their clinic visit, two clinical ratings of current disability, the Kurtzke Extended Disability Status Scale (EDSS) ^[12] and the Scripps Neurological Rating Scale (NRS), ^[13] were recorded. The clinical sample was also given a second copy of the FAMS to be completed at home 3 to 7 days later.

Subjects.

The survey and clinical samples consisted of patients treated for MS at one of two Chicago hospitals: The University of Chicago Hospital Neurology Clinic or the Rush-Presbyterian-St. Luke's Multiple Sclerosis Center. The patient population came from a range of urban, suburban, and rural areas in the Metropolitan Chicago area and outlying counties and states. To maintain confidentiality, records were kept according to identification numbers and not patient names.

The survey sample consisted of 377 patients of 508 packets mailed (or handed) out or a 74% return rate for analyzed records. Thirty-two patients (6%) returned packets too late to be included in the analyses for this report. Of the 97 packets not returned, 1 patient had died, 2 were unable to participate because of severe disability requiring nursing home admission, 5 refused, 31 had moved and left no forwarding address, and 58 were still being pursued with telephone contact at the time of data analysis. Thus, the return rate of possible returns was 85%.

The clinical sample consisted of 56 patients from the same two treatment centers, derived from a pool of 69 eligible patients for whom at least partial clinical data are available. Although none of the 69 approached patients refused to participate, 13 (19%) did not provide complete data, so they were excluded from the analyses.

The survey and clinical samples were comparable in demographic and clinical characteristics. Average age for the entire sample of 433 patients was 44.9 years (range 22 to 86); 70% of the patients were women. Most were married (73%) and were white (non-Hispanic) (90%). It was a generally well-educated group, with a mean education of 14 years (range 7 to 29 years). One-third of the patients were on disability. There was an even balance between patients with relapsing/remitting disease (n = 194) and those with progressive disease (n = 174). Mean EDSS score for the clinical sample was 4.6 +/- 2.2 (range 0 to 8). Mean NRS score was 71 +/- 16.4 (range 31 to 100). More detailed demographic and clinical data are on file with NAPS.

Measures and procedure.

All patients (survey and clinical samples) were administered a battery of self-report instruments: the 88-item FAMS version described above, the Eastern Cooperative Oncology Group (ECOG) Performance Status Rating (PSR), ^[14] the RAND 36-item Health Survey (SF-36), ^[15] the Multiscale Depression Inventory (MDI), ^[16] The Hospital Anxiety and Depression Scale (HADS), ^[17] and a 10-item short form of the Marlowe-Crowne Social Desirability Scale (MCSDS). ^[18] Each measure was chosen for a specific reason. The ECOG PSR is a single-item rating of current activity level, ranging from 0 (fully ambulatory without symptoms) to 4 (bedridden). It was selected because it is commonly used in clinical oncology research and would therefore permit categorizing MS patients according to the same criteria used in oncology. This allows for future comparison of quality of life across the two illnesses and differing activity levels. The SF-36 is a commonly used generic health status instrument that allows comparison of FAMS responses to a questionnaire that purports to measure the same thing as the FAMS (i.e., health-related quality of life), albeit without the MS-specific questions. The SF-36 produces two summary health status scores, a Physical Component Summary (PCS) scale and a Mental Component Summary (MCS) scale, ^[15] which were used in this study to compare patient scores across the two questionnaires. High associations between scores on the PCS were expected with FAMS subscales measuring the more physical aspects of quality of life; lower association was expected between SF-36 PCS scores and FAMS subscales, which are more psychological in nature. The reverse set of expectations applied to the SF-36 MCS scores.

The MDI and HADS are mood questionnaires, so it was expected that higher coefficients of association would be obtained when their scores were compared with the more psychosocial subscales of the FAMS than when compared with the more physical subscales. Logically, then, the overall relationship between the FAMS and these questionnaires was expected to be lower (but still significant) than that with the SF-36. The MDI produces scores summarizing three types of symptoms associated with depression: mood, evaluative, and vegetative. ^[16] The mood score summarizes the affective component of depression; the evaluative and vegetative scores summarize its cognitive and physical components, respectively. The HADS is a brief indicator of two mood states, anxiety and depression, which has the feature of eliminating the physical symptoms associated with mood disorders, particularly depression. ^[17] This is attractive because the physical (vegetative) symptoms of depression are often confounded with chronic illness symptoms such as fatigue, sleep disturbance, and loss of appetite or weight, when other standard depression measures are applied to medical patients. ^[17]

Finally, the MCSDS was included in the assessment packet to demonstrate that what is measured by the FAMS is not correlated with social desirability (the tendency to be motivated by the approval of others). Including a measure of an unrelated concept permits examination of the divergent validity of the FAMS that, from a test validation standpoint, is as important as showing convergent validity (i.e., positive association with measures of related concepts).

The neurologist completed the Kurtzke EDSS ^[12] and the Scripps NRS ^[13] for the clinical sample only. The clinical sample was also handed a second copy of the FAMS and asked to complete and return it in a preposted self-addressed envelope within 3 to 7 days. Frequent telephone contact ensured an 81% (56/69) return rate, providing 56 complete records for clinical sample analyses.

Factor analysis of survey sample data.

Factor analysis was conducted using the System for Statistics (SYSTAT) computer program (Evanston, IL). To investigate the underlying structure of the 88-item version of the FAMS, responses of the survey sample (N = 377) were subjected to a principal components analysis with varimax rotation. In this technique, interrelated questions are extracted from the total pool to form ``subscales'' that depict underlying dimensions or concepts reflected by that collection of questions. Principal components analysis uses ``orthogonal'' factor rotations, thereby treating every factor as independent of all others. This was selected to achieve the simplest reasonable test structure. Using this procedure, factors are extracted singly, and the relationship between each question and the factor is expressed as a correlation coefficient (``factor loading''). Based on previous experience with the FACT Measurement System, a five-factor solution was specified. The procedure extracted five factors with easily identifiable conceptual meaning, accounting for 47.7% of the total variance. By requiring a minimum item-factor loading of 0.40, we retained 63 questions loading significantly on these five factors. The first factor (factor 1; 10.7% of variance) contained nine items and was named ``Mobility.'' The second factor (factor 2; 8.7% of variance) contained nine items and was named ``Symptoms.'' The third factor (factor 3; 14.5% of variance) contained 27 items and was named ``Emotional Well-being.'' The fourth factor (factor 4; 8.9% of variance) contained twelve items and was named ``Thinking/Fatigue.'' The fifth factor (factor 5; 4.9% of variance) contained six items and was named ``Family/Social Well-being.'' Specific factor loadings are on file with NAPS (see ``Note'' on page 137.)

Rasch model scaling.

To further investigate and evaluate the content homogeneity (i.e. unidimensionality) of the obtained factors, the Rasch measurement model was used. The objective was to develop the most efficient (i.e., brief), methodologically appropriate, and conceptually meaningful scales of the five-factor solution. Analyses were performed using the BIGSTEPS ^[19] Rasch analysis computer program. Before beginning Rasch measurement analyses, the third factor (27 items) was divided into two subscales. This was done for both conceptual and practical reasons. It seemed reasonable to reclassify the first eight items in factor 3 (those with the highest loadings) as a separate subscale measuring depression. Many of the other 19 questions appeared more general, tapping issues such as acceptance of one's illness and general contentment. A smaller number of questions (items 8, 72, and 74) appeared to tap more of a social than a primarily psychological concern. They were evaluated with the 19 questions remaining in original factor 3, but they were also added to the six items loading on factor 5 for review. This afforded the opportunity to divide a rather lengthy subscale (factor) into two more clinically distinct and practical measures and to add more questions to the Family/Social Well-being subscale before testing its formal unidimensionality.

Rasch ^[20] described a statistical measurement model to analyze test scores. The dichotomous (right/wrong; yes/no) model has been more fully explained by Wright and Stone. ^[21] Andrich, ^[22] Masters, ^[23] and Wright and Masters ^[24] extended the model to rating scales and other observations embedded in ordered categories, such as the Likert-type responses used in the FAMS (see Appendix 1). The extended model specifies that each test response is an outcome of the probabilistic linear interaction between a person ability measure and a question difficulty measure. It jointly locates each person and question on an interval scale. It also specifies that all questions measure the same trait (i.e., must be unidimensional). Unidimensionality is a basic requirement for measurement because it ensures that all of the items in a scale tend to measure the same thing, allowing us to make meaningful quantitative descriptions about people.

The results of Rasch analyses revealed six psychometrically sound subscales, for which the calibrated items spread out in meaningful directions. Of the 63 items retained in the principal components analysis, 44 were retained after Rasch measurement criteria were applied to eliminate misfitting items. The mean square (MNSQ ^[19]) fit statistics, measuring each item's adherence to the Rasch model restrictions concerning scale unidimensionality, was used to determine how well the items fit together to define the underlying variable on a linear scale. The MNSQ has an expected value of 1.0. Higher values indicate noise in the data. These are identified as misfitting items not practically or conceptually related to the other items in the scale. The final 44 items comprising the six subscales, which are summed to produce a total score, are presented in Appendix 1. (Although 44 of the original 88 FAMS (version 1) items have been retained for scoring, the FAMS (version 2) is 59 items in length, due to retention of 15 unscored items [questions 45 to 59] based on their potential clinical and empirical value.)

The first subscale (``Mobility'') contains seven of the original nine items from factor 1. The second subscale (``Symptoms'') contains seven of the original nine items from factor 2. The third subscale (``Emotional Well-being'') contains seven of the eight items that loaded most highly on factor 3 (i.e., those that measure depression). The fourth subscale (``General Contentment'') contains 7 of the remaining 19 items that loaded on factor 3. The fifth subscale (``Thinking/Fatigue'') contains 9 of the original 12 items from factor 4. The sixth subscale (``Family/Social Well-being'') contains seven items: five of six original items from factor 5 and two of three added from factor 3.

(Table 1) presents descriptive and reliability data on the six FAMS subscales and the 44-item total score. Raw scores can be computed by first reversing (subtracting from 4) the score of all negatively worded questions (e.g., ``I have trouble walking''), so that for all questions, a high score reflects good quality of life. After appropriate reversal, the scores are added within subscale, and then subscale scores are summed to produce a total FAMS score. Thus, each of the five seven-item subscales has a possible score range of 0 to 28, and the nine-item Thinking/Fatigue subscale has a possible score range of 0 to 36. The FAMS total score range is therefore 0 to 176 [5(28)+(1)36]. Table 1 provides raw score means, standard deviations, internal consistency (alpha) reliability, and test-retest reliability. Across the board, the survey and clinical samples are comparable, with standard deviations of the survey group marginally higher in some instances. Means provide a basis for comparison with other patient groups and healthy control subjects. Cronbach's alpha coefficients are universally high (range 0.82 to 0.96), reflecting homogeneity of the item pool within subscales and for the total score. Test-retest reliability coefficients are universally high, with coefficients ranging from 0.85 to 0.91 (see Table 1).

Concurrent and construct validity.

(Table 2 and Table 3) present concurrent and construct validity correlation coefficients for the survey Table 2 and clinical Table 3 samples. The SF-36 component summary scales, the HADS, the MDI subscales, and three disability ratings (PSR, EDSS, NRS) were given along with the FAMS to obtain validity coefficients. In comparing correlations among these variables and FAMS scores, higher coefficients were expected when the comparisons included similar quality of life domains and lower ones when the domains differed. The MCSDS, a measure of social desirability, was given to obtain evidence of divergent validity. No relationship would be expected between quality of life questionnaire and social desirability responses, and none was found (see last row of Table 2 and Table 3).

The survey sample data (see Table 2) revealed very high association between the PCS of the SF-36 and the Mobility FAMS subscale. In addition to the Family/Social Well-being subscale, all other FAMS subscales and its total score correlated moderately with the SF-36 PCS. In contrast, the MCS score correlated best with the Depression Emotional Well-being subscale of the FAMS. Moderate correlations between the SF-36 MCS and the Symptoms, Thinking/Fatigue, and Family Social Well-being subscales and the total FAMS score were relatively high as well, reflecting some degree of relationship among these subscales and mental well-being as measured by the SF-36. Correlation between the MCS and the Mobility subscale was, in contrast to that between the Mobility and the PCS, rather low.

Convergent validity coefficients relating to psychological distress and symptomatology can be found within Table 2 and Table 3. For example, in both samples, the HADS Anxiety scale, HADS Depression scale, MDI Mood scale, and MDI Evaluative scale correlate with the Emotional Well-being and the General Contentment subscales of the FAMS more highly than with any other subscales. In addition, the FAMS Thinking/Fatigue subscale correlated most strongly with the MDI Vegetative subscale in both samples.

The disability ratings (PSR, EDSS, NRS) are all significantly related to the Mobility subscale of the FAMS. This was found in both samples, although only PSR was obtained in the survey sample. Interestingly, there was no significant association between these disability ratings and the Symptom or Thinking/Fatigue subscales. For the clinical sample (see Table 3), associations between these indicators of disability and Emotional Well-being or Family/Social Well-being were rather low, reflecting the measurement of distinct dimensions. For the survey sample (see Table 2), the correlation of Emotional Well-being and General Contentment with PSR was considerably higher, but reflects in part the fact that both pieces of information were obtained from the same source (i.e., the patient).

(Table 4) provides data from both samples on the intersubscale and subscale-total correlations. All subscales are, as expected, intercorrelated, with coefficients of association ranging from 0.11 (Family/Social Well-being with Mobility) to 0.87 (Emotional Well-being with General Contentment). Thus, the range of shared variance across all subscale-subscale correlations is 1 to 80% across both samples. Overall, Family/Social Well-being bears the weakest association with other subscales. With the exception of Family/Social Well-being, subscale-total correlations reflect a relatively high degree of shared variability and yet, as will be seen in the forthcoming tables, very different responsiveness to known group differences.

Criterion-related validity.

Criterion-related validity is demonstrated by the use of FAMS scores to distinguish groups known to differ from one another. The ``known groups difference'' method entails defining the groups according to a clinically meaningful distinction and then determining the extent to which the instrument scores confirm the known distinction. We chose three criteria by which the survey and clinical samples might be divided. The results of the analyses are presented in Table 5 (survey sample) and Table 6 (clinical sample). First, patients were divided into those with relapsing/remitting disease and those with progressive disease. Each group in this breakdown included patients who had been stable for at least 18 months and those who had worsened or had an illness attack within the past 18 months. The second breakdown separated out a subsample of 80 patients (of either relapsing/remitting or progressive type) who had been stable for at least 18 months. In both cases, significant group differentiation was obtained on all the subscales with only one exception. In the survey sample, the relapsing remitting patients were not distinguishable from the progressive patients on the Symptoms subscale. All other FAMS subscales significantly differentiated disease category, demonstrating better quality of life across the board for relapsing/remitting patients than progressive patients. Stable patients (whether relapsing/remitting or progressive) appear also to have consistently superior quality of life than progressive patients and, to a lesser degree, relapsing/remitting patients.

The third criterion for the survey sample was the ECOG PSR, a single-item patient rating of activity level that is highly correlated with the Kurtzke EDSS (r = 0.64) and the Scripps NRS (r = -0.67). Across all subscales of the FAMS, patients who require some degree of bedrest during the day (N = 241) scored worse than those who did not require daytime bedrest (N = 131). Means, standard deviations, and significance levels for all known groups analyses can be found in Table 5.

In the clinical sample, the ECOG PSR, the Kurtzke EDSS, and the Scripps NRS comprise the three criteria for evaluation of FAMS criterion-related validity. The Mobility subscale of the FAMS produced dramatic and highly significant differentiation of groups on all three criterion variables (see Table 6). In contrast, there was little differentiation of groups by any of the other subscales.