Abstract
Objectives: To 1) develop a short instrument (Stroke Impact Scale–16 [SIS-16]) to assess physical function in patients with stroke at approximately 1 to 3 months poststroke using items from the composite physical domain of the Stroke Impact Scale (SIS) version 3.0, and 2) compare the SIS-16 and a commonly used disability measure, the Barthel Index (BI), in terms of their ability to discriminate disability.
Methods: A total of 621 subjects enrolled in the GAIN Americas randomized stroke trial were included in this study. Rasch analysis, which models the probability of a subject’s response to an item using both subject ability and item difficulty, was used to construct the SIS-16, describe its properties, and compare its ordering and range of item difficulties to those of the BI. Box plots and analysis of variance were used to examine differences in BI and SIS-16 scores across modified Rankin categories.
Results: The study sample had an average age of 68 ± 12.4 years and 56% were men. Stroke diagnoses were classified as minor in 91 patients (NIH Stroke Scale score [NIHSS] 0 to 5), moderate in 304 (NIHSS 6 to 13), and major in 226 (NIHSS ≥ 14). Twelve of the original 28 items in the SIS version 3.0 composite physical domain were eliminated to produce the SIS-16, with a minimal loss of reliability. As compared to the BI, the SIS-16 contains more difficult items that can differentiate patients with less severe limitations, and therefore has less pronounced ceiling effects. SIS-16 scores were significantly different across Rankin levels 0 to 1, 2, 3, 4, and 5, whereas BI was significantly different only across Rankin levels 0 to 2, 3, 4, and 5.
Conclusion: Compared to the BI, the SIS-16 is an excellent collection of items suitable for assessing a wide range of physical function limitations of patients with stroke at 1 to 3 months poststroke. Because of a less pronounced ceiling effect, the SIS-16 can differentiate lower levels of disability as compared to the BI.
Although physical function limitations following stroke have a major impact on quality of life and contribute to the economic burden of the disease,1,2⇓ none of the currently used measurement instruments are sensitive across the entire continuum of stroke severity.3-5⇓⇓ The most common measure of stroke-related physical disability is the Barthel Index (BI),6 which has ceiling effects where patients with the best score nevertheless have substantial disabilities.7-8⇓ The physical domain of the Short Form-36 (SF-36) has also been used to measure physical function after stroke.9,10⇓ In contrast to the BI, the SF-36 has floor effects where patients with the lowest score may have further clinical deterioration. Although different instruments could be used for patients at different levels of severity, there are benefits to having a single instrument.
The Stroke Impact Scale (SIS) is a new comprehensive and psychometrically robust stroke-specific outcome measure,4-11⇓⇓⇓⇓⇓⇓⇓ which was developed to extend the range of function measured by the BI and the physical domain of the SF-36. The SIS was developed from the perspective of patients, caregivers, and health professionals with stroke expertise and consists of items measuring eight domains (strength, hand function, activities of daily living/instrumental activities of daily living, mobility, communication, emotion, memory and thinking, and participation).4
The 28 items in the first four domains are combined to produce a composite physical domain, but all 28 items may not be needed for effective patient assessment. Item redundancy unnecessarily extends the time required for patient assessment, thereby placing an increased burden on patients, clinicians, and clinical researchers. We sought to reduce the number of SIS items without sacrificing reliability. The specific objectives were to 1) remove items from the previously developed SIS version 3.0 composite physical domain to develop a shorter version (SIS-16) to be used approximately 1 to 3 months poststroke, and 2) compare the SIS-16 and the BI in terms of floor and ceiling effects and ability to discriminate among individuals with differing levels of poststroke disability.
Methods.
Subjects.
Subjects were participants in the Glycine Antagonist in Neuroprotection (GAIN) Americas Trial,12 a multicenter, randomized, double-blind, placebo-controlled trial of a neuroprotective agent, gavestinel (GV150526), in patients with acute stroke. Patients with a predefined level of limb weakness and functional independence before stroke were enrolled. The complete eligibility criteria are outlined in Appendix 1. A total of 1,605 individuals with ischemic or hemorrhagic stroke were randomized and treated at 132 hospital centers in the United States and Canada. The study was already underway when the SIS substudy was instituted; therefore, not all patients were able to participate in the SIS assessments. The results of the main trial conclusively demonstrated that the investigational drug did not improve 3-month functional outcome on the BI among patients with ischemic stroke.12 Not only were the two groups not significantly different, but also the magnitude of the difference was very small. Similar results were observed when SIS was used as an outcome (P. Ordonneau, GAIN Americas Investigators, private communication, 2001). Therefore, in the current study, the active treatment and placebo groups were pooled. Before administration of the SIS, the patient was asked to follow a three-step command (e.g., take this piece of paper in your hand, fold it in half, and put it in your lap), intended to identify patients with significant cognitive impairment or major global or receptive aphasia. Stroke survivors who were not able to respond to the SIS and required a proxy to respond were also included in the study sample. The validity of proxy responses to the SIS has been established in a previous study.13 A total of 621 stroke survivors constituted the current study sample, for whom the BI, modified Rankin score, and a SIS assessment were available at 3 months.
Assessments.
Baseline assessments reported in this study include demographics, initial stroke severity by the NIH Stroke Scale (NIHSS),14 stroke type (ischemic or hemorrhagic), and stroke subtype by the Oxfordshire Community Survey Project (OCSP) classification.15 Evaluations include 1 and 3 months poststroke BI6 and SIS version 2.0 assessments, as well as the 3-month modified Rankin Scale score.16 Study personnel were trained and certified in the use of the NIHSS scale through video training procedures. Guidelines for administering the BI and the Rankin were reviewed at prestudy training meetings and standardized worksheets were used for all assessments. The SIS version 2.0 was administered during an in-person interview with the patient. For the purposes of this study we only included 59 items of the original 64 items of version 2. The 59 items comprise SIS version 3.0.
Measures.
The NIHSS measures stroke-related impairments including level of consciousness, ability to respond to questions, ability to follow simple commands, deviation of gaze, hemianopsia, facial palsy, limb movements of paretic and nonparetic limbs, limb ataxia, sensory loss, neglect, dysarthria, and aphasia.14 Each item is rated on a 3- to 5-point ordinal scale, with lower scores indicating less impairment.14 In the GAIN trial, study subjects were classified as having mild (NIHSS 0 to 5), moderate (NIHSS 6 to 13), or severe (NIHSS ≥ 14) strokes.12 The BI includes 10 items that can be divided into self-care (feeding, grooming, bathing, dressing, bowel and bladder continence, and toileting) and mobility (ambulation, transfers, and stair climbing).6 The patient is rated according to the level of independence with which he or she can perform each of the activities. An ordinal scale from 0 to 100 is used, with 100 indicating that the patient performs these activities without any assistance.6 The modified Rankin Scale is a global outcomes rating scale for patients with stroke.16 The scale includes 6 grades, from 0 to 5 (Appendix 2), with 0 indicating no disability. The SIS Version 3.011 includes 59 items that assess eight domains: strength, memory and thinking, emotion, communication, hand function, activities of daily living (ADL) and instrumental ADL (IADL), mobility, and social participation. Four of the domains (strength, hand function, ADL/IADL, and mobility) were aggregated to produce a composite physical domain.4 In contrast to the Barthel, the SIS strength items are rated in terms of amount of strength. Memory, communication, ADL/IADL, mobility, and hand function items are rated in terms of amount of difficulty, and the emotion and social participation items are rated in terms of frequency. In the current analysis, we consider only the 28 items that comprise the composite physical domain of the SIS version 3.0.
Analysis.
Descriptive statistics were used to summarize demographics, stroke severity, and other characteristics. Rasch analysis,17 which focuses on individual items in an instrument rather than the aggregate score and models a subject’s response based on subject ability and item difficulty, was conducted using Winsteps, version 3.31.18 Rasch analysis was used to 1) select a minimal subset of items from the SIS version 3.0 composite physical domain to develop a shorter version (SIS-16), and 2) describe the desirable characteristics of the SIS-16. A one-parameter rating scale model was employed. Data from both time periods were combined in a single analysis to develop an instrument that could be used at both 1 and 3 months. In order to accomplish this while satisfying the statistical requirement of independent patient responses, we selected 311 1-month SIS assessments from the total sample, and 310 3-month assessments from the remaining subjects to combine and form the study sample. The first group consisted of a random sample of 311 subjects from 563 with both 1- and 3-month assessments. The second group consisted of the remaining 310 subjects, including 58 subjects with only the 3-month assessment and 252 subjects with both assessments who were not included in the first group. This sampling strategy was employed to maximize the use of available data while preserving the independence of patient responses. The approximately equal proportions of 1-month and 3-month assessments reflect the low stroke mortality rate between 1 and 3 months poststroke, and provide a collection of assessments reasonably representative of those seen in the 1- to 3-month poststroke population.
The same indicators as in a previous study11 were used to evaluate instrument quality: 1) fit or the extent to which the items in an instrument measure a single construct (in this case, the level of physical functioning), 2) item difficulty or the ordering of tasks from easiest to most difficult to perform, 3) targeting or the extent to which the set of items is of appropriate difficulty for patients approximately 1 to 3 months poststroke, and 4) separation or the extent to which the items distinguish distinct levels of functioning in patients with stroke. To evaluate fit, the extent to which individual ratings deviated from expectation or misfit was measured using the infit mean square statistic, where a value greater than 1.3 was considered a misfit.19 To evaluate the extent to which an instrument could distinguish those with different levels of physical function, the person separation statistic, where a value of 2.0 is comparable to a reliability of 0.80 and therefore acceptable, was used. A more complete description of the Rasch model and these techniques exists elsewhere in the literature.19,20⇓
In selecting items for the short version of the composite physical domain, the fit statistics were used to identify misfitting items, and the order of item difficulties of the SIS composite physical domain was used to identify those items with the same level of difficulty. A series of analyses was performed, deleting a few items at a time based on the above criteria to identify a minimal set of items that maximized spread in item difficulty and minimized error of measurement and therefore resulted in a minimal reduction of reliability in patient assessments conducted using the resulting instrument as compared to the original version. Content considerations are also paramount in selecting items to be included in the short version. The desire to develop a short scale that measures not only mobility, but also upper extremity function, self-care, and IADL, guided the stopping point of item removal. The short version of the composite physical domain that was derived below included 16 items and was subsequently named the SIS-16. Items from the BI and SIS-16 were combined and reanalyzed to examine the order and level of item difficulties between the two instruments to understand the extent of ceiling and floor effects found in each.
The random sampling of subjects from the available cohort, although necessary in the current study to avoid complex analytical issues, introduces a second level of sampling error. If sampling is not done appropriately, a mathematical possibility exists that a different investigator using the same data may conclude different results owing to this second level of sampling error. In order to evaluate its effect on the results of the current study, three additional random samples, constructed using the same method, were analyzed in terms of the SIS-16. The results were compared to those obtained from the original sample.
Finally, box plots were used to visually examine differences in the distributions of aggregate BI and SIS-16 domain score across Rankin levels, using only those with a 3-month modified Rankin assessment. To examine the concurrent validity of the SIS-16, analysis of variance (ANOVA) with Tukey-Kramer corrections for multiple comparisons was undertaken to quantitatively examine the differences in mean scores for the Barthel vs the SIS-16.
Results.
A total of 621 patients completed the SIS, modified Rankin, and BI at 3-month assessment, with 563 completing both the 1-month and the 3-month, and 58 completing only the 3-month. Table 1 summarizes the demographics and stroke characteristics of the subjects at baseline.
Table 1 Demographic and baseline characteristics
In the development of the SIS-16, the content consideration was that measures of both upper and lower extremity functions, mobility, self-care, and ADL be included. Twelve repeated analyses were needed to select a minimal subset of items that met the content considerations, had good reliability, and had only a very small amount of misfit. A summary of the item selection process is presented in table 2.
Table 2 Summary of item reduction from the original instrument
The first several items to be removed based on misfit—pick up a dime, clip toenails, turn a doorknob, tie a shoelace, grip strength, open can or jar, cut food with knife and fork, and arm strength—were those primarily measuring upper extremity function and strength. All items measuring upper extremity except carry heavy objects misfit when included in an abbreviated subset. Most likely, this is because the number of items related to mobility, self-care, and ADL is high, compared to the fewer number of upper extremity items in the full scale. This has the effect of tilting the construct we are attempting to measure toward mobility, self-care, and ADL. Use of hands, however, is required in two remaining ADL/IADL items (bathe self and dress upper body) in the final version, thereby increasing the number of items that measure upper extremity function to three. The next group of items deleted included climb several flights of stairs, leg strength, foot and ankle strength, and light household tasks. These items were deleted because there were other items in the instrument with approximately the same level of difficulty. At this stage, all four of the strength items had been removed. Because responses to these items are made in terms of the amount of strength, compared to the degree of difficulty in other items, deleting them resulted in the additional advantage of simplifying responding to the shorter version. An optimal set of items consists of seven ADL/IADL items, eight mobility items, and one hand function item (Appendix 3), and subsequently was named the SIS-16.
The person separation reliability decreased slightly with the reduction in the number of items, from a value of 0.95 to a value of 0.94. Considering that the SIS-16 is much shorter than the original instrument, the decrease is minor and this level of reliability is still considered excellent. The person separation index (the ratio of spread in item difficulties to the error in estimating them) for this sample (3.82) indicates that the scores on the SIS-16 can distinguish at least four different levels (strata) of patient physical function (high, above average, below average, and low). The item separation index increased from 15.76 to 19.54, indicating that the items cover a wide range of difficulty. The results of the Rasch analysis of SIS-16 conducted using the three additional random samples showed essentially the same results, with very little variation among them.
A combined map of the item ordering for the BI and the SIS-16 (figure 1) demonstrates that the SIS-16 contains more difficult items and has a greater spread of item difficulties than the BI. It also shows both instruments adequately measuring lower levels of functional ability, but only the SIS-16 adequately measuring higher levels of ability. The ordering of SIS-16 items follows clinical expectation whereas the ordering of Barthel items at the harder end is counterintuitive. In the BI, bathing self appears to be a more difficult task than using stairs, confirming a finding by Tennant et al.21 This difference could reflect the influence of dichotomous scoring of the Barthel bathing self item, in which one would need to require no help of any kind to receive the higher rating, compared to the five-level scoring for the using stairs item. Despite the differences in rating scales across instruments, items measuring dressing and using the toilet have comparable estimates of item difficulty. Differences are found in some items, where the Barthel bathing self item is more difficult than the SIS-16 version, whereas the SIS-16 stair climbing and move from bed to chair items are more difficult than in the Barthel version.
Figure 1. Ordering of item difficulties for Stroke Impact Scale–16 and Barthel Index.
Another indication of the smaller ceiling effects in the SIS-16 is presented in table 3. This table shows that the percentage of individuals who scored the highest possible score (ceiling effect) at both 1 and 3 months poststroke on the BI was much higher than for the SIS-16. The percentage who scored the lowest possible score (floor effect) was also slightly higher for the BI, compared to the SIS-16.
Table 3 Floor and ceiling effects poststroke of those with Barthel Index (BI) and Stroke Impact Scale (SIS) assessments
Finally, plots of Barthel and SIS-16 scores for each Rankin level show the increased sensitivity of the SIS-16 over the Barthel at lower levels of disability. The plot of the distributions of Barthel scores for each modified Rankin score (figure 2) reveals that the BI discriminates among the more disabled groups (Rankin 3 to 5) but is insensitive to differences among the less impaired (Rankin 0 to 2). In comparison, the SIS-16 scores discriminate differences in physical function for all levels of stroke severity (figure 3). The means and standard deviations of 3-month BI and SIS-16 scores by modified Rankin score are presented in table 4. Results from Tukey-Kramer corrections for multiple comparisons showed that no significant differences in 3-month Barthel scores were observed among modified Rankin score 0, 1, and 2 (p > 0.25); however, among modified Rankin score 2, 3, 4, and 5, the 3-month Barthel scores were significantly different from each other (p < 0.0001). The mean scores for SIS-16 were marginally significantly different between modified Rankin score 0 and 1 (p = 0.0886); they were significantly different across modified Rankin score categories 1, 2, 3, 4, and 5 (p < 0.0001). Therefore, in contrast to the BI, SIS-16 can differentiate between those with no significant disability and slight disability.
Figure 2. Barthel Index vs modified Rankin category.
Figure 3. Stroke Impact Scale–16 vs modified Rankin category.
Table 4 Comparison of Barthel Index (BI) and Stroke Impact Scale–16 (SIS-16) across the 3-month modified Rankin categories
Discussion.
In this study, Rasch analysis was used to identify 12 redundant or unnecessary items in the composite physical domain of the SIS version 3.0. The resulting SIS-16 demonstrated good instrument reliability and concurrent validity. It expands the range of physical function assessment beyond the ADL measured by the BI by including more complex activities such as climbing a flight of stairs, walking one block, going shopping, walking fast, carrying heavy objects, and doing heavy household chores. Unlike the BI, the SIS-16 does not suffer from ceiling effects, and it discriminates across all levels of stroke disability, whereas the BI only discriminates across the more disabled levels.
Stroke is a major public health problem. Outcome scales that adequately measure the broad range of functional limitations experienced by patients are required. The BI is not sensitive to physical limitations resulting from milder strokes. It was originally developed to assess performance in chronically, severely ill patients before and after treatment, and to indicate the amount of nursing care needed.22 In contrast, the SIS-16 is designed specifically to measure stroke-related deficits and better captures the broad range of poststroke physical limitations.4 As such, it is more useful in monitoring improvement in function over time in this population.
This study has important implications for the selection of outcome measures for clinical stroke trials. Investigators must consider the impairment severity of the stroke population they are enrolling. If the study enrolls individuals with mild and moderate stroke, at 3 months of the stroke up to half (48% in this study) of the surviving patients may be graded as independent if assessed using the BI and a cutoff of 95. In such a population, the magnitude of the BI difference representing an effect of the intervention will be decreased. The SIS-16 does not exhibit these limitations. This study has demonstrated that the SIS-16 can detect residual impairments even in those who present with mild strokes. Such impairments can have a great impact on an individual’s ability to function without assistance, and will often prevent the patient from returning to his or her full prestroke activity level.6
The SIS-16 measures a single domain of health—the physical domain. However, stroke survivors are often impaired in many other domains: communication, memory and thinking, emotion, and participation in social and physical roles. To capture these aspects of quality of life, an instrument that measures performance in all of these domains is needed. In such situations, the SIS version 3.0 is a more appropriate instrument. However, if the intent is to screen for or monitor physical limitations over time, the SIS-16 is an appropriate instrument that is short, easy to administer, and psychometrically sound.
Appendix 1: Inclusion and exclusion criteria
Inclusion criteria
-
Aged 18 years or older.
-
Symptoms consistent with acute stroke and present at the time of study treatment.
-
Treatment can be initiated within 6 hours of symptom onset.
-
Limb weakness present. If both arm and leg are affected, there must be drift within 10 seconds for the arm and 5 seconds for the leg; if only one limb is affected, the limb must touch the bed within 10 seconds for the arm or 5 seconds for the leg.
-
Previously independent (modified Rankin score ≤1).
-
Written informed consent given by the subject or a legally authorized representative.
-
Women must be of nonchildbearing potential, or if of childbearing potential, with negative pregnancy test at screen and confirmation of adequate contraception use.
Exclusion criteria
-
Obtunded, or responding only with reflex motor or autonomic effects, or totally unresponsive, flaccid, or areflexic.
-
Symptoms rapidly improving and likely to resolve completely within 24 hours.
-
Diagnosis or suspicion of subarachnoid hemorrhage.
-
Known serious life-threatening illness likely to lead to death in the next 3 months.
-
Symptoms consistent with severe congestive heart failure.
-
Presence of malignant hypertension.
-
Known history of significant renal impairment.
-
Known history of significant hepatic disease.
-
Participation in a clinical trial with an investigational drug or internal device within the past 3 months.
-
Previous treatment with GV150526.
-
Unlikely to be available for follow-up.
Appendix 2: Modified Rankin Scale
0 No symptoms at all
1 No significant disability despite symptoms; able to carry out all usual duties and activities
2 Slight disability; unable to carry out all previous activities, but able to look after own affairs without assistance
3 Moderate disability requiring some help, but able to walk without assistance
4 Moderate severe disability; unable to walk without assistance and unable to attend to own bodily needs without assistance
5 Severe disability; bedridden, incontinent, and requiring constant nursing care and attention
Appendix 3: The Stroke Impact Scale–16 (SIS-16) questionnaire
In the past 2 weeks, how difficult was it to:
-
a. Dress the top part of your body?
-
b. Bathe yourself?
-
c. Get to the toilet on time?
-
d. Control your bladder (not have an accident)?
-
e. Control your bowels (not have an accident)?
-
f. Stand without losing balance?
-
g. Go shopping?
-
h. Do heavy household chores (e.g. vacuum, laundry, or yard work)?
-
i. Stay sitting without losing your balance?
-
j. Walk without losing your balance?
-
k. Move from a bed to a chair?
-
l. Walk fast?
-
m. Climb one flight of stairs?
-
n. Walk one block?
-
o. Get in and out of a car?
-
p. Carry heavy objects (e.g., bag of groceries) with your affected hand?
1 = Could not do at all; 2 = very difficult; 3 = somewhat difficult; 4 = a little difficult; 5 = not difficult at all.
Acknowledgments
Supported by funds from Glaxo-Wellcome Inc., the University of Kansas Medical Center Claude D. Pepper Older Americans Independence Center (AG-96–003), and the American Heart Association Pharmaceutical Roundtable for Outcomes Research. Financial and material support for this study (protocol GLYA3002) was provided by Glaxo-Wellcome, Inc.
Acknowledgment
The authors thank the many patients who participated in this study and the GAIN Americas Investigators.12
Footnotes
-
Additional material related to this article can be found on the Neurology Web site. Go to www.neurology.org and scroll down the Table of Contents for the January 28 issue to find the title link for this article.
- Received September 6, 2001.
- Accepted August 22, 2002.
References
Letters: Rapid online correspondence
REQUIREMENTS
You must ensure that your Disclosures have been updated within the previous six months. Please go to our Submission Site to add or update your Disclosure information.
Your co-authors must send a completed Publishing Agreement Form to Neurology Staff (not necessary for the lead/corresponding author as the form below will suffice) before you upload your comment.
If you are responding to a comment that was written about an article you originally authored:
You (and co-authors) do not need to fill out forms or check disclosures as author forms are still valid
and apply to letter.
Submission specifications:
- Submissions must be < 200 words with < 5 references. Reference 1 must be the article on which you are commenting.
- Submissions should not have more than 5 authors. (Exception: original author replies can include all original authors of the article)
- Submit only on articles published within 6 months of issue date.
- Do not be redundant. Read any comments already posted on the article prior to submission.
- Submitted comments are subject to editing and editor review prior to posting.
