Abstract
The behavior of the control groups can substantially affect the power and outcome of a clinical trial.We report a meta-analysis of the control groups of four large, double-blind, placebo-controlled clinical trials of immunosuppressive treatment of progressive MS to address the sensitivity of five hypothetical definitions of treatment failure (TF). The rate of TF in the aggregate control groups (n = 427) was 31% when a confirmed increase of 1.0 expanded disability status scale (EDSS) point was required at the end of the trial; it was 51% when confirmation was not required and TF was allowed at the first point where the criteria for TF were met. The rate of confirmed TF was 45% when the TF criteria were indexed to baseline EDSS, accounting for the observed differences in staying times at different EDSS levels. We developed models predicting TF in progressive MS. In addition to baseline EDSS, the pyramidal functional score and, for one definition, brainstem functional score were associated with probability of TF.
NEUROLOGY 1996;46: 1613-1619
Results of large randomized clinical trials of nonspecific immunosuppressive treatments in patients with progressive MS have generally shown modest or no benefit over short-term follow-up of 2 to 3 years. While lack of efficacy of immunosuppressive therapy may be the major explanation, the behavior of the control groups may have contributed substantially to the apparent lack of efficacy. The rate of deterioration of the control groups subsequent to enrollment, most commonly expressed in terms of change in expanded disability status scale (EDSS) scores, [1] has generally been much lower than the pre-enrollment rate of deterioration upon which patients were selected. [2] If one expects continued deterioration at the pre-enrollment rate, the power of the study will be overestimated. Furthermore, many patients experience fluctuations in disability scores, and apparent treatment failure (TF) may not be confirmed at the time of a follow-up visit a short time later. In the recent interferon beta-1b study, 39% of patients had deterioration by 1 point on the EDSS at 3 years after entry, but the endpoint was not confirmed one visit later in 11% of patients. [3] Some of this instability in the endpoint may be due to fluctuations resulting from acute attacks, but the majority is probably due to random error (noise) in the application of the definition of treatment failure. Such "noise'' may result from intrarater variability and fluctuation in patient performance due to patient fatigue, and could obscure a true reduction in the proportion of TFs. This problem might be magnified in trials that use differences in mean EDSS from the beginning to the end of the study as the primary outcome rather than a TF outcome, because mean EDSS is more sensitive to minor changes in EDSS scores, which are more likely due to noise. A TF definition requires a priori definition of the criteria for unequivocal deterioration. The criteria may differ at different baseline levels of disability, given our observation, [4] confirmed by others, [5] of substantial differences in the average times spent at different levels of the DSS, the precursor to the EDSS.
With these considerations in mind, we report a meta-analysis of the placebo-treated control groups of four large studies of progressive MS, published in the last 7 years, each of which enrolled 100 or more patients. Our goal was not to address the efficacy of intervention, but rather to address the behavior of the control groups using various TF endpoints. We set the following goals: (1) To establish a mechanism to predict patients at the highest risk of deterioration, so that the most informative patients might be identified for enrollment, thereby allowing smaller, less costly, and more powerful clinical trials ("predictive enrollment''). (2) To examine several hypothetical definitions of TF in terms of durability, responsiveness, and dependence on baseline EDSS.
Methods.
We obtained the raw data from the principal investigators of four randomized, placebo-controlled clinical trials: The UCLA Azathioprine-Methylprednisolone Trial, [6] The British-Dutch Azathioprine Trial, [7] The USA Cyclosporine A Study, [8] and The Canadian Cooperative Study of Cyclophosphamide and Plasma Exchange [9] Table 1. The trials were chosen because of their large size, inclusion of patients with progressive MS, careful follow-up and data collection, and cooperation of the principal investigators. We established five uniformly applied endpoint definitions based on hypothetical but widely accepted criteria of TF Table 2. These definitions differed according to whether the TF definition was applied at the point that the primary endpoint was first reached (definitions 1 to 3) or only at the end of the follow-up period (definitions 4 and 5), whether confirmation of the primary endpoint was required at two consecutive visits at least 3 months apart (definitions 2, 3, 5), and whether the degree of change in the primary outcome was stratified according to baseline EDSS (definition 3). These definitions were applied to each individual trial and to the aggregate placebo group Table 2. A two-way analysis of variance was performed on the different definitions of TF to study the variability in behavior of the TF definition by study.
Table 1. Selection criteria and baseline demographic and clinical data of placebo control groups
Table 2. Patients meeting treatment failure criteria in placebo groups
Predictors of progression over the course of the trials were identified by studying the association of TF and a number of prognostic factors by chi-square test for categorical variables and t tests for noncategorical variables for the aggregate placebo group with each definition of TF Table 3. We constructed models based on multiple prognostic factors using multiple logistic regression (SAS/LOGISTIC, SAS Institute, Inc., SAS/STAT User's Guide, Version 6, Fourth Edition, Cary NC: SAS Institute Inc. 1989). As a preliminary step, we identified co-associations between pairs of factors with a correlation analysis to identify potentially redundant parameters, followed by a stepwise selection of factors with entry criterion of p < 0.1. Thus, the first factor to enter a model was the factor with the most significant association; the second was that with the highest association when the first was controlled, and so on. Any variable that became nonsignificant after controlling for a prior variable was removed. We performed this analysis for the first three hypothetical definitions of TF. We constructed models based on the data in the clinical trials, utilizing data from the first two trials in the first instance, then from the first three trials, and subsequently from all four clinical trials, validating the results on subsequent trials Table 4. We retrospectively classified patients who were recruited to the trials as being appropriate or inappropriate for inclusion according to the predicted probability of TF for each patient as calculated from the regression equations. We examined the increase in the observed proportion of patients that would have reached the primary endpoint (TF) in patients chosen based on these models (p1) compared with the observed proportion worsening in all patients in the study (p0). We also considered the proportion (r1) of the patients selected for the trials that would be eligible for a clinical trial based on this approach of predictive enrollment. A similar analysis was performed to derive models that predicted time to TF (as opposed to the proportion of patients experiencing TF). These models were derived using SAS/LIFEREG eliminating any factors with p > 0.10 from the full model containing all prognostic factors.
Table 3. Prognostic factors associated with treatment failure
Table 4. Models predicting treatment failure
Results.
Comparison of the trials.
All of these studies were randomized, placebo-controlled, and double-blind. Two had three treatment groups and two had two. All were confined to patients with progressive MS, either primarily progressive or secondarily progressive, except for the British-Dutch Azathioprine study; in this trial, we confined our analysis to 61 patients in the control group with progressive MS. It was not possible to separate primary progressive from secondary progressive MS based on the data available to us. The range of EDSS admission scores varied among the trials. Patients in the EDSS 4.0 to 6.0 range were eligible for all studies, but the range in the aggregate group included patients with baseline EDSS score 3.0 to 7.0. The mean EDSS score was 4.5 to 5.8; patients in the British-Dutch study were significantly less disabled at entry. The follow-up interval was 3 months except for the Canadian study in which it was 6 months. The entry criteria for these studies are shown in Table 1.
Characteristics of the control group.
The demographic characteristics of the control groups are shown in Table 1. A female excess was seen for each study except for the British-Dutch study. The duration of MS was shorter for the US Cyclosporine A study. Mean available follow-up was shortest for the US Cyclosporine A study, which was a 2-year study. The rates of TF according to the definitions considered are shown in Table 2 for each study and for the aggregate control group consisting of 427 patients. Where two neurologists evaluated patients for TF (e.g., the Canadian Cooperative Study), the blinded evaluating neurologist's evaluation was used. The first two quartiles of the time to TF are shown for definitions 1 to 3, which allow for TF at times other than the conclusion of the study. The 25th and 50th percentiles were derived using the product limit method for censored data utilizing data on all subjects; for those whose disease did not progress, time on study was used. In general, definition 1 was the most sensitive, as it is the least restrictive. The most conservative definition, definition 5, requiring a one-point deterioration at the conclusion of follow-up confirmed at two assessments 3 or more months apart, was the least sensitive. The rest were intermediate. Allowing a 0.5 deterioration at EDSS 6 or greater to be considered criterion for TF improved the sensitivity of the definition of TF, almost to that of definition 1 in spite of requiring confirmation. This was true for each of the studies except for the UCLA study, where the DSS rather than the EDSS was used, which made it impossible to evaluate the added sensitivity of using 0.5 point deterioration as criterion for TF at EDSS score 6 or greater.
Association of treatment failure with predictive factors.
We examined the association between prognostic factors and TF for a number of potential prognostic factors, including duration of follow-up, baseline EDSS, involvement of the six Kurtzke functional scores (dichotomized as "involved'' or "not involved'' based on whether the functional score was >or=to2 or <2), duration of MS, age of onset, gender, and years of follow-up (see Table 3). The parameters that we examined were chosen from those that we had found to be associated with the long- and short-term course of MS in previous studies. [4,10] Duration of follow-up was, not surprisingly, highly associated for all definitions. Baseline EDSS was significantly associated with the probability of TF for all definitions except for definition 3, which, by including a less-restrictive definition of worsening for patients with baseline EDSS 6 or greater, essentially removed the dependence of TF on baseline EDSS. Only brainstem involvement among the six Kurtzke functional scores was significantly associated with an increased probability of worsening; this was seen only in the control group and not in the combined intervention group. Surprisingly, a significant effect of age of onset was seen in the combined intervention group only. Treated patients who had TF by any of the five definitions were significantly younger at onset of MS (e.g., 31.6 years versus 34.5 years for definition 1), whereas this difference was not seen in the control group. This suggests that older age of onset may have some bearing on responsiveness to immunosuppressive medication, being somewhat more effective in older than younger patients. Age at onset was negatively associated with duration of MS and with cerebellar, brainstem, and visual involvement, and positively associated with progression index.
Construction of predictive models and assessment of their performance characteristics.
We examined the association between pairs of predictive factors to identify redundant factors might not appear in the same model. Progression index, EDSS, and duration of MS were associated by definition (progression index = EDSS/duration of MS). Pyramidal functional scores were highly correlated with entry EDSS (r = 0.54). Age of onset and duration of MS were negatively correlated (r = -0.44). Brainstem functional scores had moderate positive correlation with cerebellar (r = 0.33) and cerebral (r = 0.26) scores.
The models derived based on control group patients enrolled in all four trials are shown for definitions 2 and 3 in Table 4; the models yield predictions for the probability of TF and for predicted time to TF. Note that the absolute value of the regression coefficients changes from a positive to a negative value and vice versa in the models predicting time to TF rather than probability of TF; this is because a factor that increases the probability of TF decreases the predicted time to TF. Therefore, for definition 2 of TF, the lower the baseline EDSS score, within the range of EDSS scores of patients included in the studies examined, the greater the probability of TF; however, the higher the pyramidal functional score and brainstem functional score, the greater was the probability of TF. Because several of the models included only one discriminating parameter, often a Kurtzke functional score, we used the actual functional scores in constructing the models, rather than dichotomizing functional score involvement based on whether the score was >or=to2 or <2 as described above. To assess the performance of these models, patients were retrospectively classified as being appropriate or inappropriate for recruitment based on the calculated probability of TF for the regression equations. We considered the proportion of patients chosen for the trial that would have been enrolled in a hypothetical predictive enrollment trial (r1) and the proportion of those patients that would have met the definition of TF (p1) compared with the proportion actually observed to have TF in the entire sample (p0). Patients with a calculated probability of TF > 0.3 (definition 2) or >0.4 (definition 3) according to the models were enrolled. The models would have resulted in a 29% (definition 2) and a 4% (definition 3) increase in the proportion of observed TFs, while reducing the proportion eligible for enrollment by 49% (definition 2) and 20% (definition 3).
We compared the actual times to TF in patients whose treatment was predicted to fail and those predicted not to fail using a dichotomous definition according to whether the predicted time was greater than or less than the mean follow-up for the patients in the analysis (2.3 years). This arbitrary cutoff was chosen to optimize the difference between p1 and p0 without unduly lowering the proportion of eligible patients (r1). The differences in mean observed time to TF were significant for both definitions 2 and 3. Using this method of choosing patients based on time to TF, model 3 resulted in 25% of the patients being eligible for the trial.
To address the question of whether this approach would have changed the outcome of a clinical trial, we simulated a predictive enrollment trial using the patients enrolled in the Canadian Cooperative Trial of Cyclophosphamide and Plasma Exchange, which showed no significant treatment effect in the treated patients according to definition 5 for TF. We considered definitions 2 and 3 of TF to determine outcome. There were two active treatment groups: IV (treated with IV cyclophosphamide and oral prednisone) and PLEX (treated with oral cyclophosphamide, plasma exchange, and oral prednisone). The placebo control group received sham plasma exchange and placebos instead of oral cyclophosphamide and prednisone. The data are shown in Table 5. The simulated predictive enrollment trial was substantially smaller (0.39 times the sample size) when definition 2 was used to define TF. The proportion of treatment failures was increased by 23% in the hypothetical predictive enrollment trial using definition 2 for treatment failure. The p value (log rank statistic) was not different, however, for the distribution of survival in the three groups for all patients enrolled or for those eligible for the predictive enrollment trial. The model using definition 3 of TF included only one discriminating parameter (pyramidal functional score involvement), and, as a result, the model did not substantially refine patient selection.
Table 5. Simulated predictive enrollment on the Canadian Cooperative Trial (actual/PE)*
Stability of treatment failure definition.
Of 929 patients (combined treatment and control groups), 438 had an increase of one or more EDSS or DSS points from baseline (definition 2 of TF). In 915 patients where confirmation was possible (i.e., the deterioration or improvement did not occur at the last visit), 265 were confirmed (sustained at >or=to3 months). Therefore, 61.4% of observed deteriorations of >or=to1.0 point were sustained. Two hundred nine had improvements of one or more EDSS points from baseline, with 105 (51.0%) being sustained. The proportion of increases in baseline EDSS by >or=to1.0 point and the proportion of these confirmed at a subsequent visit stratified according to baseline EDSS score is shown in Table 6. Sixty-two percent of patients had baseline EDSS scores of 5.5 to 6.5 at entry. Twenty percent of patients in this category experienced a confirmed increase compared with 44% of patients at all other entry EDSS levels. Confirmation of an increase of 1.0 EDSS points is also somewhat less frequent in patients enrolled with baseline EDSS 5.5 to 6.5.
Table 6. Proportion of TF according to baseline EDSS
The maximum increase in EDSS from baseline entry scores observed during the course of the trials per individual is shown in Figure 1.
Figure 1. Maximum increases in Expanded disability status scale from entry in the aggregate control group.
Discussion.
Phase III clinical trials in MS are lengthy, costly, and labor-intensive undertakings. For these reasons, it is critical to choose not only a relevant endpoint but one that can be used to discriminate between the intervention and control groups, assuming that the treatment is efficacious. Accordingly, an endpoint must be: clinically relevant; sensitive or responsive; stable and not prone to fluctuations or "noise.'' A phenomenon observed in many MS clinical trials is that the behavior of control groups is generally more favorable than their pre-enrollment behavior. This phenomenon is, in part, due to placebo effect and in part due to regression to the mean, a tendency for a group to return to the average, rather than to sustain the higher-than-average rate of deterioration that led to its selection. [2,4]
The effect of the observed event rate in the placebo group on sample size can be illustrated by the following example. If one sought to show a 30% reduction in a baseline of event rate of 40%, 325 patients are required for a parallel design control trial, whereas to show a 30% reduction with a baseline event of 50%, 226 patients, almost a third fewer patients, are required (SAS PROC/LIFEREG). To decrease the sample size, one should identify patients at high risk to meet the TF criterion, a method we call "predictive enrollment.'' We initially reasoned that models developed to characterize the long-term behavior of MS by predicting the time from onset of MS to reach EDSS 6 [10] might be valuable in this regard. However, when we studied the association of predictions of these models with short-term deterioration over 2 years in a regional population of MS patients, the association was weak. [11] The biologic determinants and the clinical and demographic associations of the long-term and short-term natural history of MS are different. [12] The strongest predictors of short-term deterioration were baseline EDSS and baseline pyramidal functional scores. In this study, we examined predictors of short-term deterioration in a more highly targeted and selected population, namely patients selected for clinical trials of progressive MS.
A TF definition is theoretically superior to the difference in mean EDSS score as a primary endpoint for a clinical trial in progressive MS. Mean EDSS is influenced by patients with minor changes in EDSS (<or=to0.5 points) and by patients whose condition improves in the course of a trial. Eleven percent of 929 patients in the aggregate patient group enrolled in the four trials improved by >or=to1.0 EDSS point, confirmed at a second visit. When the hypothesis being tested in a clinical trial of an immunosuppressive drug in progressive MS is that intervention prevents deterioration, unexpected improvement, which is not interpretable in the context of the study hypothesis, constitutes "noise'' that obscures the primary endpoint. A TF definition that predefines the degree of worsening that one will accept as unequivocal evidence of clinical deterioration is a superior endpoint to address the study hypothesis. Time to TF might be expected to add additional power as a primary or co-primary outcome. A definition that allows TF at any point in the trial that worsening is observed and confirmed deals with the practical problem of avoiding dropouts before the primary endpoint of the trial is reached. Patients who withdraw after the primary endpoint is reached (for example, to try another new or experimental treatment) have a minor negative impact on such a study. However, dropouts may have a disastrous impact in studies where TF is permitted only at the time of last scheduled follow-up.
Others have suggested that patients who are younger and have secondarily progressive rather than primarily progressive MS may be more likely to have a response to immunosuppressive treatment than older patients with primary progressive MS. [13] We found no association of age of onset with TF in the control group, but found that older age of onset was associated with a lesser probability of treatment failure, regardless of definition, in the combined intervention group. This suggests that older age of onset within the range considered in these trials was associated with a favorable response to the immunosuppressive agents studied. This observation must be confirmed prospectively. Based on the data available to us, we were unable to address potential differences between patients with primarily progressive and secondarily progressive MS.
The proportions of patients in the control groups experiencing TF range from 31% to 51% depending on the definition considered. Because they require confirmation of TF but allow for TF during the course of a trial, we prefer definitions 2 and 3. Definition 3 allows for TF if there is an increase by 0.5 EDSS points at EDSS scores >or=to6.0. The latter definition was based on our previous observations, [3] confirmed by others, [4] that "staying times'' at DSS 6 and 7 were substantially longer than at DSS 3, 4, and 5. We described the proportion of patients in the aggregate group experiencing one or more confirmed increases by >or=to1.0 EDSS points stratified according to baseline EDSS. Most patients enrolled in the clinical trials we considered were at EDSS levels 5.5 to 6.5 at entry. In these patients, a much lower proportion experienced a deterioration of >or=to1.0 EDSS point; also, in a lower percentage of those who did have deterioration was the worsening confirmed. Goodkin et al., [14] in a controlled clinical trial of azathioprine, utilized a stratified approach to the definition of TF, wherein a 0.5 EDSS point deterioration was considered as satisfying TF if the baseline EDSS score was >or=to5.5.
Our models provide a rational basis for predictive enrollment. Using this strategy, the proportion of patients enrolled from those screened for entry can be reduced, thereby saving cost, without sacrificing power. This strategy might also be used to shorten the time of a clinical trial where the primary endpoint is time to TF. The proportion of patients experiencing TF in our hypothetical predictive enrollment scheme was increased up to 35% compared with patients enrolled in the actual trial. Elsewhere, we discuss methods of identifying viable predictive enrollment schemes based on the ratio of cost of follow-up and cost of recruitment. [15]
Several of our models are weak; some have few parameters that discriminate among the risk of TF in different patients considered for these studies. However, we studied only patients satisfying the entry criteria of the studies and possibly other selection biases exerted by the investigators that favored high risk of progression. Our models had to perform better than the investigator's judgment to be useful. The impact of predictive enrollment might have been greater had we included all potentially eligible patients who had been included in deriving these models. Other modalities such as MRI lesion volume, rate of new lesion development, or other indices might potentially be used in future predictive enrollment paradigms for patient selection. [16]
Acknowledgments
The investigators acknowledge the cooperation and help of the following in providing data from their studies: Dr. J. Wolinsky and Sandoz Research Institute (US Cyclosporine A Trial), Dr. R.A. Hughes (British Dutch Azathioprine Trial), Dr. G. Ellison (UCLA Azathioprine Study), and Dr. J. Noseworthy (Canadian Cooperative Study of Cyclophosphamide and Plasma Exchange). The investigators acknowledge the cooperation of Dr. W. Tourtellotte and K. Syndulko. Dr. G. Ebers provided useful advice and guidance. R. Tyler performed data processing. Theresa Hanson typed the manuscript.
- Copyright 1996 by Advanstar Communications Inc.
Disputes & Debates: Rapid online correspondence
REQUIREMENTS
If you are uploading a letter concerning an article:
You must have updated your disclosures within six months: http://submit.neurology.org
Your co-authors must send a completed Publishing Agreement Form to Neurology Staff (not necessary for the lead/corresponding author as the form below will suffice) before you upload your comment.
If you are responding to a comment that was written about an article you originally authored:
You (and co-authors) do not need to fill out forms or check disclosures as author forms are still valid
and apply to letter.
Submission specifications:
- Submissions must be < 200 words with < 5 references. Reference 1 must be the article on which you are commenting.
- Submissions should not have more than 5 authors. (Exception: original author replies can include all original authors of the article)
- Submit only on articles published within 6 months of issue date.
- Do not be redundant. Read any comments already posted on the article prior to submission.
- Submitted comments are subject to editing and editor review prior to posting.