Invited Article: Lost in a jungle of evidence: We need a compass
Martin M.Pincus, MD, PhD, 2 East End Ave. #4B, New York, NY 10075[email protected]
Submitted January 16, 2009
Drs. French and Gronseth outline the requirements for grading Class I studies. The AAN classification of evidence scheme (Classes I-IV) is also summarized in the Table. [1]
It is possible that these requirements may increase the risk of bias in a study. As the authors state, random allocation to treatment and control groups is the best way to minimize differences in known and unknown baseline risk factors between the compared groups. If bad luck intervenes and the randomization results in a substantial imbalance in some known risk factors, a Class I designation may be rescued by an appropriate statistical adjustment. However, it cannot be determined how this adjustment affects the balance in unknown risk factors achieved by the randomization. This may then result in other, perhaps worse, imbalances.
For randomized controlled trials with noncompliance, intention-to-treat (ITT) analysis is recommended. In this analysis, the comparison of the randomized groups is preserved. However, a difference in outcome between the compared groups does not prove a true treatment effect of the corresponding treatment. [2] The authors imply that this analysis is conservative and can "bias the study to show no treatment effect" but "if a difference in outcomes is observed it is more likely to be related to a true treatment effect than to confounding differences between treated and comparison groups." However, the opposite bias can occur due to post-randomization events such as treatment withdrawal and crossover.
For example, consider a trial comparing an experimental medication to a standard medication. Those patients withdrawn from the experimental treatment must ethically be offered the standard treatment. The outcomes of these patients are attributed to the experimental group with ITT. If the standard medication is more effective than the experimental medication, then under ITT analysis, the more adverse effects the experimental medication will have, the greater the advantage for the experimental group since these effects increase the probability of treatment withdrawal and of receiving the more effective standard medication. This advantage will be more pronounced the greater the effectiveness of the standard medication. If the number of these post randomization events is "sufficiently low to have minimal potential for bias" (Class 1.d. in Table), then there is little need for ITT analysis.
The admirable movement towards evidence based medicine should not be compromised by guidelines that do not perform their stated functions.
References
1. French J, Gronseth G. Invited article: Lost in a jungle of evidence. Neurology 2008;71:1634-1638.
2. Hennekens CH, Buring JE. Epidemiology in Medicine. Boston, MA: Little, Brown and Co;1987.
Drs. French and Gronseth outline the requirements for grading Class I studies. The AAN classification of evidence scheme (Classes I-IV) is also summarized in the Table. [1]
It is possible that these requirements may increase the risk of bias in a study. As the authors state, random allocation to treatment and control groups is the best way to minimize differences in known and unknown baseline risk factors between the compared groups. If bad luck intervenes and the randomization results in a substantial imbalance in some known risk factors, a Class I designation may be rescued by an appropriate statistical adjustment. However, it cannot be determined how this adjustment affects the balance in unknown risk factors achieved by the randomization. This may then result in other, perhaps worse, imbalances.
For randomized controlled trials with noncompliance, intention-to-treat (ITT) analysis is recommended. In this analysis, the comparison of the randomized groups is preserved. However, a difference in outcome between the compared groups does not prove a true treatment effect of the corresponding treatment. [2] The authors imply that this analysis is conservative and can "bias the study to show no treatment effect" but "if a difference in outcomes is observed it is more likely to be related to a true treatment effect than to confounding differences between treated and comparison groups." However, the opposite bias can occur due to post-randomization events such as treatment withdrawal and crossover.
For example, consider a trial comparing an experimental medication to a standard medication. Those patients withdrawn from the experimental treatment must ethically be offered the standard treatment. The outcomes of these patients are attributed to the experimental group with ITT. If the standard medication is more effective than the experimental medication, then under ITT analysis, the more adverse effects the experimental medication will have, the greater the advantage for the experimental group since these effects increase the probability of treatment withdrawal and of receiving the more effective standard medication. This advantage will be more pronounced the greater the effectiveness of the standard medication. If the number of these post randomization events is "sufficiently low to have minimal potential for bias" (Class 1.d. in Table), then there is little need for ITT analysis.
The admirable movement towards evidence based medicine should not be compromised by guidelines that do not perform their stated functions.
References
1. French J, Gronseth G. Invited article: Lost in a jungle of evidence. Neurology 2008;71:1634-1638.
2. Hennekens CH, Buring JE. Epidemiology in Medicine. Boston, MA: Little, Brown and Co;1987.
Disclosure: The author reports no disclosures.