Abstract
Objective: To study the accuracy of the ABCD2 score in predicting early stroke risk following TIA and to model post-test probability of stroke for varying cutoff scores and baseline stroke risk.
Methods: Medline, PubMed, Embase, conference proceedings, and manuscript references up to October 2010 were searched for studies reporting ABCD2 score and stroke outcome after TIA. Additional data were requested from authors. Meta-analysis, meta-regression, and post-test probability modeling were undertaken to assess prediction of stroke at 2, 7, and 90 days.
Results: Of 44 eligible studies, data were available for 33 (16,070 patients): 26/33 reported stroke at 2 days (533 strokes), 32/33 at 7 days (781 strokes), and 28/33 at 90 days (1,028 strokes) after TIA. Using scores 0–3 (“low risk”) and 4–7 (“high risk”) for stroke at 7 days, pooled measures were sensitivity 0.89 (0.87–0.91), specificity 0.34 (0.33–0.35), positive predictive value 0.08 (0.07–0.09), negative predictive value 0.98 (0.98–0.98), positive likelihood ratio (PLR) 1.43 (1.33–1.54), negative likelihood ratio (NLR) 0.40 (0.33–0.50), and area under the curve (AUC) 0.70 (0.62–0.78). Results were similar at days 2 and 90. There was moderate heterogeneity while pooling PLR (p < 0.01, I2 >50%), with stroke specialist TIA diagnosis associated with slightly higher PLR. At 5% baseline stroke risk, ABCD2 >3 indicated an absolute increase in 7-day stroke risk of only 2.0% while a score ≤3 indicated a 2.9% decrease in risk. Changes in risk were very small when baseline stroke risk was lower.
Conclusions: The ABCD2 score leads to only small revisions of baseline stroke risk particularly in settings of very low baseline risk and when used by nonspecialists.
GLOSSARY
- AUC=
- area under the curve;
- IQR=
- interquartile range;
- NLR=
- negative likelihood ratio;
- NPV=
- negative predictive value;
- PLR=
- positive likelihood ratio;
- PPV=
- positive predictive value
.
Stroke risk following TIA is approximately 5% at 7 days.1 The ABCD2 score was developed to stratify stroke risk after TIA and is calculated by assigning points for the presence of specific items (age, blood pressure, clinical features, duration of symptoms, and diabetes mellitus).2
Accurate risk prediction is essential for a prognostic rule to safely inform critical decision-making or guide allocation of clinical resources.3 The ABCD2 score has been evaluated in several settings, with varying reports of accuracy.4,–,13 Currently, some stroke guidelines recommend dichotomization of the score (“low risk”/“high risk”) to guide urgency of investigations or hospital admission policy.14,–,16 Other stroke guidelines caution against relying solely on ABCD2, raising questions about its use.17,18
A recent meta-analysis of ABCD2 reported good predictive value based on a pooled area under the receiver operating characteristic curve (AUC) of 0.72,19 however, AUC does not readily inform on performance if the score is dichotomized.20,21 A further important consideration is how ABCD2 may quantitatively affect classification of stroke risk in a TIA patient when compared with baseline risk.22 The previous meta-analysis did not evaluate this aspect of score performance since post-test probability analysis was not undertaken. Since that meta-analysis, several additional studies examining ABCD2 have been published, with conflicting conclusions.4,–,6,11,12,23,–,33
We conducted a meta-analysis of several measures of predictive performance to quantify the contribution of a dichotomized ABCD2 to stroke risk assessment in TIA patients. We modeled post-test probability of stroke to augment practical and clinical interpretation.
METHODS
Data sources and searches.
This systematic review and meta-analysis is reported in line with the MOOSE statement. Medline, PubMed, Embase, conference proceedings from the International Stroke Conference, European Stroke Conference, and World Stroke Congress, and reference lists of relevant papers were searched for studies using the ABCD2 score in TIA patients published up to October 2010 (L.S., K.C.). Full published papers and conference abstracts for both posters and oral platforms were considered. The search was not limited to the English language. Search terms included “ABCD,” “ABCD2,” “ABCD and TIA,” “ABCD and Stroke,” “ABCD2 and TIA,” and “ABCD2 and Stroke.”
Study selection.
Articles were assessed for duplicate publications from studies involving the same cohorts. Where necessary, authors were contacted for clarification of potential overlap to ensure data from each cohort were included only once in the meta-analysis. Inclusion of studies required stroke outcome at 2, 7, or 90 days to be available. Where adequate data were not available directly from the article or abstract, authors were invited to contribute their data regarding numbers of patients in categories of ABCD2 scores with stroke outcomes in a standardized tabular format.
Data extraction and quality assessment.
Two assessors (L.S., K.C.) independently extracted data for ABCD2 score and stroke outcome. Variables for subgroup analyses and meta-regression to explore heterogeneity were determined a priori, and these included study quality score, specialty of the physician making TIA diagnosis, study setting, study design, publication method, start year of cohort accrual, year of publication, method of stroke diagnosis, and inclusion in the original ABCD2 derivation or validation studies. Full articles and abstracts were assigned a consensus quality rating (table e-1 on the Neurology® Web site at www.neurology.org) by 2 senior neurologists (D.B., T.P.) using a standardized template based on a combination of criteria suggested by Altman21 and Hayden et al.34
Data synthesis and analysis.
The ABCD2 score was dichotomized at several cutpoints (>3, >4, >5), including those recommended in current stroke guidelines,4,–,6 to assign high vs low risk of stroke. Pooled sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (PLR), and negative likelihood ratio (NLR) were calculated across these thresholds for stroke outcome reported at 2, 7, and 90 days, using random effects meta-analysis.35 Formulae and definitions of predictive values and likelihood ratios are provided in figure e-1. Predictive values are subject to greater heterogeneity as they depend on pretest probability (baseline stroke risk), and thus may not be ideal statistical measures for meta-analysis. Likelihood ratios, however, are not directly affected by pretest probability, and also allow estimation of post-test probability.21 They indicate the likelihood that a given test result (ABCD2 score) would be expected in a patient with the target disorder (stroke) compared with the likelihood that the same result would be expected in a patient without that disorder. A high PLR (>10) and low NLR (<0.1) indicate sufficiently large changes in the likelihood of stroke from baseline risk to have confidence in either a positive test (high ABCD2 score) or a negative test result (low ABCD2 score). PLRs of 5–10 and NLRs of 0.1–0.2 confer moderate changes from baseline risk, while ratios of 1–2 and 0.5–1.0 rarely confer important changes.36
We used the user-written meta-analysis command “metan” in Stata 11.0 (DerSimonian and Laird option). Individual study PLRs were plotted against their corresponding NLRs to graphically display score performance, with scatterplots divided into quadrants based on the strength of evidence thresholds mentioned above. Where study data were available, the AUC from receiver operating characteristic curve analysis was generated using the command “roctab” in Stata 11.0. Pooled AUC was obtained using inverse variance weighting random effects meta-analysis. To determine magnitude and direction of change (increase or decrease) in stroke risk conferred by the ABCD2 score for different settings, the pooled likelihood ratios were used to calculate post-test odds according to Bayes' theorem21 and post-test probabilities of stroke after TIA were then calculated for a range of possible values of baseline stroke risk.
Heterogeneity was explored primarily for pooling of likelihood ratios, as these are preferred measures in meta-analyses of diagnostic tests.21 The presence of heterogeneity was assessed using Q (weighted sum of squares), with a p value of <0.05 indicating variability between studies beyond that attributable to random error. The I2 index was used to indicate the proportion of heterogeneity attributable to between-study variation. When I2 was greater than 50% (at least moderate heterogeneity),35 meta-regression and subgroup analysis were performed in Stata 11.0 for the a priori determined subgroups. Deeks' Funnel Plot Asymmetry Test37 was performed to evaluate publication bias.
RESULTS
The search (figure e-2) returned 27 published studies and 16 abstracts from conferences (e-References). Three non-English articles were identified and translated. Data from 2 author groups who developed the score were reported in a total of 7 independent studies.e9,e13,e14,e19 Additional information was requested from 34 authors, with 14 providing further information.e1,e2,e5,e8,e10–e12,e17,e18,e20,e21,e23,e27,e28 The rest declined (n = 6) or were uncontactable (n = 14).e3,e4,e6,e7,e15,e16,e24,e25,e27,e31–e41 An additional high-quality article (published in 2011) was included during the process of journal review.e42 Finally, data were available from 33 studies (16,070 patients) with 26/33 reporting stroke at 2 days (533 strokes), 32/33 at 7 days (781 strokes), and 28/33 at 90 days (1,028 strokes).e1–e29,e42 Data were available to calculate or extract AUC for 31 studies. Median stroke outcome was 1.66% (interquartile range [IQR] 0.68%–3.68%) at day 2, 4.29% (IQR 1.56%–6.34%) at day 7, and 6.28% (IQR 2.06%–10.1%) at day 90. There was no statistical evidence of publication bias (p = 0.12) (figure e-3) and mean age (p = 0.48), sex distribution (p = 0.22), and stroke outcome (p = 0.43) were similar between included studies and those with insufficient data for inclusion. Table 1 details characteristics of included studies. Table e-1 displays study quality scores.
Study characteristics
Individual and pooled likelihood ratios are presented in table 2. Pooled PLRs for different thresholds ranged from 1.42 to 2.32 at 2 days, 1.43 to 2.31 at 7 days, and 1.36 to 2.07 at 90 days. The PLR for each study is plotted against the corresponding NLR in figure 1 for varying ABCD2 thresholds and outcome periods. The majority of data points lie within the right lower quadrant of the scatterplot indicating that, in those individual studies, the score did not confidently assign either high or low risk status. The scatterplots demonstrate that no study was able to confidently assign both high and low risk status, irrespective of ABCD2 threshold or duration of follow-up. Although there was a trend for higher PLRs with increasing ABCD2 threshold, they remained uniformly low. No individual study had a PLR ≥10, and only 4 had a moderately high PLR between 5 and 10,e4,e12,e19,e25 all at ABCD2 cutoff >5. The range of pooled NLRs for different thresholds was 0.40 to 0.82 at 2 days, 0.40 to 0.81 at 7 days, and 0.42 to 0.81 at 90 days. Although individual study NLRs were lower for decreasing ABCD2 threshold, they too were uniformly poor, with NLR <0.1 in only 3 studies.e2,e3,e42 Pooled values for sensitivity, specificity, PPV, and NPV are presented in table 2. At day 7 pooled results varied as cutoff score increased; sensitivity ranged from 0.89 to 0.37, specificity 0.34 to 0.81, PPV 0.08 to 0.11, and NPV 0.98 to 0.95. The pooled AUC for stroke at 2, 7, and 90 days were 0.71 (95% CI 0.60–0.81), 0.70 (95% CI 0.62–0.78), and 0.69 (95% CI, 0.61–0.76), respectively.
Individual study positive and negative likelihood ratios and pooled summary statistics for stroke risk classification after TIA using a dichotomized ABCD2 score
Scatterplots of positive likelihood ratios (PLR) against negative likelihood ratios (NLR) for individual studies
Numbers in parentheses are 95% confidence intervals. In this setting, PLR indicates how many times more likely a person with stroke following TIA is to have been classified “high risk” (high ABCD2 score) when compared with a person without stroke. NLR indicates how many times more likely a person with stroke following TIA is to have been classified “low risk” when compared with a person without stroke. Each data point in the scatterplots represents an individual study. The study PLR is plotted against the study NLR for a particular ABCD2 threshold and outcome period. The horizontal red line represents the minimum acceptable PLR (10) to confidently assign high risk while the vertical red line represents the maximum acceptable NLR (0.1) to confidently assign low risk.36 These 2 red lines intersect to form quadrants. Data points falling within each quadrant are interpreted as follows: left upper quadrant (shaded gray)—confidently assign both high and low risk; right upper quadrant—confidently assign high risk only; left lower quadrant—confidently assign low risk only; right lower quadrant—unable to confidently assign either high or low risk.
There was moderate heterogeneity for pooling of PLR, with I2 >50% across most ABCD2 cutoff scores for stroke at all timepoints. Heterogeneity was highest at day 7 with ABCD2 cutoff >3 (I2 = 81.5%). In meta-regression (table 3), specialty of the physician diagnosing TIA accounted for 53.1% of total between-study variance. At ABCD2 cutoff of >3 for stroke 7 days after TIA, PLR was higher for TIA diagnosis by a stroke physician/neurologist (1.51, 95% CI 1.37–1.66) compared with an emergency physician (1.25, 95% CI 1.19–1.31). Method of stroke diagnosis accounted for 45.7% of variability, with PLR higher when diagnosis was made by face-to-face interview (1.50, 95% CI 1.36–1.66) vs medical record review (1.26, 95% CI 1.17–1.36). Other variables were not significant on meta-regression. When we restricted analyses to only high-quality studies (quality scores 8–10, table e-1), the pooled estimates of PLR showed little change from the analyses involving all studies, even when we included a recent high-quality study published after our search timeline.e42 There was minimal heterogeneity (I2 <30%) for pooling of NLR and therefore meta-regression of NLR was not undertaken.
Meta-regression for PLR at day 7 with ABCD2 dichotomized at >3 for high risk
Post-test probability of stroke at 7 days after TIA based on calculated pooled likelihood ratios and assuming varying levels of baseline stroke risk is shown in table 4. Applying the ABCD2 score to identify high risk led to only small revisions in stroke risk, particularly when baseline risk was low. For example, in the setting of a 2% baseline stroke risk in a TIA sample, the high threshold >5 would lead to a revised stroke risk of 4.5%, representing an absolute increase of only 2.5%. In the same setting, the commonly used threshold >3 would confer an absolute increase in stroke risk of only 0.8%. The magnitude of absolute increase in stroke risk from baseline ranged from 0.4% (very low baseline risk 1%, ABCD2 cutoff >3) to 10.4% (very high baseline risk 10%, ABCD2 cutoff >5). The magnitude of absolute decrease in stroke risk from baseline ranged from 0.6% (very low baseline risk 1%, ABCD2 cutoff ≤3) to 1.6% (very high baseline risk 10%, ABCD2 cutoff score ≤5). Results were similar for stroke at 2 and 90 days.
Modeling of post-test probability of stroke at 7 days after TIA for varying levels of pretest probability (baseline stroke risk) and ABCD2 thresholdsa
DISCUSSION
In this meta-analysis, we evaluated the ability of the ABCD2 score to accurately stratify risk of stroke within 90 days after TIA. Pooled likelihood ratios at several cutoff ABCD2 scores were of insufficient magnitude to accurately assign stroke risk in TIA patients with the high level of confidence required for urgent clinical decision-making and appropriate resource utilization. The score performed poorly when used to identify “high-risk” TIA patients in settings of low overall baseline stroke risk, and only modestly in the setting of high baseline risk. Although it performed marginally better in identifying “low-risk” patients, stroke risk was not entirely eliminated. Slight improvement in score performance was observed when used by stroke specialists and when stroke diagnosis was confirmed by face-to-face interview. Our results caution against the use of the score alone to guide decisions for urgent investigation or hospital admission in TIA patients.
The large number of studies and patients included is a strength of this analysis, with 15 additional studies since the previous meta-analysis. We followed standard search protocol21 consisting of extensive search strategy (including non-English studies), independent assessors for quality rating and data extraction, and careful subgroup analysis and meta-regression to explore heterogeneity. Although assessors of study quality were not blinded to study publication details, quality score was not an exclusion criterion nor was it associated with ABCD2 performance in meta-regression, suggesting lack of blinding was unlikely to systematically contribute to bias. Importantly, extraction of data for meta-analysis was performed independent of quality assessment. A limitation was that some study authors did not respond to our request for data, raising the possibility of bias and possible underestimation of the performance of the score. However, given the large number of studies included and similarity of the pooled AUC to the previous meta-analysis19 it is unlikely these would significantly alter our results. Although the test for publication bias was not statistically significant (p = 0.12), bias cannot be completely excluded as we did identify some small studies from conference proceedings that were unable to be included due to insufficient reporting of data, and we did not extend our search methodology to include other unpublished reports.
The pooled AUC reported in a previous meta-analysis19 is similar to that derived in our study. However, we present a different interpretation of the accuracy of the ABCD2 score. The authors of the meta-analysis described the ABCD2 score as having good predictive value based on the AUC alone.19 Their AUC of 0.7 indicates that for a random pair of TIA patients (1 truly high risk, 1 truly low risk) the ABCD2 score will assign a higher score to the truly high risk patient 70% of the time and a higher score to the low risk patient 30% of the time. We propose that this is a suboptimal level of accuracy for a prognostic score that may directly influence management. Looking beyond the AUC, we also quantified what the classification of “high” or “low” risk may actually mean if the score were to be applied dichotomously as in a clinical setting. Our results lead us to a much more cautious conclusion than that proposed by the authors of the previous meta-analysis.
The aim of ABCD2 is to reassign baseline stroke risk in TIA patients to assist in making decisions regarding urgency of investigations or hospital admission. For such a prediction rule to reliably advise decision-making, a high PLR (>10) and a low NLR (<0.1) are desirable since this represents substantial changes from baseline risk.36 Ratios ranging from 1 to 2 (PLR) or 0.5 to 1 (NLR), as found in our study, represent only small changes from baseline risk36 and may not provide enough confidence to decide on management based on the ABCD2 score. When the baseline risk is very high (∼10%), the calculated stroke risk at 7 days in a TIA patient based on our pooled NLR at ABCD2 score ≤3 would be lower than baseline risk at 4.3%, but not small in itself. Urgent investigation would therefore still be necessary to exclude modifiable high-risk stroke mechanisms not captured by the score (e.g., atrial fibrillation, carotid stenosis).6,7 However, the score may assign “low risk” more confidently in settings of very low baseline risk (e.g., 1%–2%), with the important caveat that risk is still not completely eliminated. Since score performance depends on baseline stroke risk, it is also important for clinicians to be aware of stroke rates in their local setting and subsequent implications of “low risk” classification in their patients before accepting the score into clinical practice. ABCD2 score performance for identifying high-risk patients, on the other hand, was uniformly poor over a range of baseline risk. Overcalling “high risk” has important consequences for resource utilization, especially if hospital admission is routinely advocated for such patients or if additional pressure is placed on other services such as radiology. Such a system may be cost-ineffective particularly when hospital beds are a limited resource. There are now several updated versions of the ABCD2 score with the addition of items for advanced imaging (e.g., diffusion-weighted MRI) or etiologic subtyping (e.g., large artery stenosis).38,39 An impact analysis of these scores remains to be done. Although they may have improved accuracy and assist better in management, the need for a more comprehensive range of investigations defeats the original purpose of the ABCD2 score as a widely applicable and simple clinical score to stratify risk in TIA patients.
In our study, the specialty of physician confirming TIA and method of stroke diagnosis explained a large proportion of PLR heterogeneity. While differences in subgroup analysis were statistically significant, absolute differences in PLR were small (e.g., neurologist 1.51 vs nonspecialist 1.25). Reduced ABCD2 accuracy when used by nonspecialists raises additional concerns as the score is often applied in emergency departments prior to specialist consultation. Results of a recent study showed that the score was miscalculated by emergency physicians in one-third of cases.40 The expertise of a neurologist during a face-to-face consultation may be an important component in ensuring appropriate application of the ABCD2 score. A small proportion of heterogeneity remained unexplained, possibly attributable to unmeasured factors such as treatment effect. The focus in recent years on urgent evaluation and improved management of vascular risk may also have contributed to heterogeneity by reducing stroke risk in some study populations. However, using year of study as a surrogate marker for treatment effect did not contribute to heterogeneity and we were unable to analyze this concept further as few studies systematically reported treatment protocols.
The performance of the ABCD2 score in this meta-analysis was poor when used to identify “high-risk” TIA patients and only modest when used to identify “low-risk” patients. Its use did not completely eliminate baseline risk when used to assign “low risk” status. The impact of the score was particularly low in settings of very low baseline risk and when used by nonspecialists. Given the urgency of clinical decision-making for TIA and the involvement of several physician groups in the process, it is important to consider a thorough validation process of future risk stratification tools. This should include the use of post-test measures of accuracy and randomized impact analysis studies before confidently recommending the tool in guidelines.
AUTHOR CONTRIBUTIONS
Study concept and design: Dr. Sanders, A/Prof. Srikanth, A/Prof. Phan. Acquisition of data: Dr. Sanders, Ms. Cooper. Quality rating: A/Prof. Blacker, A/Prof. Phan. Analysis and interpretation of data: Dr. Sanders, A/Prof. Srikanth, A/Prof. Jolley, A/Prof. Phan. Drafting of manuscript: Dr. Sanders, A/Prof. Srikanth, A/Prof. Phan. Critical revision of manuscript for intellectual content: all authors. Statistical analysis: Dr. Sanders, A/Prof. Jolley, A/Prof. Phan.
DISCLOSURE
L. Sanders reports receiving a National Health and Medical Research Council of Australia (NHMRC) post-graduate research scholarship. This funding source had no role in study design, data collection, data analysis, data interpretation, or writing of the report. V. Srikanth reports receiving a NHMRC/Heart Foundation Career Development Fellowship (ID:606544). This funding source had no role in study design, data collection, data analysis, data interpretation, or writing of the report. D. Blacker, D. Jolley, K. Cooper, and T. Phan report no disclosures. Go to Neurology.org for full disclosures.
ACKNOWLEDGMENT
The following authors contributed additional data to this analysis: A. Asimos (Carolinas Medical Centre, USA); H. Ay (Massachusetts General Hospital, USA); D. Morrison (University of Glasgow, UK); M. Ong (Singapore General Hospital, Singapore); H. Nguyen (Victoria University, Australia); A.M. Kelly (Joseph Epstein Centre for Emergency Medicine Research, Australia); C.J. Smith, P.J. Tyrrell, A. Vail (University of Manchester, UK); A. Rabinstein (Mayo Clinic, USA); C. Weimar (University of Duisberg-Essen, Germany); E. Dermitzakis, R. Jobst (Papageorgiou General Hospital, Greece); M. Reeves (Michigan State University, USA); P. Canhão (Hospital Santa Maria, Portugal); G. Andersen, P. von Weitzel-Mudersbach (Aarhus University Hospital, Denmark); J. Worthington, D. Ghia (Neurology Department, Liverpool Hospital and University of New South Wales, Sydney, Australia).
Footnotes
Editorial, page 958
Supplemental data at www.neurology.org
- Received October 20, 2011.
- Accepted February 29, 2012.
- Copyright © 2012 by AAN Enterprises, Inc.
REFERENCES
Letters: Rapid online correspondence
REQUIREMENTS
You must ensure that your Disclosures have been updated within the previous six months. Please go to our Submission Site to add or update your Disclosure information.
Your co-authors must send a completed Publishing Agreement Form to Neurology Staff (not necessary for the lead/corresponding author as the form below will suffice) before you upload your comment.
If you are responding to a comment that was written about an article you originally authored:
You (and co-authors) do not need to fill out forms or check disclosures as author forms are still valid
and apply to letter.
Submission specifications:
- Submissions must be < 200 words with < 5 references. Reference 1 must be the article on which you are commenting.
- Submissions should not have more than 5 authors. (Exception: original author replies can include all original authors of the article)
- Submit only on articles published within 6 months of issue date.
- Do not be redundant. Read any comments already posted on the article prior to submission.
- Submitted comments are subject to editing and editor review prior to posting.
You May Also be Interested in
Hastening the Diagnosis of Amyotrophic Lateral Sclerosis
Dr. Brian Callaghan and Dr. Kellen Quigg