The Stroke Outcomes and Neuroimaging of Intracranial Atherosclerosis (SONIA) Trial

Intracranial atherosclerosis causes an estimated 70,000 ischemic strokes in the United States each year.¹ In contrast to extracranial atherosclerosis, intracranial atherosclerosis is more common in US Hispanics, blacks, Japanese, and Chinese than in US whites.^2–5 Currently, catheter angiography is the accepted gold standard in diagnosing intracranial atherosclerosis. The Warfarin-Aspirin Symptomatic Intracranial Disease (WASID) study⁶ compared aspirin and warfarin in patients with more than 50% stenosis of an intracranial artery identified by angiography. However, angiography is a more expensive test and may cause stroke,⁷ so clinicians commonly rely on the results of noninvasive tests such as magnetic resonance angiography (MRA), transcranial Doppler (TCD), and computed tomographic angiography (CTA)^7–10 to evaluate patients suspected of intracranial atherosclerosis.

Despite the widespread use of TCD/MRA, and the increasing use of CTA, their ability to quantify the severity of intracranial atherosclerosis has not been rigorously characterized. Several studies have compared MRA, TCD, or CTA with angiography for detecting intracranial stenosis.^11–21 The studies suffer from small sample sizes, verification bias, selection bias, and retrospective bias. In addition, they do not consistently use standardized test perfor-mance protocols or well-described, reproducible measurement methods and often assess the ability of tests to detect stenoses, but not quantify them.

The Stroke Outcomes and Neuroimaging of Intracranial Atherosclerosis (SONIA) trial was a prospective, multicenter study aimed at validating the ability of TCD and MRA to diagnose intracranial atherosclerosis compared with catheter angiography. SONIA was a companion study to WASID and incorporated standardized, reproducible test performance protocols, test measurement methods, and diagnostic criteria. The trial was designed to avoid or correct for common forms of test evaluation bias.

METHODS

SONIA collaborated with WASID to study patients with TIA or ischemic stroke who were suspected of having intracranial stenosis. TCD and MRA were used to identify potential candidates for WASID, and angiography was required for entry into WASID, so a large, symptomatic patient population with both noninvasive tests and angiograms were identified. Both WASID and SONIA were prospective, multicenter trials.

WASID enrolled patients with 50 to 99% intracranial stenosis with symptoms in the past 90 days. For patients to be enrolled in SONIA, TCD or MRA had to be performed prior to angiography. Angiography could involve any number of injected vessels. If the patient had 50 to 99% stenosis, they were enrolled in WASID and SONIA. If not, they were not enrolled in WASID, but they could be enrolled in SONIA to study the correlation among their noninvasive tests and angiography.

The primary aim of SONIA was to define velocity values on TCD and anatomic abnormalities on MRA that identify 50 to 99% intracranial stenosis of the M1 segment of the middle cerebral, carotid siphon, and intracranial vertebral and basilar arteries on catheter angiography. SONIA was designed to see whether prespecified vessel-specific TCD velocity abnormalities and MRA percentage of stenosis and flow gap abnormalities (discontinuity of the blood flow column signal, with distal reconstitution) defined in a symptomatic patient population could perform with a positive predictive value (PPV) of 80% when compared against conventional angiography. PPV is the probability that a vessel that appears diseased on the noninvasive test also appears diseased on the confirmatory angiogram.

The secondary aim of SONIA was to study whether the absence of TCD and MRA abnormalities could exclude the presence of 50 to 99% stenosis by showing that they perform with a 90% negative predictive value (NPV) when compared against conventional catheter angiography. NPV is the probability that a vessel that reveals <50% stenosis on the noninvasive test also reveals <50% stenosis on the angiogram. Similar aims for CTA were added to SONIA after the study was under way, recognizing the emergence of CTA as a new imaging modality for intracranial stenosis. The aims were identical for those specified for MRA. The data for CTA do not represent the primary aim of SONIA and will be reported separately.

Patient identification and eligibility.

SONIA patients had the same inclusion and exclusion criteria as WASID patients with the exception of not requiring a positive angiogram.²² The flow diagram in figure 1 illustrates how patients were recruited in to SONIA. All SONIA patients had to be identified before their angiogram to be eligible for the study. This requirement ensured enrollment into SONIA of patients with positive noninvasive test results and negative angiogram results, as patients with negative angiography were excluded from WASID but were eligible to be enrolled in SONIA. For the patients who underwent angiography as part of routine care, not as part of the WASID protocol, SONIA would only enroll those patients if investigators notified the SONIA coordinating center that an angiogram was planned prior to its actual performance to avoid the bias of enrolling patients with predominantly positive angiography results.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1 Flow chart for identification and entry of Warfarin-Aspirin Symptomatic Intracranial Disease study candidates into Stroke Outcomes and Neuroimaging of Intracranial Atherosclerosis trial. *Transcranial Doppler ultrasound, MRI, or computed tomographic angiography. **Result could be positive based on one artery. Other arteries would be negative and also subject to angiographic verification. ***For other arteries, the noninvasive test could be either true or false positive or true or false negative depending on the results of the respective tests.

Statistical analysis.

To meet the primary and secondary aims of SONIA, PPVs and NPVs for TCD and MRA were calculated. For the purposes of this analysis, occlusions on angiography were considered to reflect an absence of stenosis. Thus, a positive noninvasive test followed by occlusion on angiography was scored as a false positive and a negative noninvasive test followed by occlusion on angiography was scored as a true negative.

The PPV and NPV values for the original and modified cutpoints were computed as proportions. Namely, for the PPV, the number of vessels with positive result on both the noninvasive test and angiography was divided by the number of vessels with a positive noninvasive test. For the NPV, the number of vessels with negative result on both the noninvasive test and angiography was divided by the number of vessels with a negative noninvasive test. This direct method of computation was possible for the pooled vessel analysis because the pooled approach provided a reasonable denominator for the computation of a proportion. Exact 95% CIs were used.

The modified cutpoints, which were defined with increased TCD velocities and increased MRA percentage of stenosis compared with the initial cutpoints, were designed to improve PPV, but the modification of cutpoints was limited by sparse data for higher values of the noninvasive continuous measures. Further improvements in PPV could not be estimated with a reasonable precision using the above direct method within the SONIA data. Accordingly, modified PPV and NPV for each vessel displayed in figure 2 were derived from a modeling process involving a logistic regression fit with the noninvasive test continuous measure predicting the probability of disease as defined by the angiogram. The model fit and smoothed density of the noninvasive test continuous measure were used to obtain the modified predictive values although approximate numeric integration.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2 Positive predictive value (PPV) and negative predictive value (NPV), with changing test cutpoints, for magnetic resonance angiography (MRA) and transcranial Doppler ultrasound. Note that for MRA, the illustrated logistic regression model excludes vessels with flow gap in the graph for PPV. For each vessel, the PPV for flow gap (95% CI) was as follows: MCA, 67% (95% CI: 57 to 76); ICA, 55% (95% CI: 32 to 77); VA, 81% (95% CI: 48 to 98); BA, 64% (95% CI: 45 to 80).

RESULTS

SONIA enrolled 407 patients at 46 sites in the United States over a 4-year period from February 1999 to July 2003. SONIA was successful in achieving widespread compliance with performance standards. The percentages of vessels meeting quality standards were TCD 94% and MRA 92%. None of the sites was dropped from SONIA participation because of test quality issues. Features of the patient population are described in table 1.

The number of vessels with corresponding angiographic data read were TCD 451 vessels and MRA 1,310 vessels. Anticipated sample size was achieved for NPV for TCD and MRA, as it was common for normal vessels on TCD/MRA to be also imaged by catheter angiography in the process of evaluating the patient as a candidate for WASID. Anticipated sample size for PPV was achieved only for MRA because this test was ordered by clinicians at a higher rate than initially anticipated, relative to the frequency of ordering TCD. SONIA anticipated from initial investigator surveys that TCD and MRA would be ordered with equal frequency. The actual ratio of test performance was 1:3 for TCD:MRA. Sample size was correspondingly not achieved for TCD PPV validation. The distribution of disease severity on centrally read angiograms is presented in table 2.

Table 3 illustrates the PPV/NPV and 95% CIs based on the initial test cutpoints. Note that these results represent the pooling of data for individual vessels because of smaller than expected sample sizes.

The primary aim of SONIA was to see whether PPVs of 80% were achievable. Adjustment of cutpoints during SONIA was prespecified in the study design but did not occur because of recruitment issues in WASID and sample size issues in SONIA. The cutpoints are measures of continuous variables such as velocity on TCD or percentage of stenosis on MRA and can be adjusted post hoc. Flow gaps on MRA are an exception as they are not continuous. They are either present or not. The MRA cutpoints retain flexibility, however, as it is still possible to adjust the percentage of stenosis required for the test to be defined as positive.

The SONIA cutpoints have thus been adjusted post hoc to see what positive predictive values are achievable. Cutpoints were adjusted for each vessel. The performance of the tests can be characterized by the eight graphs presented in figure 2. The graphs illustrate how PPVs or NPVs vary as the test cutpoint is changed. Generally, PPV increases and NPV decreases as the cutpoint is defined by a more severe abnormality. We also pooled the results for all vessels for each noninvasive test and adjusted the test cutpoints to maximize PPV. The resulting overall predictive values for catheter angiogram 50 to 99% stenosis are presented in table 4.

Cutpoints can also be adjusted to identify more severe disease, such as 70 to 99% stenosis of an intracranial artery on catheter angiogram rather than 50 to 99%. The SONIA data set determined cutpoints for 70 to 99% catheter angiogram stenosis of the individual arteries as follows: TCD mean velocity (cm/second) ICA 130, MCA 240, VA 130, BA 130; MRA (percentage of stenosis) all vessels 80%.

DISCUSSION

SONIA characterizes the performance of TCD and MRA for the detection of 50 to 99% intracranial stenosis of the MCA, cavernous ICA, intracranial VA, and BA as detected by catheter angiography. SONIA developed minimum quality standards for the performance of the diagnostic tests and standardized the cutpoints for TCD and MRA across 46 centers nationwide.

By excluding substandard tests, SONIA did not employ an intention to diagnose analysis. However, the SONIA standards for test performance reflected a general community standard that demands competence levels expected at most community and academic medical centers rather than unique state-of-the-art research facilities. Thus, the results, accepting the exclusion of a relatively small percentage of substandard tests, are easily generalizable. Different predictive values may be achieved at unique facilities performing state-of-the-art testing or more rigorous examination protocols.

SONIA was not designed to address whether MRA or TCD performed better than the other. Such an analysis could only be performed after test validation has occurred, cutpoints are fine-tuned to evaluate a specific population, and tests are compared on based on costs and clinical outcomes, not just the sum total of false-positive and false-negative results. As has been previously shown, a higher number of test errors may not only be acceptable in certain situations but desirable, where the clinical cost to the patient of a false-positive and false-negative test result differ markedly.²⁷

SONIA standardized the cutpoints for TCD and MRA across 46 sites because all local investigators agreed to the use of the same cutpoints, which were defined on a per-vessel basis. Such collaboration among investigators is important for achieving necessary sample sizes, as validation would be impossible if every local site used different cutpoints. Standardized cutpoints are not only feasible for a multicenter clinical trial, but may also be necessary for the community. Given the currently infrequent performance of catheter angiography at most institutions, the typical TCD or MRA department cannot generate enough comparisons between noninvasive tests and angiography to accurately develop and evaluate its own test cutpoints.

Many factors led to the smaller than expected sample size for PPV. The original design of SONIA called for a split-sample validation in which the PPV and NPV would be monitored throughout the course of the trial and the cutpoints for a positive noninvasive test would be adjusted to better achieve the desired target of 80% PPV. However, this design was changed due to several events that occurred throughout the WASID/SONIA collaboration. The original assumptions used to calculate sample size suggested that approximately 10% of patients would have angiograms performed outside WASID, as part of routine care, but the actual number was 75%. These patients could still be entered into SONIA if a notification call to Emory was placed prior to their angiogram, but this option was used only 60% of the time. Thus, a larger than predicted number of WASID patients were ineligible for SONIA because identifying patients only after angiography would introduce selection bias. In addition, the TCD sample size is even lower than expected because, whereas pilot surveys indicated that TCD and MRA would be performed in a 1:1 ratio, in the trial, MRA was performed much more often than TCD. Because a smaller sample size limits the accuracy of the results through widening of the resulting CIs, the National Institute of Neurological Disorders and Stroke Data and Safety Monitoring Board (DSMB) for the trial chose not to sacrifice any of the sample to the split-sample validation. The WASID/SONIA DSMB also determined that SONIA should not change the initial cutpoints as originally specified because of recruitment issues in WASID, our parent trial. SONIA also could not complete enrollment of all patients into the study, as WASID and, therefore, SONIA were terminated early because of warfarin-related safety issues in WASID. This termination affected the sample size of data available in SONIA. A small sample size for a test validation study does not preclude a result for PPV and NPV. The small sample size widens the CI around that result.

Thus, SONIA validated the cutpoints chosen at the outset of the trial and achieved NPVs very close to its targets. Our positive predictive values only approached our targets using the post hoc adjusted test cutpoints, but our results compare favorably with other studies. For example, a recent study reported PPV ranging from 55 to 78% and NPV ranging from 90 to 97% for duplex ultrasound and MRA of the extracranial carotid arteries.²⁸ A similar situation of high NPV but lower PPV has been reported for CTA of the carotid bifurcation.²⁹

There are several reasons why the PPVs found in SONIA for TCD and MRA were lower than expected. Target PPVs were determined by pilot data collected from selected sites that may perform the noninvasive tests at a higher quality standard than other sites used in the trial. Less rigorous blinding and measurement procedures used during pilot studies may have contributed to overestimates of PPV. The prevalence of disease in SONIA may be lower than in a selected group of patients comprising a cohort from which pilot data are generated, and PPV decreases as prevalence decreases. The method by which SONIA assessed arteriographic occlusions was conservative. Even though an occlusion represents severe disease, noninvasive abnormalities associated with occlusion on angiography were considered false positives because an occlusion is not treated with a stent. This approach results in SONIA increasing NPV but decreasing PPV. Technical and performance improvements in the tests, such as using gadolinium for MRA, or M-mode for TCD, may improve the PPV of these tests. Finally, these tests, TCD and MRA, when rigorously assessed, may simply perform much better at excluding the presence of stenosis rather than identifying it.

SONIA prospectively validated the cutpoint for each test for each vessel. Because of small sample sizes for some vessels, however, the summary data are presented as pooled for all vessels. The graphs in figure 2 illustrate how predictive values change with the cutpoint for each test and for each vessel. In the design of future trials employing these tests to evaluate patients, the SONIA data set allows for an informed choice of test cutpoints on an individual vessel basis.

SONIA determined PPV and NPV for TCD/MRA rather than sensitivity and specificity. Determining sensitivity and specificity directly would require that every patient having noninvasive testing also undergo angiography regardless of the result of the noninvasive test. The risks and costs of angiography are too high to ethically justify such a design. To calculate sensitivity and specificity from the SONIA data, corrections must be made to account for verification bias. Verification bias occurs when abnormal test results are verified more often than normal test results and commonly causes an overestimation of sensitivity (minimizing false negatives) and an underestimation of specificity (maximizing false positives).

In SONIA, only patients with positive TCD/MRA tests in a symptomatic vessel had angiography for safety and cost reasons, so verification bias may be present. This problem was partially corrected by correlating the noninvasive results of the normal vessels with the angiograms of those vessels. Additionally, a mathematical correction for verification bias will be performed as a separate study that reflects an exploratory aim of the SONIA trial.

Using the same criteria for patients entering SONIA and WASID ensured that the noninvasive tests examined in SONIA were performed in a symptomatic and readily identifiable patient population. Bias was avoided by requiring that all patients undergoing angiography were included in SONIA, regardless of the results of angiography. Without this requirement, patients with negative angiograms who were not enrolled in WASID would not be enrolled in SONIA, and the noninvasive tests would have a falsely high PPV. SONIA met other important criteria for reliable assessment of diagnostic tests: blinded measurements, prospective enrollment, the use of standardized techniques, and the use of a reference standard—catheter angiography.

The SONIA data set provides an initial starting point for clinicians and researchers to use these tests because the cutpoints are flexible. Because the cutpoints generally employed measurements of continuous variables, velocity on TCD, or percentage of stenosis on MRA, the cutpoints can be adjusted as needed to target unique cohorts of patients. When evaluating patients, test cutpoints can be readily adjusted to take advantage of the relatively strong NPV of the tests to exclude severe (>70%) intracranial stenosis in patients, which carries a particularly dire prognosis.³⁰ Physicians have reliable SONIA data on which to base decisions made after TCD or MRA, including referral to angiography for emerging therapies such as angioplasty and stenting.