Share

March 05, 2019; 92 (10) ArticleOpen Access

Validation of an algorithm for identifying MS cases in administrative health claims datasets

William J. Culpepper, Ruth Ann Marrie, Annette Langer-Gould, Mitchell T. Wallin, Jonathan D. Campbell, Lorene M. Nelson, Wendy E. Kaye, Laurie Wagner, Helen Tremlett, Lie H. Chen, Stella Leung, Charity Evans, Shenzhen Yao, Nicholas G. LaRocca, on behalf of the United States Multiple Sclerosis Prevalence Workgroup (MSPWG)
First published February 15, 2019, DOI: https://doi.org/10.1212/WNL.0000000000007043
Full PDF
Short Form
Citation
Permissions

Make Comment

See Comments

Downloads
709

Share

Abstract

Objective To develop a valid algorithm for identifying multiple sclerosis (MS) cases in administrative health claims (AHC) datasets.

Methods We used 4 AHC datasets from the Veterans Administration (VA), Kaiser Permanente Southern California (KPSC), Manitoba (Canada), and Saskatchewan (Canada). In the VA, KPSC, and Manitoba, we tested the performance of candidate algorithms based on inpatient, outpatient, and disease-modifying therapy (DMT) claims compared to medical records review using sensitivity, specificity, positive and negative predictive values, and interrater reliability (Youden J statistic) both overall and stratified by sex and age. In Saskatchewan, we tested the algorithms in a cohort randomly selected from the general population.

Results The preferred algorithm required ≥3 MS-related claims from any combination of inpatient, outpatient, or DMT claims within a 1-year time period; a 2-year time period provided little gain in performance. Algorithms including DMT claims performed better than those that did not. Sensitivity (86.6%–96.0%), specificity (66.7%–99.0%), positive predictive value (95.4%–99.0%), and interrater reliability (Youden J = 0.60–0.92) were generally stable across datasets and across strata. Some variation in performance in the stratified analyses was observed but largely reflected changes in the composition of the strata. In Saskatchewan, the preferred algorithm had a sensitivity of 96%, specificity of 99%, positive predictive value of 99%, and negative predictive value of 96%.

Conclusions The performance of each algorithm was remarkably consistent across datasets. The preferred algorithm required ≥3 MS-related claims from any combination of inpatient, outpatient, or DMT use within 1 year. We recommend this algorithm as the standard AHC case definition for MS.

Glossary

AHC=
administrative health claims;
CI=
confidence interval;
DMT=
disease-modifying therapy;
ICD-9=
International Classification of Disease, 9th revision;
ICD-9-CM=
International Classification of Disease, 9th revision, clinical modification;
ICD-10-CA=
International Classification of Disease, 10th revision, Canadian version;
ICD-10-CM=
International Classification of Disease, 10th revision, clinical modification;
IMS=
Intercontinental Marketing Services;
IP=
inpatient;
KPSC=
Kaiser Permanente Southern California;
MS=
multiple sclerosis;
NPV=
negative predictive value;
OP=
outpatient;
PPV=
positive predictive value;
VA=
Department of Veterans Affairs

The United States has historically been considered a region with a high prevalence of multiple sclerosis (MS). However, the prevalence of MS remains poorly described,1 and rigorous national estimates are lacking. Prevalence estimates are critical for understanding the national burden of MS, planning for health service needs, and supporting advocacy efforts.

Multiple methods have been used to estimate the prevalence of MS worldwide, including surveys, review of medical records or lists from health care institutions, and administrative health claims (AHC) data.1,–,3 AHC data are attractive because of their relatively low cost, ease of use relative to other methods, and potential for repeated use over time. However, AHC data are not collected for research purposes, and their validity must be assessed before use.4,–,6 Validation studies are possible for some AHC datasets in the United States such as the Department of Veterans Affairs (VA) but not in others such as Medicare because they cannot be linked back to clinical records.

Therefore, we aimed to test several case definitions (algorithms) for identifying persons with MS using multiple AHC datasets. We reasoned that if we could identify a best-case definition that performed well and comparably across diverse AHC datasets, this would support their use in other health claims datasets in which validation studies were not feasible. Further details and rationale for this approach are provided in one of our companion articles (Nelson et al.,7 “A New Way to Estimate Neurologic Disease Prevalence in the United States: Multiple Sclerosis”). Second, we aimed to determine how much the duration of the observation period would affect case detection (i.e., underestimate prevalence) for appropriate adjustment in our companion prevalence article (Wallin et al.,8 “The Prevalence of MS in the United States: A Population-Based Estimate Using Health Care Claims Data”).

Methods

This was a retrospective analysis using linked administrative and clinical data that was conducted in parallel at 3 primary sites (dataset characteristics in table e-1 available from Dryad, doi.org/10.5061/dryad.4c7s325) following a common protocol. The findings from the initial analysis were then tested in a fourth independent dataset. Sources of the AHC datasets are described below. Clinical data were drawn from medical records. Neurologist-confirmed diagnoses of MS based on the 2010 McDonald criteria9 served as the reference standard for this study. At the 3 primary sites, the source population for this study was limited to persons ≥18 years of age who were alive during the study period (2000–2014) and had ≥1 health care encounter associated with a diagnostic code for MS (Manitoba used any demyelinating disease to define its source population).

Standard protocol approvals, registrations, and patient consents

Each site obtained local Institutional Review Board or ethics approval: University of Maryland, Baltimore; Baltimore VA Research and Development Committee; Kaiser Permanente Southern California (KPSC); University of Manitoba Health Research Ethics Board; and University of Saskatchewan Biomedical Research Ethics Board. In Manitoba, approval for administrative data access was obtained from the Health Information Privacy Committee. In Saskatchewan, approval for data access was obtained from the Saskatchewan Ministry of Health and the Saskatchewan Health Quality Council.

Data sources

Department of Veterans Affairs

The VA provides comprehensive health care to ≈11 million military veterans at ≈300 hospitals and 1,500 outpatient clinics and Veterans Centers distributed throughout the United States, Puerto Rico, and Guam. Since 1997, the VA has prospectively collected and maintained extensive data on all funded health care services, including hospitalizations, physician visits, and prescriptions dispensed. Diagnoses were recorded using ICD-9 codes.

The initial validation cohort was drawn from the VA case-finding algorithm study in 2006,5 which included all patients in the mid-Atlantic region who were ≥18 years of age with ≥1 encounter with a diagnosis code for MS (ICD-9 code 340). We added data from the Gulf War Era MS Cohort10 and data from screening of patients for a planned longitudinal MS study. We followed the same standardized medical record review procedures (trained abstractors under the direction of M.W.) to verify the diagnosis of MS9 for all patients. In total, these studies yielded a cohort of 3,452 patients (table 1) from 1999 through 2007.

View this table:
Table 1

Demographic characteristics of the MS validation cohorts by data source

Kaiser Permanente Southern California

KPSC is a prepaid health maintenance organization with a membership during the study period of >3.5 million persons and provides health care coverage to ≈20% of the population in the region it serves. The KPSC membership is largely representative of the general population in Southern California with respect to ethnicity, age, and sex but underrepresents the lowest and highest ends of the socioeconomic spectrum.11 Physician visits, hospitalizations, and medications are captured in datasets that are linkable by a medical record number.

The validation cohort consisted of all persons ≥18 years of age with ≥1 inpatient (IP) or outpatient (OP) encounter diagnosis code for MS (ICD-9 code 340) who were KPSC health plan members between 2008 and 2010, had at least 6 months of continuous membership during the study period, and had complete electronic health records (n = 4,851). We randomly selected 2,935 (table 1) of these potential cases for in-depth medical records review by an MS specialist (A.L.-G), including IP and OP records, MRI scans, and diagnostic test results, to confirm a diagnosis of MS.9

Manitoba

In the Canadian province of Manitoba, Manitoba Health, Seniors and Active Living is responsible for the delivery of publicly funded health care to nearly all residents. Several datasets are maintained that capture demographic information (sex, dates of birth, health care coverage, and death), hospitalizations, physician services, and prescription drug dispensations. Diagnoses for physician services are recorded with the clinical modification of ICD-9 (ICD-9-CM) codes, while diagnoses for hospitalizations are coded with ICD-9-CM or the Canadian version of ICD-10 codes (ICD-10-CA), depending on the year.

The Manitoba validation cohort started with the 400 patients ≥18 years of age who were used to develop administrative case definitions of MS in 2007.6 This cohort was augmented by review of the medical records of all persons participating in the MS Clinic Registry (trained abstractors under the supervision of R.A.M.) at the Winnipeg MS Clinic who had consented to linkage of their medical records to administrative data (89% of those approached) as of June 2015 and met the inclusion criteria as described above. This expanded the validation sample to 1,654 patients (table 1).

Saskatchewan

This constituted the fourth independent site. In the Canadian province of Saskatchewan, health care is publicly funded as in Manitoba. Datasets similar to those in Manitoba capture demographic information (sex, dates of birth, health care coverage, and death), hospitalizations, physician services, and prescription drug dispensations. Diagnoses for physician services are recorded with ICD-9-CM codes, while diagnoses for hospitalizations are coded with ICD-9 or ICD-10-CA codes, depending on the year.

As described elsewhere,12 the validation cohort included 200 randomly selected cases from the Saskatoon MS Clinic with diagnoses confirmed by an MS specialist and 200 controls (table 1) from the Inpatient Rehabilitation Center database. This database specifically captures diagnoses of chronic diseases, including MS.13

MS case definition algorithms

We developed candidate algorithms using ICD-9/ICD-10 codes for MS (340/G35) and lists of MS-specific disease-modifying therapies (DMTs). To develop these definitions, we reviewed the literature on validated administrative case definitions for MS.5,6,12–15 From this review, we considered the following: (1) 1 encounter with an ICD-9/ICD-10 code for MS is too nonspecific because these may represent rule-out codes; (2) increasing the number of health care encounters required generally increases specificity at the expense of sensitivity; and (3) prescription claims for MS-specific DMTs may be helpful to improve sensitivity, specificity, or both when available. Therefore, the candidate algorithms (table 2) varied the number of IP and OP encounters with MS listed as one of the diagnoses for that encounter, the number of prescription claims for a DMT required to classify a person as an MS case, and the time period over which the case definition should be met. We included candidate algorithms that did not use prescription claims because these are not available in all AHC datasets. Because we anticipated applying these definitions in AHC datasets in which enrollment in a health insurance plan might be limited to short time periods, we considered only 1- or 2-year time periods over which to define a case. Each component of these case definitions is described further below.

View this table:
Table 2

Description of candidate algorithms for MSa

IP encounter

We defined an IP encounter as an IP admission (overnight stay) for which MS was recorded as one of the diagnoses. To account for transfers between institutions and to avoid double counting, we collapsed multiple overlapping hospital admission and discharge records into 1 IP encounter. In addition, an IP encounter with an admission date that occurred within 24 hours of a prior IP encounter was counted as a single IP encounter.5,6

Physician/OP encounters

We defined OP encounters as a visit to an OP clinic with an ICD code for MS. Multiple OP visits by 1 patient within the same day were treated as 1 OP encounter. In addition, OP encounters occurring within an MS IP (admission to discharge date) were not counted as unique OP encounters.

Disease-modifying therapies

We defined prescription claims as dispensations for any of the MS-specific DMTs approved before or during the study period, including interferon beta-1a-SC, interferon beta-1a-IM, interferon beta-1b-SC, glatiramer acetate, fingolimod, and natalizumab as identified with National Drug Codes and Drug Identification numbers (tables e-1 and e-2 available from Dryad, doi.org/10.5061/dryad.4c7s325). To avoid misclassification, claims for natalizumab were not included if the individual also had an ICD code for inflammatory bowel disease, a disorder for which this medication is approved for use.

A caveat in defining IP and OP encounters was that the VA imposes a hierarchy to the coding such that the primary diagnosis for a given encounter should be listed first, followed, in descending order of importance, by other diagnoses affecting that episode of care. Not all AHC datasets impose such a hierarchy (e.g., KPSC). Our assessment of the effect of this coding hierarchy (see Appendix 2 available from Dryad for additional methods and results, doi.org/10.5061/dryad.4c7s325) resulted in using MS as a primary diagnosis in the VA data and MS as any diagnosis in the KPSC and Manitoba datasets when counting MS encounters.

Analysis

We applied the candidate algorithms in each of the 3 primary validation cohorts. We compared the performance of the candidate algorithms with the reference standard diagnoses of MS from medical records using sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). In addition, interrater agreement was compared between the algorithms derived from AHC and medical records data using the crude accuracy score and Youden16 J statistic. We chose the J statistic over the κ statistic17,18 because it is designed for use with dichotomous data and equally weights false-positives and false-negatives. We interpreted the J statistic like κ, when neither the algorithm nor the medical records were considered the reference standard, as slight (0–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), and excellent (0.81–1.0).19

To determine whether the algorithms performed consistently across sociodemographic subgroups, we stratified the reference population by (1) sex, and (2) age group (18–24, 25–34, 35–44, 45–54, 55–64, 65–74, ≥75 years) and repeated our analyses for the 4 algorithms that performed best across the 3 AHC datasets. The ultimate choice of an algorithm was based on performance and pragmatic considerations, including ease of application and interpretation.

Because our 3 primary validation datasets were drawn from patients with at least 1 encounter for MS (or demyelinating disease in Manitoba), not the general population, we expected that the specificity and predictive values of the case definitions would be artificially depressed compared to the performance in the general population. Therefore, we tested our case definition algorithms in a fourth independent dataset in Saskatchewan,14 which included individuals with and without MS as noted above.

Prevalence with limited duration ascertainment periods

We intended to apply our optimal algorithm to multiple AHC datasets available in the United States20 to determine prevalence in 2010 based on a 3-year ascertainment period. However, limited periods of observation may substantially reduce case detection and underestimate prevalence, as is well recognized in the literature.21,–,23 Capocaccia et al.22 describe how they dealt with adjusting data from cancer registries that had been active for short durations when creating the Europe-wide project to estimate the prevalence of the most important cancers (EUROPREVAL). They computed what they called a completeness index for multiple observation periods using data from the Connecticut Cancer Registry because it has been in operation for >50 years. In the 4 registries that had only 5 observation years, the completeness index ranged from 0.42 to 0.45. The completeness index improved systematically as the observation period increased. They go on to describe their model-based methods to estimate the complete prevalence based on a limited period using age- and sex-specific incidence and survival rates for cancer.22 This information is lacking in the US MS population and cannot be accurately estimated from data of only 3 years’ duration.23 Therefore, to assess the effect of a short ascertainment period, we used the approaches used by Ng et al.21 and Capocaccia et al.22 and estimated the prevalence of MS using 3 years (2008–2010) and 10 years (1999–2010) of data in the VA and Manitoba datasets and compared the difference in these estimates in 2010. Prevalence in each year was defined as all persons ≥18 years meeting our preferred algorithm definition of MS ([IP + OP + DMT] ≥ 3) among those alive during the calendar year. The 3-year and 10-year prevalence estimates in 2010 were cumulative estimates based on all those who had met the definition at any point in the ascertainment period and who were alive in 2010.

To validate the above analyses, we repeated these procedures using the PharMetrics Plus Health Plan Claims Dataset from Intercontinental Marketing Services (IMS), which was not used to develop the algorithm. Here, we compared the estimated prevalence derived from 3 years (2013–2015) and 9 years (the longest period available) ending in 2015. The IMS dataset captures health care claims from ≈120 health plans across the United States accounting for ≥42 million covered lives as of 2011, mirrors the demographics of the US Census population, and has been used for national and regional benchmarking of health care use and cost.20,24 The data contained in this dataset are comparable to the data in validation datasets that included ICD-9-CM diagnosis and IP procedure codes, provider codes, Current Procedural Terminology (OP procedures) codes, National Drug Code, place and date of service, enrollment date(s), sex, year of birth, and US Census region. We report 95% confidence intervals (CIs)25 for the ratio of the prevalence estimates of longer vs shorter intervals. Although these prevalence estimates are highly correlated, which would narrow these intervals substantially, we report uncorrected 95% CIs to be conservative.

Statistical analyses were conducted with either SAS version 9.4 (SAS Institute Inc, Cary, NC) or SPSS version 22 (IBM Inc, Armonk, NY).

Data availability

The datasets used in this study constitute secondary analyses of data from prior studies. Therefore, there are no data-sharing agreements in place for these data. The exception concerns the IMS data, which are a limited-use dataset provided gratis to University of Maryland, Baltimore faculty (W.J.C.), and data sharing is not allowed under that agreement.

Results

The characteristics of the 3 primary validation cohorts are representative of the general MS population (table 1).5,6,10,13,26 As expected, there was a higher proportion of males in the VA data compared with KPSC and Manitoba, but the age distributions were similar across cohorts.

Initial validation of algorithm

Generally, when an algorithm was applied using 1 year of data, the sensitivity was lower than when applied using 2 years of data, while specificity was higher when applied using 1 year of data (table 3). The more stringent the case definition, that is, the greater the number of claims required, was, the lower the sensitivity was and the higher the specificity was. The addition of prescription claims for DMT to an algorithm increased sensitivity without a loss of specificity, leading to better performance overall.

View this table:
Table 3

Summary of test statistics for each algorithm based on years of data for each data source

Algorithm MS_E ([IP + OP + DMT] ≥ 3) had the highest sensitivity for both time periods. In contrast, the algorithm with the highest specificity was MS_C (≥2 IP or ≥5 OP) in the 1-year and 2-year time periods. The PPVs were high for all algorithms, whereas the NPVs were moderately low, suggesting that the algorithms were better at ruling in cases of MS than ruling out non-MS cases (in a population with at least 1 claim for MS).

The proportion of misclassified cases (false-positives and false-negatives) was relatively low, as reflected in the moderately high to high accuracy scores; algorithm MS_E had the highest accuracy over both time periods. The Youden J statistic mirrored the crude accuracy scores, revealing a substantial level of agreement between each algorithm and the reference standard. Algorithm MS_E had the highest or second highest J statistic for the 1-year time period with scores of ≥0.60.

We performed the stratified analyses on only the algorithms that included DMTs because these were the best-performing definitions (MS_D and MS_E in tables 3). Because the stratified results were nearly identical across these 4 case definitions, we present only the results for MS_E-1 ([IP + OP + DMT] ≥ 3) for the 1-year time period (figures 1 and 2).

Figure 1
Figure 1 Performance of algorithm MS_E-1 [(IP + OP + DMT) ≥ 3] stratified by sex across the VA, KPSC, and MB datasets

(A) Men and women, (B) men only, and (C) women only.Data are presented as a proportion that can range between 0 and 1. DMT = disease-modifying therapy; IP = inpatient; KPSC = Kaiser Permanente Southern California; MB = Manitoba; MS = multiple sclerosis; NPV = negative predictive value; OP = outpatient; PPV = positive predictive value; VA = Department of Veterans Affairs.

Figure 2
Figure 2 Performance of algorithm MS_E-1 [(IP + OP + DMT) ≥ 3] stratified by age group across the VA, KPSC, and MB datasets

(A) Sensitivity, (B) specificity, (C) positive predictive value, (D) negative predictive value, (E) accuracy, and (F) Youden J statistic.Data from Manitoba (MB) for the 55- to 64- and 64- to 74-year age groups are suppressed due to small cell sizes. Data are presented as a proportion that can range between 0 and 1. DMT = disease-modifying therapy; IP = inpatient; KPSC = Kaiser Permanente Southern California; MS = multiple sclerosis; NPV = negative predictive value; OP = outpatient; PPV = positive predictive value; VA = Department of Veterans Affairs.

The test characteristics remained stable by sex (men, figure 1B; women, figure 1C), including sensitivity, specificity, PPV, and accuracy. In the VA, the NPV and J statistic were lower among women compared to men but were stable across sexes in the KPSC and Manitoba datasets. Similarly, sensitivity, PPV, and accuracy were relatively consistent across age groups and somewhat stable for specificity. The NPV and J statistic showed the greatest variation; however, despite this variability, the MS_E-1 algorithm performed well across datasets and age groups. Thus, given its consistent and superior performance, algorithm MS_E-1 was chosen as the preferred algorithm.

Validation in the general population

The characteristics of the Saskatchewan validation cohort were comparable to those of the other cohorts. When we applied the algorithms to the Saskatchewan general population, specificities and PPVs were all ≥0.96 and the NPVs were all ≥0.91. Of the algorithms using 1 year of data, algorithm MS_E-1 performed the best, along with algorithm MS_D-1. For preferred algorithm MS_E, the sensitivity was 0.96, specificity was 0.99, PPV was 0.99, NPV was 0.96, and Youden J was 0.96. One of the 2-year algorithms, MS_D-2, also had a specificity of 0.99 but a marginally higher sensitivity of 0.97.

Corrections for limited-duration ascertainment periods

Figure 3 compares estimated prevalence based on a 3-year (2008–2010) ascertainment period to that from 10 years (2000–2010) of data as of 2010 in the VA and Manitoba datasets. The underestimation of a 3-year ascertainment period ranged from 37% (95% CI 13%–66%) in the VA dataset to 47% (95% CI 23%–76%) in the Manitoba dataset. In the IMS (validation) dataset, the underestimation of a 3-year (2013–2015) ascertainment period compared to a 9-year period (2007–2015) as of 2015 was 39% (95% CI 13%–71%). The estimated prevalence of MS rose steeply up to about 2010 in all 3 datasets (figures 3 and 4) and then increased an average of 2.3%/y thereafter (2.1% in the VA, figure 3A; 2.5% in the IMS, figure 4).

Figure 3
Figure 3 Comparison of prevalence based on a 3- vs 10-year ascertainment period as of 2010 in the (A) VA and (B) MB datasets

CI = confidence interval; MB = Manitoba; VA = Department of Veterans Affairs.

Figure 4
Figure 4 Comparison of prevalence based on a 3- vs 9-year ascertainment period as of 2015 in the IMS (validation) dataset

CI = confidence interval; IMS = Intercontinental Marketing Services.

Discussion

We undertook this study to identify an algorithm that would accurately and consistently identify cases of MS across disparate AHC datasets representing different types of health care systems and geographic regions. The consistency of the findings across these datasets supports the generalizability of the algorithms to other AHC datasets in the United States when such validation studies are not feasible. The preferred algorithm (MS_E-1) required ≥3 encounters (IP + OP + DMT) for MS in a 1-year period. Algorithms that did not include DMT performed less well than those that did. However, we included the non-DMT algorithms to assess their ability to accurately identify cases of MS in studies in which pharmacy claims data might not be available; MS_A-1 performed nearly as well as the preferred algorithm (MS_E-1) and could be used when DMT is not available. Thus, by testing a variety of algorithms, our study provides value, and the findings can be applied in future research into the prevalence of MS across jurisdictions.

Algorithm MS E-1 ([IP + OP + DMT] ≥ 3) was preferred because of its consistent, high performance and ease of application. We hereafter refer to it as the MS algorithm. The 1-year algorithm readily allows annually updating of prevalence estimates in datasets with limited years of data. Sensitivity and PPV were high, consistent with previously published definitions developed with similar methods in the United States (VA) and Canada.5,6,12,13 Specificity in our primary validation cohorts was modest, reflecting that our study cohorts were selected on the basis of having at least 1 diagnosis for MS. Tonelli et al.27 evaluated algorithms to measure 40 comorbid conditions in AHC data from Calgary, Alberta, Canada. They defined an algorithm as valid if both sensitivity and PPV were ≥0.70 compared to a reference cohort. They did not consider specificity or NPV because they are generally ≥0.90 in the assessment of chronic disease in the general population. Consistent with these observations and recommendations, when the MS algorithm was applied to the general population in Saskatchewan, sensitivity (0.96), specificity (0.99), and PPV (0.99) markedly improved. These findings support our preferred algorithm as a valid method for identifying cases of MS in AHC datasets.

Minor variation in performance was observed when stratified by sex and age. However, accuracy remained high, and the Youden J statistic remained substantial. These findings indicate that the MS algorithm was an accurate and reliable algorithm for identifying cases of MS in these disparate datasets and outperforms similar algorithms for identifying amyotrophic lateral sclerosis28 and Parkinson disease.29

When using AHC data, it is important for investigators to determine whether there is any hierarchy to the way that diagnosis codes are recorded because the performance of an algorithm can be adversely affected. In the VA, such a coding hierarchy exists, and when encounters with MS in any diagnosis position were used vs in the primary diagnosis position, specificity was compromised, as was the Youden J statistic. Surprisingly, accuracy was only slightly reduced but would likely be more adversely affected when applied to an entire dataset as opposed to the subset of cases with a diagnosis code for MS as used in this study.

Prior reports have defined cohorts in claims data based on a single encounter with MS listed as one of the diagnoses.30,31 This strategy can result in the inclusion of a substantial number of false-positives. Prior work on the VA algorithm5 showed that a rigorous algorithm (similar to the MS algorithm) ruled out 43% of the cases who had only a single MS encounter. This guided our decision to require multiple encounters as part of all of our algorithms. We strongly encourage other investigators to avoid the use of a single MS encounter to identify an MS cohort because, depending on the proportion of false-positives included, results could be substantially biased.

We found that prevalence estimates varied with the duration of the observation period in 3 different datasets (2 from the United States and 1 from Manitoba, Canada), suggesting that observation periods should be maximized to optimize case detection. Specifically, we found that an observation period of 3 years underestimated the prevalence of MS by 37% (95% CI 13%–66%) in the VA dataset and 47% (95% CI 23%–76%) in the Manitoba dataset. It should be noted that the 95% CI for these adjustment factors overlapped, implying that the growth range is nonsignificantly different for these 2 datasets. The findings in the IMS (validation) dataset corroborated results from the VA and suggest that a 3-year ascertainment period underestimates the 10-year prevalence in US datasets by ≈38% regardless of the final year of the time period of interest (2010 vs 2015). This is consistent with previous findings in Quebec, Canada, regarding systemic lupus erythematosus, a chronic disease like MS for which health care use may vary over time or with disease activity.21 In that study, when a 3-year period was used, prevalence was underestimated by 66% compared with a 15-year period, while a 5-year period underestimated prevalence by 39%. Similar results have been reported for the assessment of the prevalence of cancer in Europe.22 Registries with 5 years of data underestimated prevalence by 42% to 45%; as the duration of the registry increased, the underestimation of prevalence decreased systematically. For investigators without access to many years of data, these findings suggest that an adjustment factor should be applied to obtain a more stable and accurate prevalence estimate. The difference between the US (VA and IMS) and Manitoba datasets likely results because Manitoba has a universal health system with a single government payer and its AHC datasets capture >98% of the population,6 whereas the US health care system involves multiple insurance payers, making it more difficult to achieve complete ascertainment of cases. Thus, the underestimation value of 37% provides a conservative (i.e., lower bound) inflation factor, whereas an inflation factor of 47% based on the Manitoba data likely better reflects what might be expected with nearly full capture of the US population over a 10-year period (i.e., upper bound).

This study included a limited number of AHC datasets that were restricted to North America, which may limit generalizability to datasets from other countries. However, the datasets that were included provided a diverse sampling of the MS population. The IMS validation dataset did not cover the same time period as the VA and Manitoba datasets; however, the 3-year underestimation in prevalence was nearly the same as the VA dataset despite differences in the time periods assessed. Our algorithms relied on ICD-9-CM coding (except for IP encounters in the Manitoba dataset, which used ICD-10-CM coding from 2004 onward) and have not been validated with ICD-10-CM coding. It is unlikely the algorithms will perform differently when ICD-10-CM coding is used because MS is still represented by a single 3-digit code.

We conducted a rigorous assessment of algorithms to identify cases of MS in several different AHC datasets. The algorithm of choice required the sum of ≥3 MS-related IP admissions, MS-related OP encounters, or pharmacy claims for a DMT within a calendar year. Furthermore, this algorithm performed remarkably well across different datasets and demographic groups. We propose that this MS algorithm be adopted for future MS-related studies using AHC datasets to enhance comparability across studies.

Study funding

The study was funded by contracts from the National MS Society to the principal investigators of the US Multiple Sclerosis Prevalence Workgroup (HC-1508-05693).

Disclosure

W. Culpepper has additional research funding from the National MS Society and receives support from the VA MS Center of Excellence. R. Marrie is supported by the Waugh Family Chair in Multiple Sclerosis and receives research funding from Canadian Institutes of Health Research, Research Manitoba, Multiple Sclerosis Society of Canada, Multiple Sclerosis Scientific Foundation, National Multiple Sclerosis Society, Crohn's and Colitis Canada, and the Consortium of Multiple Sclerosis Centers. She also serves on the Editorial Board of Neurology. A. Langer-Gould was site principal investigator for 2 industry-sponsored phase 3 clinical trials (Biogen Idec; Hoffman-LaRoche) and was site principal investigator for 1 industry-sponsored observation study (Biogen Idec). She receives grant support from the NIH, National Institute of Neurological Disorders and Stroke (NINDS), Patient-Centered Outcomes Research Institute, and the National MS Society. M. Wallin has served on data safety monitoring boards for NINDS-funded trials, has been an associate editor for the Encyclopedia of the Neurologic Sciences, and received funding support from the National MS Society and the Department of Veterans Affairs Merit Review Research Program. L. Nelson receives grants from the Centers for Disease Control and Prevention, NIH, and National MS Society and contracts from the Agency for Toxic Substances and Diseases Registry. She receives compensation for serving as a consultant to Acumen, Inc and is on the Data Monitoring Committee of Neuropace. J. Campbell has consultancy or research grants over the past 5 years with the Agency for Healthcare Research and Quality, ALSAM Foundation, Amgen, AstraZeneca, Bayer, Biogen Idec, Boehringer Ingelheim, Centers for Disease Control and Prevention, Colorado Medicaid, Enterprise Community Partners Inc, Institute for Clinical and Economic Review, Mallinckrodt, NIH, National MS Society, Kaiser Permanente, PhRMA Foundation, Teva, Research in Real Life Ltd, Respiratory Effectiveness Group, and Zogenix Inc. H. Tremlett is the Canada Research Chair for Neuroepidemiology and Multiple Sclerosis. She receives research support from the National MS Society, the Canadian Institutes of Health Research, the Multiple Sclerosis Society of Canada, and the Multiple Sclerosis Scientific Research Foundation. In addition, in the last 5 years, she has received research support from the Multiple Sclerosis Society of Canada, the Michael Smith Foundation for Health Research, and the UK MS Trust, as well as travel expenses to attend conferences; all speaker honoraria are either declined or donated to an MS charity or to an unrestricted grant for use by her research group. W. Kaye receives funding from the Agency for Toxic Substances and Disease Registry, the National MS Society, and the Association for the Accreditation of Human Research Protection Programs. L. Wagner receives funding from the Agency for Toxic Substances and Disease Registry and National MS Society. L. Chen is employed full-time by KPSC. S. Leung is employed full-time by the University of Manitoba. C. Evans receives research funding from the Canadian Institutes of Health Research, Saskatchewan Health Research Foundation, and the Multiple Sclerosis Society of Canada. S. Yao is employed full-time by the Health Quality Council. N. LaRocca is employed full-time by the National MS Society. Go to Neurology.org/N for full disclosures.

Acknowledgment

The authors thank the following members of the US Multiple Sclerosis Prevalence Workgroup for their contributions: Albert Lo, Robert McBurney, Oleg Muravov, Bari Talente, Steve Buka, and Leslie Ritter. The authors also thank the following key staff members at the National MS Society for their contributions to this project: Cathy Carlson, Tim Coetzee, Sherri Giger, Weyman Johnson, Eileen Madray, Graham McReynolds, and Cyndi Zagieboylo. In addition, the authors are indebted to Shan Jin (Baltimore VA Medical Center) for his assistance in compiling the VA data. The results and conclusions presented are those of the authors. No official endorsement by the Department of Veterans Affairs, Manitoba Health, Saskatchewan Ministry of Health, Saskatchewan Health Quality Council, or Pharmetrics, Inc is intended or should be inferred.

Appendix 1 Authors

Appendix 2 Coinvestigators and members of the US Multiple Sclerosis Prevalence Workgroup

Table5

Footnotes

  • Go to Neurology.org/N for full disclosures. Funding information and disclosures deemed relevant by the authors, if any, are provided at the end of the article.

  • United States Multiple Sclerosis Prevalence Workgroup (MSPWG) coinvestigators are listed in appendix 2 at the end of the article.

  • The Article Processing Charge was funded by the National Multiple Sclerosis Society.

  • See page e1029

    See page 469

  • Received July 20, 2018.
  • Accepted in final form November 24, 2018.

This is an open access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License 4.0 (CC BY-NC-ND), which permits downloading and sharing the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal.

References

  1. 1.↵
    1. Evans C,
    2. Beland SG,
    3. Kulaga S, et al
    . Incidence and prevalence of multiple sclerosis in the Americas: a systematic review. Neuroepidemiology 2013;40:195–210.
    OpenUrlCrossRefPubMed
  2. 2.↵
    1. Kingwell E,
    2. Marriott JJ,
    3. Jetté N, et al
    . Incidence and prevalence of multiple sclerosis in Europe: a systematic review. BMC Neurol 2013;13:128.
    OpenUrlCrossRefPubMed
  3. 3.↵
    1. Makhani N,
    2. Morrow SA,
    3. Fisk J, et al
    . MS incidence and prevalence in Africa, Asia, Australia and New Zealand: a systematic review. Mult Scler Relat Disord 2014;3:48–60.
    OpenUrlCrossRefPubMed
  4. 4.↵
    1. Tricco AC,
    2. Pham B,
    3. Rawson NSB
    . Manitoba and Saskatchewan administrative health care utilization databases are used differently to answer epidemiologic research questions. J Clin Epidemiol 2008;61:192–197.e112.
    OpenUrlCrossRefPubMed
  5. 5.↵
    1. Culpepper WJ,
    2. Ehrmantraut M,
    3. Wallin MT,
    4. Flannery K,
    5. Bradham DD
    . Veterans Health Administration Multiple Sclerosis Surveillance Registry: the problem of case-finding from administrative databases. J Rehabil Res Dev 2006;43:17.
    OpenUrlCrossRefPubMed
  6. 6.↵
    1. Marrie RA,
    2. Yu N,
    3. Blanchard JF,
    4. Leung S,
    5. Elliott L
    . The rising prevalence and changing age distribution of multiple sclerosis in Manitoba. Neurology 2010;74:465–471.
    OpenUrlAbstract/FREE Full Text
  7. 7.↵
    1. Nelson MT,
    2. Wallin MT,
    3. Marrie RA, et al
    . A new way to estimate neurologic disease prevalence in the United States: illustrated with MS. Neurology2019;92:469–480.
    OpenUrlAbstract/FREE Full Text
  8. 8.↵
    1. Wallin MT,
    2. Culpepper WJ,
    3. Campbell J, et al
    . The prevalence of multiple sclerosis in the United States: a population-based estimate using health care claims datasets. Neurology2019;92:e1029–e1040.
    OpenUrlAbstract/FREE Full Text
  9. 9.↵
    1. Polman CH,
    2. Reingold SC,
    3. Banwell B, et al
    . Diagnostic criteria for multiple sclerosis: 2010 revisions to the McDonald criteria. Ann Neurol 2011;69:292–302.
    OpenUrlCrossRefPubMed
  10. 10.↵
    1. Wallin MT,
    2. Culpepper WJ,
    3. Coffman P, et al
    . The Gulf War era multiple sclerosis cohort: age and incidence rates by sex, race and service. Brain 2012;135:1778–1785.
    OpenUrlCrossRefPubMed
  11. 11.↵
    1. Koebnick C,
    2. Langer Gould A,
    3. Gould MK, et al
    . Do the sociodemographic characteristics of members of a large, integrated health care system represent the population of interest?. Permanente J 2012;16:37–41.
    OpenUrl
  12. 12.↵
    1. Widdifield J,
    2. Ivers NM,
    3. Young J, et al
    . Development and validation of an administrative data algorithm to estimate the disease burden and epidemiology of multiple sclerosis in Ontario, Canada. Mult Scler 2015;21:1045–1054.
    OpenUrlCrossRefPubMed
  13. 13.↵
    1. Al-Sakran LH,
    2. Marrie RA,
    3. Blackburn DF,
    4. Knox KB,
    5. Evans CD
    . Establishing the incidence and prevalence of multiple sclerosis in Saskatchewan. Can J Neurol Sci 2018;45:295–303.
    OpenUrl
  14. 14.↵
    1. Marrie RA,
    2. Fisk J,
    3. Stadnyk K, et al
    . The incidence and prevalence of multiple sclerosis in Nova Scotia. Can J Neurol Sci 2013;40:824–831.
    OpenUrlCrossRefPubMed
  15. 15.↵
    Canadian Institute for Health Information. Rehabilitation. 2016. Available at: cihi.ca/en/types-of-care/hospital-care/rehabilitation. Accessed May 5, 2016.
  16. 16.↵
    1. Youden WJ
    . Index for rating diagnostic tests. Cancer 1950;3:32–35.
    OpenUrlCrossRefPubMed
  17. 17.↵
    1. Landis JR,
    2. Koch GG
    . The measurement of observer agreement for categorical data. Biometrics 1977;33:159–174.
    OpenUrlCrossRefPubMed
  18. 18.↵
    1. Sim J,
    2. Wright CC
    . The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther 2005;85:257–268.
    OpenUrlAbstract/FREE Full Text
  19. 19.↵
    1. Viera AJ,
    2. Garrett JM
    . Understanding interobserver agreement: the kappa statistic. Fam Med 2005;37:360–363.
    OpenUrlPubMed
  20. 20.↵
    1. Trisolini M,
    2. Honeycutt A,
    3. Wiener J,
    4. Lesesne S
    . Global Economic Impact of Multiple Sclerosis. Research Triangle Park: RTI International; 2010.
  21. 21.↵
    1. Ng R,
    2. Bernatsky S,
    3. Rahme E
    . Observation period effects on estimation of systemic lupus erythematosus incidence and prevalence in Quebec. J Rheumatol 2013;40:1334–1336.
    OpenUrlAbstract/FREE Full Text
  22. 22.↵
    1. Capocaccia R,
    2. Colonna M,
    3. Corazziari I, et al
    . Measuring cancer prevalence in Europe: the EUROPREVAL Project. Ann Oncol 2002;13:831–839.
    OpenUrlCrossRefPubMed
  23. 23.↵
    1. Griffiths RI,
    2. O'Malley CD,
    3. Herbert RJ,
    4. Danese MD
    . Misclassification of incident conditions using claims data: impact of varying the period used to exclude pre-existing disease. BMC Med Res Methodol 2013;13:32.
    OpenUrl
  24. 24.↵
    1. Dilokthornsakul P,
    2. Valuck RJ,
    3. Nair KV,
    4. Corboy JR,
    5. Allen RR,
    6. Campbell JD
    . Multiple sclerosis prevalence in the United States commercially insured population. Neurology 2016;86:1014–1021.
    OpenUrlAbstract/FREE Full Text
  25. 25.↵
    1. Tiwari,
    2. Li Y,
    3. Zou Z
    . Interval estimation for ratios of correlated age-adjusted rates. J Data Sci 2010;8:471–482.
    OpenUrl
  26. 26.↵
    1. Culpepper WJ,
    2. Wallin MT,
    3. Magder LS, et al
    . VHA Multiple Sclerosis Surveillance Registry and its similarities to other multiple sclerosis cohorts. J Rehab Res Dev 2015;52(3):263–272.
    OpenUrl
  27. 27.↵
    1. Tonelli M,
    2. Wiebe N,
    3. Fortin M, et al
    . Methods for identifying 30 chronic conditions: application to administrative data. BMC Med Inform Decis Mak 2015;15:31.
    OpenUrlCrossRefPubMed
  28. 28.↵
    1. Kaye WE,
    2. Sanchez M,
    3. Wu J
    . Feasibility of creating a national registry using administrative data in the United States. Amytroph Lateral Scler Frontotemporal Degener 2014;15:433–439.
    OpenUrl
  29. 29.↵
    1. Szumski NR,
    2. Cheng EM
    . Optimizing algorithms to identify Parkinson's disease cases within an administrative database. Mov Disord 2009;24:51–56.
    OpenUrlCrossRefPubMed
  30. 30.↵
    1. Höer A,
    2. Schiffhorst G,
    3. Zimmermann A, et al
    . Multiple sclerosis in Germany: data analysis of administrative prevalence and healthcare delivery in the statutory health system. BMC Health Serv Res 2004;14:381.
    OpenUrl
  31. 31.↵
    1. Chastek BJ,
    2. Oleen-Burkey M,
    3. Lopez-Bresnahan MV
    . Medical chart validation of an algorithm for identifying multiple sclerosis relapse in healthcare claims. J Med Econ 2010;13:618–625.
    OpenUrlCrossRefPubMed
View Abstract

Disputes & Debates: Rapid online correspondence

No comments have been published for this article.
Comment

NOTE: All authors' disclosures must be entered and current in our database before comments can be posted. Enter and update disclosures at http://submit.neurology.org. Exception: replies to comments concerning an article you originally authored do not require updated disclosures.

  • Stay timely. Submit only on articles published within the last 8 weeks.
  • Do not be redundant. Read any comments already posted on the article prior to submission.
  • 200 words maximum.
  • 5 references maximum. Reference 1 must be the article on which you are commenting.
  • 5 authors maximum. Exception: replies can include all original authors of the article.
  • Submitted comments are subject to editing and editor review prior to posting.

More guidelines and information on Disputes & Debates

Compose Comment

More information about text formats

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
Author Information
NOTE: The first author must also be the corresponding author of the comment.
First or given name, e.g. 'Peter'.
Your last, or family, name, e.g. 'MacMoody'.
Your email address, e.g. higgs-boson@gmail.com
Your role and/or occupation, e.g. 'Orthopedic Surgeon'.
Your organization or institution (if applicable), e.g. 'Royal Free Hospital'.
Publishing Agreement
NOTE: All authors, besides the first/corresponding author, must complete a separate Disputes & Debates Submission Form and provide via email to the editorial office before comments can be posted.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.

Vertical Tabs

You May Also be Interested in

Back to top
Advertisement

Related Articles

Topics Discussed

Alert Me