Share

March 24, 2020; 94 (12) ArticleOpen Access

Validation of the NIH Toolbox Cognitive Battery in intellectual disability

Rebecca H. Shields, Aaron J. Kaat, Forrest J. McKenzie, Andrea Drayton, Stephanie M. Sansone, Jeanine Coleman, Claire Michalak, Karen Riley, Elizabeth Berry-Kravis, Richard C. Gershon, Keith F. Widaman, David Hessl
First published February 24, 2020, DOI: https://doi.org/10.1212/WNL.0000000000009131
Full PDF
Short Form
Citation
Permissions

Make Comment

See Comments

Downloads
2637

Share

Abstract

Objective To advance the science of cognitive outcome measurement for individuals with intellectual disability (ID), we established administration guidelines and evaluated the psychometric properties of the NIH-Toolbox Cognitive Battery (NIHTB-CB) for use in clinical research.

Methods We assessed feasibility, test-retest reliability, and convergent validity of the NIHTB-CB (measuring executive function, processing speed, memory, and language) by assessing 242 individuals with fragile X syndrome (FXS), Down syndrome (DS), and other ID, ages 6 through 25 years, with retesting completed after 1 month. To facilitate accessibility and measurement accuracy, we developed accommodations and standard assessment guidelines, documented in an e-manual. Finally, we assessed the sensitivity of the battery to expected syndrome-specific cognitive phenotypes.

Results Above a mental age of 5.0 years, all tests had excellent feasibility. More varied feasibility across tests was seen between mental ages of 3 and 4 years. Reliability and convergent validity ranged from moderate to strong. Each test and the Crystallized and Fluid Composite scores correlated moderately to strongly with IQ, and the Crystallized Composite had modest correlations with adaptive behavior. The NIHTB-CB showed known-groups validity by detecting expected executive function deficits in FXS and a receptive language deficit in DS.

Conclusion The NIHTB-CB is a reliable and valid test battery for children and young adults with ID with a mental age of ≈5 years and above. Adaptations for very low-functioning or younger children with ID are needed for some subtests to expand the developmental range of the battery. Studies examining sensitivity to developmental and treatment changes are now warranted.

Glossary

CAT=
computerized adaptive testing;
DCCS=
Dimensional Change Card Sort;
DEXT=
Developmental Extension;
DS=
Down syndrome;
EF=
executive function;
Flanker=
Flanker Inhibitory Control and Attention;
FM=
Forward Memory;
FXS=
fragile X syndrome;
FSIQ=
full-scale IQ;
ICC=
intraclass correlation;
ID=
intellectual disability;
LS=
List Sorting Working Memory;
NEPSY-In=
NEPSY Inhibition subtest;
NIHTB-CB=
NIH Toolbox Cognitive Battery;
OID=
ID of other or unknown cause;
OR=
Oral Reading Recognition;
PC=
Pattern Comparison Processing Speed;
PSM=
Picture Sequence Memory;
PPVT-4=
Peabody Picture Vocabulary Test, 4th edition;
PVT=
Picture Vocabulary;
SB-5=
Stanford-Binet, 5th edition;
VABS-3=
Vineland Adaptive Behavior Scales, 3rd edition

Approximately 2.0% of the global population has an intellectual disability (ID), an understudied condition with lifelong effects on academic, vocational, and personal functioning.1 As the etiologies and mechanisms underlying specific forms of ID are discovered, targeted treatments and human clinical trials soon follow in the translational process,2,3 raising the potential for medical remediation of disability. The preclinical development of promising targeted treatments for ID-associated disorders has not been followed by successes in human trials. Unfortunately, testing accessibility issues, pervasive floor effects, and lack of consensus on acceptable cognitive endpoints have been obstacles in the field. Although other barriers such as limitations of animal models are involved, it is apparent that the IDs continue to lag behind other neurologic or psychiatric conditions on scalable, psychometrically supported, and broadly accepted endpoints.4,5

The NIH Toolbox Cognitive Battery6,–,8 (NIHTB-CB), an iPad-based battery of brief memory, executive function (EF), processing speed, and language tests, was developed within the NIH Blueprint for Neuroscience Research. The NIHTB-CB has the potential to provide a highly standardized, objective, and scalable tool for use across laboratories and clinical trial sites. As an extension of our pilot work,9 the present study reflects progress made over a 4-year period to empirically validate and refine the NIHTB-CB for ID (aim 1), including standardized administration guidelines required for this challenging population, with the goal of supporting its use as a set of outcome measures for clinical trials and other clinical research. The second aim was to measure the sensitivity of the battery to detect known cognitive phenotypes in 2 ID-associated syndromes that are a focus of translational research: Down syndrome5 (DS) and fragile X syndrome10 (FXS). On the basis of prior research, we hypothesized EF deficits in FXS and DS and episodic memory deficits in DS (relative to a heterogeneous other-ID group), a relative language strength in FXS, and no group differences on visual processing speed.

Methods

Standard protocol approvals, registrations, and patient consents

Institutional Review Board approval was obtained at each site (University of California Davis MIND Institute, University of Denver, and Rush University Medical Center) before study initiation. Written consent was obtained from each guardian-participant pair according to Institutional Review Board requirements.

Participants

Because of the study aim to measure the sensitivity of the battery to syndrome-specific cognitive phenotypes, 3 groups were recruited. Two ID-associated syndromes were chosen: DS (affecting ≈1 in 700)11 and FXS (affecting ≈1 in 7,000 males and 1 in 11,000 females).12 Individuals with ID of other or unknown cause (OID) were also recruited to evaluate the NIHTB-CB within a more heterogeneous group and to serve as comparison to FXS and DS. Eligible participants met the following criteria: chronologic age of 6 through 25 years; full-scale IQ (FSIQ) <80 on the Stanford-Binet, 5th edition (SB-5),13 mental age of at least 3.0 years on the SB-5 (in concordance with the lowest chronologic age limit of the NIHTB-CB), adaptive behavior deficits as measured by the Vineland Adaptive Behavior Scales, 3rd edition Comprehensive Interview14 (VABS-3), speech of at least short phrases, English as first language, and stable medication and intervention regimen for 6 weeks before enrollment. Exclusion criteria were uncorrected vision impairment, uncontrolled seizures, motor impairment affecting touchscreen use, and a history of head trauma, brain infection, or stroke.

In all, 288 participants consented to the study. After completion of the SB-5, 45 participants were ineligible: 16 with an FSIQ >80 and 29 with a mental age <3.0 years. One participant discontinued due to behavioral noncompliance. Across sites, 242 participants completed initial neuropsychological testing, with 228 completing retesting of the NIHTB-CB ≈1 month later to examine test reliability. This retest duration was selected to evaluate reliability within a typical time interval used in clinical trials. Participants included 91 with DS, 75 with FXS, and 76 with OID. A subset of 21 participants with OID had a diagnosed ID-associated syndrome; the represented syndromes were 16p11.2 deletion (1), 22q11.2 deletion (1), Bannayan Riley Ruvalcaba (1), cri-du-chat (1), fetal alcohol (5), Floating-Harbor (1), Kleefstra (1), mitochondrial disease (1), mosaic trisomy 8 (1), neurofibromatosis type 1 (2), Phelan-McDermid (1), Potocki-Lupski (4), and Williams (1).

Protocol

The NIHTB-CB and all convergent validity measures (see below) were completed at visit 1 across 2 days. After completion of the SB-5, the order of remaining assessments was randomized with the exception of the NIHTB-CB, which was the first assessment of day 2. The order of the NIHTB-CB tests was randomized for each participant. At visit 2, the NIHTB-CB was readministered with the same test order within participants.

The NIHTB-CB

The NIHTB-CB is a computerized assessment validated in ages 3 to 85 years in the general population. The battery includes 7 tests: Dimensional Change Card Sort (DCCS), Flanker Inhibitory Control and Attention (Flanker), List Sorting Working Memory (LS), Pattern Comparison Processing Speed (PC), Picture Sequence Memory (PSM), Picture Vocabulary (PVT), and Oral Reading Recognition (OR).9,15 The NIHTB-CB provided experimental Developmental Extension (DEXT) versions of DCCS and Flanker designed to be more accessible to lower-functioning or very young participants. Because tests have multiple age versions, the participant's mental age derived from the SB-5 was used to select test versions, allowing for a starting point of reasonable difficulty, thereby reducing frustration and improving compliance. The DEXT versions were used for participants in the 3- to 7-year mental age range. For PVT and OR, there is 1 computerized adaptive testing (CAT) version, and the start point is typically based on age (children) or education (adults). Instead, we used the education override feature, entering the grade equivalent of the mental age as the start point.

In addition, PSM has multiple forms available for each test version. Pilot reliability results suggested nonequivalence of forms. To assess PSM reliability, Form A was used at visit 1 and for half of participants at visit 2 (PSM A-A); the other half received Form B at visit 2 (PSM A-B). For LS, pilot studies demonstrated that additional teaching items improved feasibility. For the current study, PowerPoint slides of these teaching items were used before test items on LS Age 3–6. The NIHTB-CB developers then released the LS Age 3–6 Experimental version with these extended instructions during the study, which was subsequently used.

Convergent validity tests

Six tests were preselected as convergent validity measures for the NIHTB-CB. The NEPSY Inhibition subtest16 (NEPSY-In), iPad version, was used as the convergent measure for DCCS. The NEPSY-In measures cognitive flexibility and inhibitory control. From piloting the NEPSY-In, we found that participants could rarely do the most difficult level (Switching). We thus administered only the Naming and Inhibition portions and created a prorated score indicating the number of correct items per minute. For Flanker, we used the Conners Kiddie Continuous Performance Test 2nd Edition17, administered on a computer with a spacebar as the response button. The hit reaction time SD was used as the convergent validity variable. For LS convergent validity, the SB-5 verbal working memory raw score was used. PC validity was measured with the Wechsler Preschool and Primary Scale of Intelligence, 4th Edition18 Bug Search, from the number of correct items per minute. The Leiter International Performance Scale, 3rd Edition19 Forward Memory (FM) subtest assesses sequential memory span. The raw score was the convergent variable for PSM. The Peabody Picture Vocabulary Test, 4th edition20 (PPVT-4) measures receptive vocabulary. The raw score from the PPVT-4 iPad version was used for PVT. For OR, the Woodcock Johnson 4th Edition21 Letter-Word Identification was used, which measures letter recognition and single word reading. Discriminant measures for NIHTB-CB tests were selected out of these measures by choosing a feasible measure of a different construct than the NIHTB-CB test.

Adaptations

To increase feasibility and to improve reliability and validity in ID, we developed a manual of standardized procedures regarding the test environment and NIHTB-CB administration: the “NIH Toolbox Cognitive Battery Supplemental Administrator's Manual for Intellectual and Developmental Disabilities” (e-Manual; hereafter Supplemental Manual, links.lww.com/WNL/B58).22 Strategies to proactively improve feasibility and to reduce participant stress included using a visual schedule before the visit, a caregiver questionnaire on behaviors and potential reinforcement rewards, and a visual token board of the NIHTB-CB for the participant to check off during testing. Best practices in administering standardized assessments with appropriate accommodations for ID were used.23 Test-specific guidelines are available in the manual to aid future users of the NIHTB-CB in standard administration and feasibility specifically for the ID population. In addition, the Supplemental Manual includes the Administration Form that we developed to document test environment, behavioral responses, and validity of tests for each participant.

Data cleaning and scoring

After every administration of the NIHTB-CB, the Administration Form was used to record whether each test was considered valid for the participant. All analyses used only valid scores. The most common reasons for invalid scores were an invalid response pattern, refusal, and excessive prompting (3.0%, 0.78%, and 0.72% of all scores, respectively).

Before conducting analyses, we visually inspected all data and bivariate correlations for normality and the presence of outliers. All NIHTB-CB tests were examined for floor or ceiling issues. Only 2 tests had such issues. At visit 1, 21 individuals received a score at the floor on LS Age 3–6, 33 received a score at the ceiling on PSM Age 3–4, and 4 received a score at the floor on PSM Age 5–6. After a thorough review of administration details and reliability and validity analyses, LS floored scores were kept in the analyses. Because PSM has multiple age versions and these participants likely should have received a harder or easier version, PSM scores at floor or ceiling were excluded.

Data analysis

SAS version 9.4 (SAS Institute Inc, Cary, NC) and R version 3.6.0 (R Foundation for Statistical Computing, Vienna, Austria) were used for analyses. Visit 1 data were used for feasibility and validity analyses. For validity and reliability, NIHTB-CB raw scores were used: computed score on DCCS, Flanker, and PC; theta score on PSM, PVT, and OR; raw score on LS and Flanker DEXT; and percent correct on DCCS DEXT. Because DEXT scores are currently on a different scale than standard Flanker and DCCS scores, DEXT results are presented separately. For test-retest reliability, single-score intraclass correlations (ICCs) were used. The Cohen d was used to evaluate potential practice effects, with paired-sample t tests to measure the significance of change. Convergent, discriminant, and ecologic validities were measured with Pearson correlations.

Our prior work on IQ measurement demonstrated the utility of deviation-based scoring to deal with problematic floor effects in ID.24 We used this method in the current study in place of NIHTB-CB standard scores to circumvent imposed floored scores (e.g., the current NIHTB-CB winsorizes age-corrected standard scores at 54). We created z scores by transforming participant raw scores on each NIHTB-CB test using normative means and SDs for their chronologic age band. The z scores were used to create deviation-based composites following the previously defined criteria.25 For a Crystallized Composite, 1 of 2 valid scores on PVT and OR was required; for a Fluid Composite, 4 of 5 valid scores on Flanker, DCCS, LS, PC, and PSM were required. The Crystallized and Fluid z scores were used to create a Cognitive Function Composite, used in ecologic validity analyses. Deviation scores were also created for FSIQ on the SB-524 and were used in FSIQ analyses. Group comparisons to examine known-groups validity were assessed with a 2-way mixed-model analysis of variance on NIHTB-CB z scores. Significant results were followed up with the Tukey honest significant difference tests to examine group differences.

Data availability

On request to the corresponding author, anonymized data are available to share.

Results

Descriptive statistics

Table 1 presents descriptive statistics by diagnostic group and overall. Groups did not differ significantly by chronologic age (F2,238 = 0.91, p = 0.41) or by VABS-3 Adaptive Behavior Composite (F2,227 = 1.50, p = 0.23). However, FSIQ differed significantly by group (F2,238 = 31.6, p < 0.001), with FSIQ higher in OID than in both DS [t(238) = 7.57, p < 0.001] and FXS [t(238) = 6.05, p < 0.001]. FSIQ did not significantly differ between DS and FXS [t(238) = 1.23, p = 0.22]. Similarly, mental age was significantly different by group (F2,238 = 25.3, p < 0.001), with mental age higher in OID than in both DS [t(238) = 6.76, p < 0.001] and FXS [t(238) = 5.40, p < 0.001]. Mental age did not differ significantly between DS and FXS [t(238) = 1.10, p = 0.27]. Figure 1A shows the distribution of the NIHTB-CB Cognitive Function Composite age-adjusted standard scores of the sample without the current imposed floor, illustrating the variability of the sample below this floor and the benefit of deviation-based composites (used in all analyses).

View this table:
Table 1

Participant descriptive information

Figure 1
Figure 1 NIHTB-CB cognitive function composite score distribution and association with FSIQ

(A) Histogram showing the true distribution of cognitive function composite scores in the full sample of individuals with intellectual disability (ID). The NIH Toolbox Cognitive Battery (NIHTB-CB) current floor (winsorized at 54; vertical blue line) excluded more than half of the present sample from more accurate measurement below that level. (B) Association between the deviation-based cognitive function composite and deviation-based full-scale IQ (FSIQ) by group.

Feasibility

Feasibility data are provided in table 2 as the percentage of participants with valid scores on each test. Feasibility overall was similar to the normative 3- to 15-year-old sample,26 with DCCS, PC, and LS having slightly lower feasibility and Flanker, PSM, PVT, and OR having similar or higher feasibility rates than the normative sample. Even down to a mental age of 3 years, PSM, PVT, and OR feasibility was very good. The feasibility of the remaining tests improved particularly at 5 years. All tests were feasible for nearly every participant with a mental age of ≥6 years.

View this table:
Table 2

At visit 1, proportion of participants who received a valid scorea

Test-retest reliability

Test-retest reliability was assessed after ≈1 month (mean = 31.7 days, SD = 6.3 days) (table 3). ICCs on each test were moderate to strong, with the exception of DCCS DEXT and PSM A-A, which were in the high 0.40s. All composites had strong reliability. Three tests had small but significant visit 1 to visit 2 effect sizes, reflecting a modest increase in performance: Flanker, PC, and the PSM A-B group. However, the PSM A-B increase likely reflects nonequivalent forms rather than a practice effect because the PSM A-A group had no practice effect. The Fluid and Cognitive Function composites also had small but significant increases, suggestive of small practice effects. Within groups, reliability coefficients were mostly moderate to strong; some DEXT and PSM reliabilities in small sample sizes were not significant: Flanker DEXT in OID (ICC = 0.46, p = 0.14) and both PSM groups in DS (A-A: ICC = 0.24, p = 0.15; A-B: ICC = 0.33, p = 0.05). In FXS, DCCS reliability (ICC = 0.41) was notably lower than in the total sample (ICC = 0.71), but FXS Flanker reliability (ICC = 0.84) was stronger than in the total sample (ICC = 0.74).

View this table:
Table 3

Test-retest reliability and examination of practice effects

Construct validity

Convergent and discriminant validity results are presented in table 4. Convergent correlations ranged from moderate to strong, with the exception of Flanker DEXT (n = 29, r = −0.26, p = 0.17). Group results generally reflect validity similar to that of the overall results. Of note, PSM was only weakly correlated with Leiter-FM in DS (r = 0.33, p = 0.01) and in OID (r = 0.32, p = 0.01).

View this table:
Table 4

Convergent and discriminant validity of NIHTB-CB tests (Pearson r)

In DCCS, for a subgroup of participants below a raw score of 1.88, there appeared to be no association with the NEPSY-In; in this subgroup, DCCS score was not significantly correlated with NEPSY-In score (r = 0.29, p = 0.16). This DCCS score represents participants who did not pass the introductory switching portion of the test. When this subgroup was removed, validity improved (r = 0.57, p < 0.001).

Ecologic validity

Table 5 provides the ecologic validity of NIHTB-CB tests and composites. The composites each had moderate to strong correlations with FSIQ (figure 1B), as did all NIHTB-CB test scores other than Flanker DEXT. VABS-3 Adaptive Behavior Composite had small but significant correlations with several tests (Flanker, PC, PSM, PVT, and OR) and with the Crystallized and Cognitive Function composites, with better performance associated with higher levels of adaptive behavior.

View this table:
Table 5

Ecologic validity (Pearson r)

Syndrome-specific comparisons (known-groups validity)

To examine the specificity of the NIHTB-CB to detect syndrome-specific performance, a 2-way mixed-model analysis of variance was conducted on NIHTB-CB test z scores with group as a between-participants factor and NIHTB-CB test as a within-participants factor; we also examined their interaction (figure 2). IQ was included as a repeated-measures varying covariate, and an IQ-by-test interaction term was included to allow the effect of IQ to vary by test. Because groups differed on FSIQ, covarying IQ aimed to clarify whether group results reflect phenotype-specific impairments or if they simply reflect globally poorer performance due to overall level of cognitive functioning. To avoid overcontrolling for the domain of interest (because NIHTB-CB domains overlap with components of IQ), verbal IQ was used as the covariate for the fluid NIHTB-CB test outcomes (DCCS, Flanker, LS, PC, and PSM), and nonverbal IQ was used as the covariate for crystallized NIHTB-CB tests (PVT and OR). To obtain effect sizes for pairwise comparisons, the Cohen d was calculated from the estimated marginal means from the model to account for the effects of IQ.

Figure 2
Figure 2 Profile plot of NIH Toolbox Cognitive Battery test z scores by group

The z scores on each test (representing group performance relative to the general population average performance) are shown, derived from the mixed-model analysis of variance, adjusted for IQ. A z score of 0 (horizontal line at top) represents the average performance in the general population normative sample. The z scores <0 represent the number of SDs below the general population average for the chronologic age band. Error bars represent 95% confidence intervals. *Comparison between fragile X syndrome (FXS) and intellectual disability (ID) of other or unknown cause. †Comparison between FXS and Down syndrome. *p < 0.05; **p < 0.01; ***p < 0.001; †p < 0.05; ††p < 0.01; †††p < 0.001. DCCS = Dimensional Change Card Sort; LS = List Sorting Working Memory; OR = Oral Reading and Recognition; PC = Pattern Comparison Processing Speed; PSM = Picture Sequence Memory; PVT = Picture Vocabulary.

There was a main effect of group on NIHTB-CB z scores (F2,238 = 4.90, p = 0.008), as well as a main effect of test on z scores (F6,1,067 = 17.26, p < 0.001). These were qualified by a significant group × test interaction (F12,1,067 = 3.68, p < 0.001) and a significant IQ × test interaction (F6,1,067 = 6.72, p < 0.001). Follow up Tukey tests showed that FXS performed worse on DCCS than OID [t(238) = 4.02, p < 0.001, d = 0.52] and worse than DS [t(238) = −3.04, p = 0.007, d = 0.39]. On Flanker, FXS performed worse than OID [t(238) = 3.45, p = 0.002, d = 0.45] and worse than DS [t(238) = −2.85, p = 0.01, d = 0.37]. These 2 EF test results supported the hypothesized EF impairment in FXS, although the DS impairment relative to OID was not supported. On PVT, FXS performed better than DS, fitting with expected language strength in FXS [t(238) = −2.77, p = 0.02, d = 0.36]. On PC, DS showed a poorer performance than OID, which approached significance [t(238) = 2.24, p = 0.07, d = 0.29]. There were no significant group differences on LS, PSM, or OR.

Discussion

This study provides the first comprehensive examination of the psychometric properties and feasibility of the NIHTB-CB for individuals with ID, with an initial focus on 2 of the most common genetic causes with robust translational research programs: FXS and DS. Overall, the NIHTB-CB has demonstrated strong potential for use as an objective, standardized outcome measure that can be confidently used in ID trials with participants with a mental age of 5 years or higher. Results of the study demonstrate very strong psychometrics for the Crystallized reasoning tests (PVT and OR) and good to excellent performance of Fluid reasoning tests, with more variation across FXS, DS, and OID groups for some measures (e.g., strong reliability for Flanker in FXS compared to DS, and vice versa for DCCS). Indeed, the Fluid Composite appeared to have more consistently strong reliability across conditions and a solid convergent association with FSIQ. Thus, it may be a good candidate outcome measure for studies seeking to examine broad nonverbal cognitive changes for individuals with a mental age of ≥5 years. Below a mental age of 5 years, feasibility was more variable across tests, indicating the need for further adaptations, scoring algorithms on developmental extensions, or new tests targeting these lower-functioning individuals.

The Supplemental Manual was compiled after hundreds of administrations of the NIHTB-CB to individuals with ID, and the study results on feasibility, reliability, and validity support its use. We encourage researchers and examiners planning to use the NIHTB-CB to follow these guidelines for the ID population.

Group comparisons demonstrated that the NIHTB-CB is sensitive to substantial EF deficits among individuals with ID in that all 3 groups performed relatively poorly on Flanker and DCCS. In particular, participants with FXS showed weakness in inhibitory control and attention and cognitive flexibility, in excess of their general cognitive level and compared to controls with other forms of ID. This aligns with previous research showing that boys with FXS are impaired in inhibitory control, set shifting, and planning relative to mental age–matched controls27 and that boys with FXS have impairments in inhibition and attention relative to mental age–matched controls and relative to children with DS.28 The hypothesized EF weakness in DS compared to OID (to a lesser extent than FXS) was not found; however, there have been some mixed results on EF in children with DS compared to children with other IDs.29,30 The low-scoring subgroup on DCCS are those who failed introduction to the switching portion of the test; for participants who cannot perform switching at the earliest level, the test may be less sensitive to variation in EF. Removing this subgroup from analyses strengthened the convergent validity correlation. This suggests that these individuals' level or lack of cognitive flexibility may not be captured by the introductory switching portion of DCCS. An extension of DCCS that does capture this construct in very young or low-functioning persons would have much value. The lower reliability of DCCS in the FXS group may also reflect this limitation in that participants with FXS overall scored very low on this test. Because anxiety and hyperactivity are common in FXS, it is possible that individual state interacts with the task complexity and cognitive flexibility to result in more variable performance over time in this group. When participants needed Flanker DEXT or DCCS DEXT, they were almost always able to perform the tests; however, a few issues remain to be worked out regarding these experimental versions, notably the ability to interpret these scores relative to the standard Flanker and DCCS scores. This study highlighted other concerns with the DEXT measures (e.g., difficulty may be ordered incorrectly, or portions are overly burdensome). Our results provide clear evidence that DEXT levels are necessary (and extremely feasible), especially in those with FXS and DS, but that further refinements and modifications are necessary.

To improve feasibility of LS, extra instructions and practice items were developed and used. The LS Age 3–6 experimental version with these additions is now available. Feasibility did improve after our initial pilot studies; however, the test remains challenging for this population, and some participants pass practice but get no test sequences completely correct. The limited feasibility and floored or low variability in raw scores suggest that a lower range of LS is necessary. A potential approach may be to give some credit on early sequences that are partially recalled or items recalled out of sequence because the construct of working memory builds on more basic short-term memory (e.g., SB-5 Working Memory index and Wechsler Working Memory indices).13,31

Both language tests, PVT and OR, had excellent performance in our samples of individuals with ID. Both demonstrated strong reliability and clear domain specificity with a much higher convergent than discriminant correlation. The NIHTB-CB was sensitive to expected language characteristics in that DS was impaired relative to FXS on PVT.32 These tests have an advantage over other language measures such as the PPVT-4 in that PVT and OR are brief (≈3 minutes each) but accurate, owing to CAT; the ability to obtain results with a brief assessment is especially important in individuals with frequent behavior or attention issues such as in those with ID.

Episodic memory is relevant to clinical trials, particularly for DS, in which memory impairments are well documented,33 and FXS, in which memory of sequential information is especially impaired.34 It is important to emphasize that despite these known weaknesses, PSM was among the highest of the test scores in each group (figure 2), suggesting that individuals may have compensatory strategies for performing well, perhaps such as use of the contextual information in the stories. Reliability was moderate and lower than that of the normative 3- 15-year-old study, although improved from our pilot study. The significant effect size in the PSM A-B group suggests nonequivalence of Forms A and B. Notably, in DS, ICCs were small and nonsignificant, suggesting that in its present form, PSM appears contraindicated as a separate outcome measure for this population.

The high rate of ceiling scores on PSM 3–4 suggests that mental age may not be a good indicator of start point on PSM in ID, at least at this mental age level. It is also possible that this age version is too easy in comparison to PSM scores on older versions; perhaps a person's score on 1 PSM version is not equivalent to that person’s score on another. Development of a CAT PSM test with the full range of version difficulties would likely simplify the testing process and yield more comparable and reliable scores.

PC showed adequate feasibility overall and good feasibility in FXS and OID, with sufficient feasibility at a mental age of ≥4 years. While the ICC was excellent, there was a small but significant practice effect, although smaller than that found in the normative sample.35 The most common reason for lack of feasibility on PC was an invalid alternating response pattern, especially common in DS. The task has an inherent challenge of understanding “same” and “different” while mapping this choice onto smiley face and frown face options. For participants without invalid response patterns, PC performs well in ID. On the basis of the feasibility challenges, we developed a new processing speed task, Speeded Matching, now available as an experimental version in the app. The task is to select the animal face among 3 foils that matches a target image. Future psychometric studies will provide more information about its performance.

The Fluid, Crystallized, and Cognitive Function composites demonstrated reliability results similar to those of the normative age 3 to 6 sample, with small practice effects in Fluid and Cognitive Function composites (although smaller than in the normative sample). Each composite was well correlated with FSIQ. Although not all participants were able to receive a composite (due to missing valid scores on some tests), when complete, the composites appear to perform well. The deviation method used to create these composites has a clear advantage over the current age-corrected standard scores, on which more than half of our sample obtained scores at the lower limit, currently set at 54. We are conducting analyses to identify the best option for composite scores below this floor (deviation approach vs extension of existing age-adjusted standard scores). These findings provide further evidence that test developers should consider and address the range and sensitivity of tests and scores for individuals with moderate to severe ID.24,36

This study has some important limitations that warrant consideration. Construct validity challenges are inherent with the ID population in that fully adequate convergent validity measures are not always available or they present their own feasibility and psychometric limitations. The lack of clear discriminant validity in most measures is another challenge. While discriminant correlations are generally desired to be markedly lower than the convergent correlations, in early development, domains of cognition (especially EFs) are thought to be unidimensional, with increasing differentiation of constructs occurring through early adulthood.37 In ID, there is likely even less differentiation between domains than in typically developing children. Our discriminant validity results are similar to normative 3- to 6-year-old results.38 Therefore, the Fluid Composite may be a good outcome measure choice on the basis of its psychometric performance and some limitations of subtest construct differentiation in this population. The study was also limited by sample size in evaluations of group-specific results. Future work with larger samples should provide more clarity about reliability and validity within individual ID subgroups.

The NIHTB-CB is a promising outcome measure for ID clinical trials and for many types of nonintervention observational studies. Although not originally intended for clinical use or for special education purposes, ongoing and future research may be done to explore such applications23 such as in the school psychology setting for accurate and feasible assessment of students with IDs. In addition, the Supplemental Manual developed from this study provides key guidance to examiners and researchers working with ID populations; the procedures on administration, test environment, fidelity, and scoring not only improve examiner familiarity and comfort but, more important, extend accessibility of the NIHTB-CB to more cognitively impaired or behaviorally challenged individuals.

Besides evaluating the NIHTB-CB as an appropriate assessment for ID in general, the present results demonstrate the sensitivity of the battery to known syndrome-specific cognitive phenotypes, such as the impairment of EFs in FXS relative to other ID and to DS. A critical remaining question is the degree to which the battery is sensitive to change, especially to effects of intervention. As an initial test of sensitivity to change, we are currently collecting longitudinal data from study participants to explore natural developmental changes within each NIHTB-CB test and the composites relative to measures already established as change sensitive such as the SB-5 and Vineland. The present results warrant the next step of evaluating the NIHTB-CB in ID for individuals down to a mental age of 5 years to demonstrate the treatment-specific sensitivity of the battery and to determine the degree to which measure gains reflect functional improvements in daily life. Below a mental age of 5 years, the NIHTB-CB performs more variably, and adaptations to the lower test ranges or scoring adaptations are needed for some measurement domains. Studies of the performance of the battery in older adults with ID are needed, especially focusing on those experiencing cognitive decline or dementia. Overall, the present validation results represent an important step toward providing an objective, scalable, and standardized method for successfully measuring cognition and tracking cognitive changes in ID.

Study funding

Funding provided by the National Institute of Child Health and Human Development (R01HD076189), Health and Human Services Administration of Developmental Disabilities (90DD0596), the MIND Institute Intellectual and Developmental Disabilities Research Center (U54 HD079125), and the National Center for Advancing Translational Sciences, NIH, through grant UL1 TR000002.

Disclosure

R. Shields, A. Kaat, F. McKenzie, A. Drayton, S. Sansone, J. Coleman, and C. Michalak report no disclosures. K. Riley has received compensation for consulting to Novartis regarding FXS clinical trials. E. Berry-Kravis has received funding (all funding including consulting goes to Rush University Medical Center) from Seaside Therapeutics, Novartis, Roche, Alcobra, Neuren, Cydan, Fulcrum, GW, Neurotrope, Marinus, Acadia, Zynerba, BioMarin, Ovid, Acadia, Yamo, Ionis, Lumos, Ultragenyx, Vtesse, Sucampo, and Mallinkcrodt Pharmaceuticals to consult on trial design or development strategies and/or to conduct clinical trials in FXS or other neurogenetic disorders, and from Asuragen Inc to develop testing standards and to do validation testing for FMR1 and SMN testing. R. Gershon and K. Widaman report no disclosures. D. Hessl has received funding for consulting from Novartis, Roche, Zynerba, Autifony, and Ovid pharmaceutical companies regarding FXS clinical trials. Go to Neurology.org/N for full disclosures.

Acknowledgment

The authors extend their appreciation to the families who gave their time and effort to participate in the study and to Leonard Abbeduto, LeAnn Baer, Ruth McClure Barnes, Kyle Bersted, Mikayla Brown, Ana Candelaria, Erin Carmody, Darian Crowley, Suzanne Delap, Randi Hagerman, Anne Hoffmann, Londi Howard, Paige Landau, Caroline Leonczyk, Michael Nelson, Jacklyn Perales, Lacey Pomerantz, Shanelle Rodriguez, Melanie Rothfuss, Ryan Shickman, Andrea Schneider, Haleigh Scott, Laurel Snider, Rachel Teune, Talia Thompson, Denny Tran, and Jamie Woods for their contributions to the study.

Appendix Authors

Table

Footnotes

  • Go to Neurology.org/N for full disclosures. Funding information and disclosures deemed relevant by the authors, if any, are provided at the end of the article.

  • The Article Processing Charge was funded by authors.

  • Editorial, page 507

  • Received August 14, 2019.
  • Accepted in final form October 31, 2019.

This is an open access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License 4.0 (CC BY-NC-ND), which permits downloading and sharing the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal.

References

  1. 1.↵
    1. Olusanya BO,
    2. Davis AC,
    3. Wertlieb D, et al
    . Developmental disabilities among children younger than 5 years in 195 countries and territories, 1990-2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet Glob Health 2018;6:e1100–e1121.
    OpenUrl
  2. 2.↵
    1. Lee AW,
    2. Ventola P,
    3. Budimirovic D,
    4. Berry-Kravis E,
    5. Visootsak J
    . Clinical development of targeted fragile X syndrome treatments: an industry perspective. Brain Sci 2018;8:E214.
    OpenUrl
  3. 3.↵
    1. Picker JD,
    2. Walsh CA
    . New innovations: therapeutic opportunities for intellectual disabilities. Ann Neurol 2013;74:382–390.
    OpenUrl
  4. 4.↵
    1. Budimirovic DB,
    2. Berry-Kravis E,
    3. Erickson CA, et al
    . Updated report on tools to measure outcomes of clinical trials in fragile X syndrome. J Neurodev Disord 2017;9:14.
    OpenUrl
  5. 5.↵
    1. Esbensen AJ,
    2. Hooper SR,
    3. Fidler D, et al
    . Outcome measures for clinical trials in Down syndrome. Am J Intellect Dev Disabil 2017;122:247–281.
    OpenUrl
  6. 6.↵
    1. Gershon RC,
    2. Wagster MV,
    3. Hendrie HC,
    4. Fox NA,
    5. Cook KF,
    6. Nowinski CJ
    . NIH Toolbox for assessment of neurological and behavioral function. Neurology 2013;80:S2–S6.
    OpenUrlAbstract/FREE Full Text
  7. 7.↵
    1. Weintraub S,
    2. Bauer PJ,
    3. Zelazo PD, et al
    . I, NIH toolbox cognition battery (CB): introduction and pediatric data. Monogr Soc Res Child Dev 2013;78:1–15.
    OpenUrlCrossRef
  8. 8.↵
    1. Casaletto KB,
    2. Umlauf A,
    3. Beaumont J, et al
    . Demographically corrected normative standards for the English version of the NIH Toolbox Cognition Battery. J Int Neuropsychol Soc 2015;21:378–391.
    OpenUrlCrossRefPubMed
  9. 9.↵
    1. Hessl D,
    2. Sansone SM,
    3. Berry-Kravis E, et al
    . The NIH Toolbox Cognitive Battery for intellectual disabilities: three preliminary studies and future directions. J Neurodev Disord 2016;8:35.
    OpenUrl
  10. 10.↵
    1. Berry-Kravis E,
    2. Lindemann L,
    3. Jonch AE, et al
    . Drug development for neurodevelopmental disorders: lessons learned from fragile X syndrome. Nat Rev Drug Discov 2018;17:280–299.
    OpenUrl
  11. 11.↵
    1. Parker SE,
    2. Mai CT,
    3. Canfield MA, et al
    . Updated national birth prevalence estimates for selected birth defects in the United States, 2004-2006. Birth Defects Res A Clin Mol Teratol 2010;88:1008–1016.
    OpenUrlCrossRefPubMed
  12. 12.↵
    1. Hunter J,
    2. Rivero-Arias O,
    3. Angelov A,
    4. Kim E,
    5. Fotheringham I,
    6. Leal J
    . Epidemiology of fragile X syndrome: a systematic review and meta-analysis. Am J Med Genet A 2014;164A:1648–1658.
    OpenUrlCrossRefPubMed
  13. 13.↵
    1. Roid G,
    2. Miller L
    . Stanford-Binet Intelligence Scales, 5th ed. San Antonio: Pearson; 2003.
  14. 14.↵
    1. Sparrow S,
    2. Cicchetti D,
    3. Saulnier C
    . Vineland Adaptive Behavior Scales, 3rd ed. San Antonio: Pearson; 2016.
  15. 15.↵
    1. Weintraub S,
    2. Dikmen SS,
    3. Heaton RK, et al
    . Cognition assessment using the NIH Toolbox. Neurology 2013;80:S54–S64.
    OpenUrlAbstract/FREE Full Text
  16. 16.↵
    1. Korkman M,
    2. Kirk U,
    3. Kemp S
    . NEPSY-II: A Developmental Neuropsychological Assessment. San Antonio: Pearson; 2007.
  17. 17.↵
    1. Conners C
    . Conners Kiddie Continuous Performance Test, 2nd ed. Toronto: Multi-Health Systems Inc.; 2015.
  18. 18.↵
    1. Wechsler D
    . Wechsler Preschool and Primary Scale of Intelligence, 4th ed. San Antonio: Pearson; 2012.
  19. 19.↵
    1. Roid G,
    2. Miller L,
    3. Pomplum M,
    4. Koch C
    . Leiter International Performance Scale, 3rd ed. Wood Dale: Stoelting Co; 2013.
  20. 20.↵
    1. Dunn L,
    2. Dunn D
    . Peabody Picture Vocabulary Test. 4th ed. San Antonio: Pearson; 2007.
  21. 21.↵
    1. Woodcock R,
    2. Johnston M
    . Woodcock-Johnson Tests of Achievement, 4th ed. Itasca: Riverside Publishing; 2014.
  22. 22.↵
    1. McKenzie FJ,
    2. Drayton A,
    3. Shields R, et al
    . NIH Toolbox Cognitive Battery supplemental administrator's manual for intellectual and developmental disabilities [online]. Available at: nihtoolbox.desk.com/customer/portal/articles/2981718. Accessed September 30, 2019.
  23. 23.↵
    1. Thompson T,
    2. Coleman JM,
    3. Riley K, et al
    . Standardized assessment accommodations for individuals with intellectual disability. Contemp Sch Psychol 2018;22:443–457.
    OpenUrl
  24. 24.↵
    1. Sansone SM,
    2. Schneider A,
    3. Bickel E,
    4. Berry-Kravis E,
    5. Prescott C,
    6. Hessl D
    . Improving IQ measurement in intellectual disabilities using true deviation from population norms. J Neurodev Disord 2014;6:16.
    OpenUrl
  25. 25.↵
    1. Akshoomoff N,
    2. Beaumont JL,
    3. Bauer PJ, et al
    . VIII. NIH Toolbox Cognition Battery (CB): composite scores of crystallized, fluid, and overall cognition. Monogr Soc Res Child Dev 2013;78:119–132.
    OpenUrl
  26. 26.↵
    1. Zelazo P,
    2. Bauer P
    . National Institutes of Health Toolbox cognition battery (NIH Toolbox CB): validation for children between 3 and 15 years. Monogr Soc Res Child Dev 2013;78:1–172.
    OpenUrlCrossRef
  27. 27.↵
    1. Hooper SR,
    2. Hatton D,
    3. Sideris J, et al
    . Executive functions in young males with fragile X syndrome in comparison to mental age-matched controls: baseline findings from a longitudinal study. Neuropsychology 2008;22:36–47.
    OpenUrlCrossRefPubMed
  28. 28.↵
    1. Wilding J,
    2. Cornish K,
    3. Munir F
    . Further delineation of the executive deficit in males with fragile-X syndrome. Neuropsychologia 2002;40:1343–1349.
    OpenUrlCrossRefPubMed
  29. 29.↵
    1. Pennington BF,
    2. Moon J,
    3. Edgin J,
    4. Stedron J,
    5. Nadel L
    . The neuropsychology of Down syndrome: evidence for hippocampal dysfunction. Child Dev 2003;74:75–93.
    OpenUrlCrossRefPubMed
  30. 30.↵
    1. Vicari S,
    2. Bellucci S,
    3. Carlesimo GA
    . Implicit and explicit memory: a functional dissociation in persons with Down syndrome. Neuropsychologia 2000;38:240–251.
    OpenUrlCrossRefPubMed
  31. 31.↵
    1. Wechsler D
    . Weschler Intelligence Scale for Children, 5th ed. Bloomington: Pearson; 2014.
  32. 32.↵
    1. Abbeduto L,
    2. Murphy MM,
    3. Cawthon SW, et al
    . Receptive language skills of adolescents and young adults with down or fragile X syndrome. Am J Ment Retard 2003;108:149–160.
    OpenUrlCrossRefPubMed
  33. 33.↵
    1. Godfrey M,
    2. Lee NR
    . Memory profiles in Down syndrome across development: a review of memory abilities through the lifespan. J Neurodev Disord 2018;10:5.
    OpenUrl
  34. 34.↵
    1. Dykens EM,
    2. Hodapp RM,
    3. Leckman JF
    . Strengths and weaknesses in the intellectual functioning of males with fragile X syndrome. Am J Ment Defic 1987;92:234–236.
    OpenUrlPubMed
  35. 35.↵
    1. Carlozzi NE,
    2. Tulsky DS,
    3. Kail RV,
    4. Beaumont JL
    . VI, NIH Toolbox Cognition Battery (CB): measuring processing speed. Monogr Soc Res Child Dev 2013;78:88–102.
    OpenUrlPubMed
  36. 36.↵
    1. Hessl D,
    2. Nguyen DV,
    3. Green C, et al
    . A solution to limitations of cognitive testing in children with intellectual disabilities: the case of fragile X syndrome. J Neurodev Disord 2009;1:33–45.
    OpenUrlCrossRefPubMed
  37. 37.↵
    1. Wiebe SA,
    2. Espy KA,
    3. Charak D
    . Using confirmatory factor analysis to understand executive control in preschool children: I, latent structure. Dev Psychol 2008;44:575–587.
    OpenUrlCrossRefPubMed
  38. 38.↵
    1. Mungas D,
    2. Widaman K,
    3. Zelazo PD, et al
    . VII, NIH Toolbox Cognition Battery (CB): factor structure for 3 to 15 year olds. Monogr Soc Res Child Dev 2013;78:103–118.
    OpenUrlPubMed

Letters: Rapid online correspondence

No comments have been published for this article.
Comment

REQUIREMENTS

If you are uploading a letter concerning an article:
You must have updated your disclosures within six months: http://submit.neurology.org

Your co-authors must send a completed Publishing Agreement Form to Neurology Staff (not necessary for the lead/corresponding author as the form below will suffice) before you upload your comment.

If you are responding to a comment that was written about an article you originally authored:
You (and co-authors) do not need to fill out forms or check disclosures as author forms are still valid
and apply to letter.

Submission specifications:

  • Submissions must be < 200 words with < 5 references. Reference 1 must be the article on which you are commenting.
  • Submissions should not have more than 5 authors. (Exception: original author replies can include all original authors of the article)
  • Submit only on articles published within 6 months of issue date.
  • Do not be redundant. Read any comments already posted on the article prior to submission.
  • Submitted comments are subject to editing and editor review prior to posting.

More guidelines and information on Disputes & Debates

Compose Comment

More information about text formats

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
Author Information
NOTE: The first author must also be the corresponding author of the comment.
First or given name, e.g. 'Peter'.
Your last, or family, name, e.g. 'MacMoody'.
Your email address, e.g. higgs-boson@gmail.com
Your role and/or occupation, e.g. 'Orthopedic Surgeon'.
Your organization or institution (if applicable), e.g. 'Royal Free Hospital'.
Publishing Agreement
NOTE: All authors, besides the first/corresponding author, must complete a separate Publishing Agreement Form and provide via email to the editorial office before comments can be posted.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.

Vertical Tabs

You May Also be Interested in

Back to top
Advertisement

Related Articles

Topics Discussed

Alert Me

Recommended articles