Abstract
Objective To enhance residency education by implementing the 6 principles of adult learning theory (ALT) in a large academic neurology residency program.
Methods We implemented a set of curricular interventions aimed at Resident Education through Adult Learning in Neurology (REAL Neurology), in a large, academic neurology residency program. Interventions included didactic reform, increasing resident-as-teacher activities, and enhancing residents' interaction. The primary outcome was the change in mean Residency In-service Training Examination (RITE) percentile between the preintervention and postintervention cohorts, adjusting for US Medical Licensing Examination step 1 and 2 score. Other analysis included evaluating the effect of the duration of intervention exposure on outcome and evaluating the intervention effect on the proportion of advanced performers.
Results A total of 134 RITE score reports were evaluated (87 preintervention and 47 postintervention). The mean RITE score percentile postintervention was 11.7 points higher than preintervention (adjusted, longitudinal analysis: fit linear mixed model, p < 0.0001). Postgraduate year 3 learners who had 1 and 2 years of exposure scored 13.4 and 18.9 points higher than those with no exposure at all, respectively (analysis of covariance, p = 0.04). The adjusted odds of better performance with REAL Neurology was 5.77 (ordinal logistic regression, 95% confidence interval 2.37–14.07, p < 0.05).
Conclusion This study evaluated the efficacy and feasibility of an ALT-based curricular program in neurology education. The results show robust and sustainable benefit for residents in training without imposing a financial or logistical burden on programs. REAL Neurology could serve as a model for curricular reform in other programs across subspecialties.
Glossary
- AAN=
- American Academy of Neurology;
- ALT=
- adult learning theory;
- ANOVA=
- analysis of variance;
- GME=
- graduate medical education;
- PGY=
- postgraduate year;
- REAL Neurology=
- Resident Education through Adult Learning in Neurology;
- RITE=
- Residency In-service Training Examination;
- USMLE=
- US Medical Licensing Examination
There is well-deserved emphasis on remodeling graduate medical education (GME) programs to develop the role of residents as teachers. The American Council for Graduate Medical Education recognized the importance of well-structured, goal-oriented educational curricula with the introduction of the educational milestones.1 Solid foundation in the philosophy of learning is necessary for such programs to be efficacious and sustainable in an ever-changing health care system.
Many learning theories have been incorporated into medical education.2 Perhaps most notable is Malcolm Knowles' pivotal adult learning theory (ALT).3 Knowles proposed 6 characteristics that distinguish adult learning from pedagogy, those being readiness, self-directedness, internal motivation, relevance, goal orientation, and implementation of life experiences. Knowles' framework has spurred educational research and expanded the educational initiatives for adult learners in medicine.4,–,6 Even so, there are scarce data about ALT-based educational curricula in GME programs generally, and in neurology specifically.
We hypothesized that the implementation of ALT-based curriculum is an effective and feasible intervention to enhance residency education and improve objective measures of residents' knowledge. We utilized change in Residency In-service Training Examination (RITE) scores as our primary outcome.
Methods
Study population
Neurology residency in the United States is a 3-year advanced training program of postgraduate year (PGY) 2 through 4 residents. Our program is at a large academic center that expanded during the study period, resulting in 6–8 residents per PGY class for a total of 18–24 residents. Performance of all residents on the annual RITE was divided based on the year of curricular implementation into preintervention and postintervention cohorts (2010–2014 and 2015–2016, respectively). The study was reviewed and exempted by the local institutional review board.
Intervention
Prior to program implementation in 2015, the residency educational curriculum was delivered in a series of 60-minute didactic sessions, the format and content of which was shaped by faculty educators' experience and preference. There was no formal teaching role for residents. Resident Education through Adult Learning in Neurology (REAL Neurology) is a resident-led ALT-based curricular reform, designed by the authors based on their educational experience, in collaboration with the stakeholders in faculty and learners. The objective of the program is to recalibrate the educational process towards the learners, by implementing 3 prespecified ALT-based interventions: didactic reform, enhancing the residents' teaching roles, and increasing resident engagement in educational activities (table).
Resident Education through Adult Learning in Neurology (REAL Neurology) interventions
Outcome assessment tools
Our primary outcome was RITE performance. The RITE is an annual self-assessment tool developed by the American Academy of Neurology (AAN) to gauge resident knowledge of neurology and neuroscience and identify areas for potential growth.7 Results are reported as percentiles comparing each resident's performance to PGY-matched nationwide test-takers.
Statistical methods
A linear mixed model was fitted with RITE percentiles as the outcome and curricular intervention as fixed effect, after adjusting for residents' US Medical Licensing Examination (USMLE) step 1 and 2 scores, using F test, with α level set to 5%. Kolmogorov-Smirnov test was used to confirm the normal distribution of RITE and USMLE scores (p > 0.05). We accounted for within-subject correlation using longitudinal analysis. We imputed missing data with available case analysis. We assessed baseline USMLE scores of the residents by one-way analysis of variance (ANOVA) and t test. The correlation of USMLE scores to RITE scores was evaluated by Pearson correlation. We further categorized RITE performance outcome into advanced (>75th percentile), average (25th–75th percentile), and poor (<25th percentile). We assessed category proportions by ordinal logistic regression after adjusting for USMLE step 1 and step 2 scores. Finally, to assess the effect of accumulative exposure to the intervention, we performed a subgroup analysis of outcome of 24 learners; 8 from each of 3 consecutive PGY3 classes (2014–2016). As the intervention was implemented in 2015, they had 0, 1, or 2 years of exposure to the new curriculum. An adjusted linear model was fitted with years of exposure to intervention as covariate, and type III analysis of effects was reported. All analyses were performed using SAS version 9.4 (SAS Institute, Cary, NC).
Results
Baseline characteristics
There were 134 RITE score reports: 87 in the preintervention and 47 in the postintervention cohort. The mean USMLE step 1 and step 2 scores for all learners in both cohorts was 227 (SD 15.9) and 239 (SD 15.86), respectively, with no difference between the academic years (one-way ANOVA, p = 0.55 and 0.42, respectively). USMLE step 1 and 2 scores correlated with residents' RITE percentiles at all PGY levels (Pearson correlation, USMLE step 1; PGY2: R = 0.61, PGY3: R = 0.53, and PGY4: R = 0.66, p < 0.001; USMLE step 2; PGY2: R = 0.50, PGY3: R = 0.49, and PGY4: R = 0.60, p < 0.001).
Primary outcome
The mean RITE score percentile for the postintervention cohort was significantly higher than the preintervention cohort (68.3 vs 53.0, p = 0.0002) (figure). This remained statistically significant after adjusting for USMLE scores and accounting for repeated measures (absolute difference 11.73, p < 0.0001). A similar significant difference was seen in each PGY class separately after adjusting for USMLE scores (PGY2: 70.1 vs 55.6, p = 0.00484; PGY3: 65.6 vs 48.6, p = 0.0192; PGY4: 69.2 vs 54.9, p = 0.030933) (figure).
The RITE score percentiles in the pre and post REAL Neurology cohorts are shown. REAL Neurology significantly increased the mean RITE score percentile for all the postintervention learners together, even after adjusting for US Medical Licensing Examination (USMLE) scores and repeated measures (p < 0.0001) (A) as well as when comparing each postgraduate year (PGY) level separately and controlling for USMLE scores (PGY2: p = 0.00484; PGY3: p = 0.0192; PGY4: p = 0.030933). (B) The analysis of examination performance distribution based on RITE percentile (poor <25th, average 25th–75th, and advanced >75th percentiles) reveals a significant shift towards higher performance after REAL neurology implementation (odds ratio 5.77, 95% confidence interval [CI] 2.37–14.07, p = 0.0001) (C). Error bars represent 95% CI; α level was set at 0.05. *p < 0.05, **p < 0.01, ***p < 0.001, ****p < 0.0001.
Upward shift in examination performance
Postintervention analysis revealed a significant increase in the proportion of residents with advanced performance and a corresponding reduction in average or poor performance (advanced: 44.7% vs 17.2%, average: 51.0% vs 67.8%, and poor: 4.0% vs 14.9%) (ordinal logistic regression, adjusted odds 5.77, 95% confidence interval 2.37–14.07, p = 0.0001) (figure).
Positive accumulative impact of REAL Neurology
In the subgroup of PGY3 learners, longer duration of exposure was associated with a higher RITE score percentile after controlling for USMLE scores. Residents who had 1 year (PGY3 in 2015) vs 2 years (PGY3 in 2016) of exposure respectively scored 13.4 and 18.9 percentiles higher than those who had no exposure at all (PGY3 in 2014) (analysis of covariance, p = 0.0357).
Discussion
This study systemically evaluates the efficacy and feasibility of an ALT-based educational model. We found a clear and sustainable effect on objective educational outcomes at all PGY levels during training, which was augmented by longer exposure in a dose-dependent fashion. REAL Neurology did not simply expand the proportion of average learners at the expense of advanced learners. There was an upward learning shift into higher tiers of performance with the largest gain being in the advanced performance tier, which increased by more than double the baseline.
We chose RITE scores, a tool developed by the AAN for knowledge self-assessment, as our primary efficacy endpoint. RITE performance positively correlates with American Board of Psychiatry and Neurology pass rates in neurology.8 Juul et al.8 estimated that scoring <59th percentile on RITE was associated with 72% board pass rate, while scoring 65th–69th percentile was associated with 98% board pass rate. The mean percentile of our residents increased from 53 to 68.3 after REAL Neurology implementation.
Any institutional change may be met with resistance. In our experience, this was mitigated by engaging all the educational stakeholders via strong collaboration between the residency program director and the chief resident of academic affairs. While residents' attitude about the curricular changes was not formally assessed, they expressed increased motivation and satisfaction on informal surveys.
We emphasize that this educational intervention was feasible. The knowledge necessary for curricular reform is available in academic workshops, open-source courses, or self-teaching. The technological upgrades in teleconferencing and audience response systems are available, often in free versions, utilizing ubiquitous smartphones. There was no change in total didactic time and no additional costs to the department (supplemental data, links.lww.com/WNL/A602).
Several limitations warrant specific discussion. Confounding factors may have affected the outcome, like learners' attitudes or awareness of the curricular reform process, which may result in increased self-study. This increased engagement could be both a confounder as well as an outcome of the intervention. Use of a case-control trial design could clarify these concerns. We used the preintervention cohort as a historic comparator as it was logistically prohibitive to divide the residents within the same program into 2 curricular cohorts. In addition, only 2 years of outcome data were available at the time of writing this manuscript. The effect on residents who were exposed to the intervention for the full 3 years of their training is unknown. The RITE measures knowledge acquisition in the Kirkpatrick hierarchy of educational outcomes, which is not the only objective of educational curricula.9 The intervention's effect on more advanced hierarchical domains, such as patient care or clinical outcomes, is unknown and could be explored in further studies. Finally, although the basic principles that went into this curricular reform are generalizable, this intervention was implemented in a single program of one specialty. Considerations for the design of future trials should include larger samples with contemporaneous control groups and representation from multiple specialties.
The cornerstone of this project is an implementation of the 6 principles of ALT, while engaging residents as leaders in education. For REAL Neurology, this specifically required a curricular refinement, a learner-focused session structure, and more resident teaching activities. REAL Neurology can potentially serve as model for other forms of resident-led ALT-based curricular reform.
Author contributions
Dr. Shoirah designed and administered the intervention, conducted the described research, coauthored the manuscript, and supervised the data analysis. Dr. Ntranos conducted and supervised the data analysis and coauthored the manuscript. Dr. Brandstadter administered the intervention and coauthored the manuscript. Y. Liu conducted the statistical analysis. Elisha Medina conducted data collection and entry. J. Kwan conducted data collection and entry. Dr. Krieger designed and conducted the described research and provided mentorship and supervision for the statistical analysis and manuscript writing including review, revision, and editing.
Study funding
No targeted funding reported.
Disclosure
The authors report no disclosures relevant to the manuscript. Go to Neurology.org/N for full disclosures.
Footnotes
↵* These authors contributed equally to this work.
Go to Neurology.org/N for full disclosures. Funding information and disclosures deemed relevant by the authors, if any, are provided at the end of the article.
- © 2018 American Academy of Neurology
Disputes & Debates: Rapid online correspondence
REQUIREMENTS
If you are uploading a letter concerning an article:
You must have updated your disclosures within six months: http://submit.neurology.org
Your co-authors must send a completed Publishing Agreement Form to Neurology Staff (not necessary for the lead/corresponding author as the form below will suffice) before you upload your comment.
If you are responding to a comment that was written about an article you originally authored:
You (and co-authors) do not need to fill out forms or check disclosures as author forms are still valid
and apply to letter.
Submission specifications:
- Submissions must be < 200 words with < 5 references. Reference 1 must be the article on which you are commenting.
- Submissions should not have more than 5 authors. (Exception: original author replies can include all original authors of the article)
- Submit only on articles published within 6 months of issue date.
- Do not be redundant. Read any comments already posted on the article prior to submission.
- Submitted comments are subject to editing and editor review prior to posting.