Abstract
Objective To describe cases presented by junior neurology residents and to evaluate resident diagnostic patterns to help address individual and systemic educational needs.
Methods For 6 academic years, details of all morning report cases assessed and presented by junior neurology residents were logged, including the resident's independent initial diagnostic impression. Cases were later revisited at subsequent morning reports to “close the loop” on a final diagnosis. We conducted retrospective review to quantify case demographics and to determine resident diagnostic accuracy based on prespecified localization pathways.
Results Demographic analysis included 1,472 cases; of these, 1,301 qualified for accuracy analysis due to diagnostic uncertainty at time of morning report. Non-neurologic etiologies represented 26.0% of cases. CNS etiologies were the majority (86.0%) of neurologic cases. The most frequent diagnoses were ischemic stroke and seizure. Overall resident diagnostic accuracy was 64.0%. Accuracy was similar between central and peripheral etiologies. Of 1,301 cases, 15.3% were overcalled as neurologic, while neurologic disease was rarely mistaken as non-neurologic (5.1%). Most diagnostic errors (49.1%) occurred when determining whether a case was neurologic. Where in the localization pathway errors occurred varied between etiologies.
Conclusion Overall diagnostic accuracy for neurology junior residents in our cohort was similar to prior work conducted in smaller samples. Analysis of errors, particularly at the critical “neurologic or non-neurologic” decision point, warrants further investigation. Close the loop methodology is simple to employ and can guide educational and quality initiatives to improve neurology resident clinical acumen.
Glossary
- CTL=
- close the loop;
- ED=
- emergency department;
- NOS=
- not otherwise specified;
- PNS=
- peripheral nervous system
Neurology residency programs strive for trainees to achieve core clinical competencies and to maximize the education derived from patient care. Endeavors to improve clinical acumen among residents are challenging, as diagnostic patterns are often difficult to capture and quantify. This is in part due to limited systematic data available regarding the types of cases seen in neurology residency. Prior work has captured broad etiologic categories of cases encountered by individual residents but has been restricted by small sample sizes or lack of follow-up data on patient outcomes.1
Furthermore, cognitive error research in neurology has also focused on small cohorts and is often subspecialized.2 We characterized the types of cases presented by junior neurology residents in morning report over a 6-year period with the aim of evaluating resident diagnostic patterns in order to identify individual and systemic educational needs.
Methods
Prospective Database Creation
All inpatients assessed and presented by on-call postgraduate year 2 residents during daily morning report for 6 academic years (July 2010 through June 2016) were captured in a case log. Case demographics collected included date of consultation, age and sex of the patient, encounter venue (e.g., emergency department [ED], medical floor, direct admission), and discharge disposition (e.g., home, medical floor, neurology floor). Details including history of the present illness, neurologic examination, and available laboratory/radiologic data were logged. The resident's initial diagnostic impression (prior to discussion with a neurology attending) was recorded. In subsequent morning report sessions, residents “closed the loop” by revisiting each previously presented case to record the final diagnosis when available. Cases in which a diagnosis was known at the time of initial presentation were excluded from accuracy analyses (e.g., consult for intracranial hemorrhage already visible on CT scan). Final diagnosis was defined as the diagnosis upon discharge or afterward if available. The Mount Sinai Hospital Institutional Review Board granted an exemption for this project.
Retrospective Case Classification
Retrospective review was conducted to categorize cases and determine accuracy of residents' initial diagnoses. Both initial impressions and final diagnoses were separately classified according to a prespecified localization and diagnostic pathway, designed to distinguished errors of localization from those of disease state (etiology). Our categorization scheme as shown in figure 1 is described as follows: initial impressions and final diagnoses were deemed to be either neurologic or non-neurologic, and neurologic cases were sorted by localization as either CNS or peripheral nervous system (PNS) and further subcategorized by etiology. By etiology we intend specific categories of neurologic disease; the CNS etiologies are listed in the table, with CNS not otherwise specified (NOS) reserved for more complex cases that did not clearly fit one category (e.g., autoimmune encephalitis, posterior reversible encephalopathy syndrome). PNS classifications are listed in the table, including PNS NOS (mostly peripheral vertigo). Non-neurologic cases were further classified as medical, psychiatric, or ophthalmologic. Frequencies, means, and SDs of particular localizations were calculated.
MS = multiple sclerosis; NMJ = neuromuscular junction; PNS = peripheral nervous system; SAH = subarachnoid hemorrhage; SDH = subdural hematoma.
Case Frequencies
Accuracy
Accuracy of the initial resident's diagnosis was categorized as fully accurate, partially accurate, or inaccurate. Cases with accuracy that was difficult to score were discussed among the authors, who would agree as a group on an accuracy score. Accuracy was judged by 2 raters with 94.8% interrater agreement (second rater assessing 154 cases in total, observed Cohen kappa 0.89). The partially accurate designation was applied to a small percentage of cases in which overall localization and general category of etiology were correct, but specific localization or etiology was not (e.g., a TIA diagnosis mistaken for a stroke; stroke recognized as etiology but proposed to be in the incorrect anatomic area; viral encephalitis mistaken for autoimmune encephalitis). Non-neurologic cases were considered partially accurate if the case was appropriately considered to be non-neurologic but the more precise designation was incorrect (e.g., an ophthalmologic diagnosis mistaken for medical).
Results
Case Demographics
A total of 1,472 morning report inpatient cases were included in demographic analysis, 1,301 of which had diagnosis unknown at presentation, and thereby were included in the close the loop (CTL) cohort. Of all cases, 58.1% were female. Mean age was 57.5 years (SD 19.1, range 9–96). A total of 72.2% of cases were from the ED, and 19.3% from inpatient floors (primarily internal medicine services). See the table for case frequencies.
Accuracy: Overall
The overall diagnostic accuracy was 64.0% with an additional 4.8% cases deemed partially accurate. Among cases with a final neurologic diagnosis, accuracy was higher at 67.5% (with an additional 5.8% partially accurate); CNS (67.9% accurate, 6.2% partially accurate) and PNS cases (66.3% accurate, 3.4% partially accurate) had similar overall accuracy. Accuracy for medical cases was 62.3% (2.0% partially accurate); for psychiatric cases, accuracy was 54.5% (partially accurate 3.0%).
Accuracy: Neurologic vs Non-Neurologic
Most diagnostic errors (49.1%) occurred at the neurologic vs non-neurologic distinction. Cases were more likely to be “overcalled” as neurologic (15.3% of all cases) rather than missed as neurologic (5.1% of all cases) (figure 2A). Determination of a neurologic diagnosis by junior residents carried a 94.9% sensitivity, 59.1% specificity, 84.8% positive predictive value, and 82.7% negative predictive value. Of all medical cases, 36.1% were initially deemed neurologic. Conversely, of cases initially deemed medical, 19.7% were actually neurologic.
Resident accuracy depicted as targets, with degrees of accuracy shown as being increasingly off-target. Accurate diagnoses are shown in green; partially accurate in yellow; and wrong diagnoses shown in orange and red. The wrong diagnoses are subdivided into errors at different stages along the diagnostic pathway, with errors in distinguishing neurologic from non-neurologic disease comprising the outermost ring. (A) Accuracy target for the full CTL cohort (n = 1,301). (B) Ischemic stroke cases (n = 237). (C) Headache cases (n = 88). PNS = peripheral nervous system.
Similarly, 38.4% of true psychiatric cases were initially deemed neurologic whereas 11.5% of cases initially classified as psychiatric were in fact neurologic disorders.
Accuracy: CNS vs PNS
An error was made at the CNS/PNS decision point in 3.4% of all cases (10.9% of all errors; figure 2A). Of incorrect cases that were correctly identified as neurologic, 22% of these were missed at the CNS/PNS level.
Accuracy: Etiologies
Of all incorrect cases, 38.0% were missed at the etiologic level (correctly categorized as CNS or PNS, but incorrect etiology within either CNS or PNS). Accuracies for the 5 most common etiologies are as follows: ischemic stroke (75.5% accurate/4.8% partially accurate), seizure (79.6%/2.0%), headache (55.7%/6.8%), neuropathy (50%/3.0%), and CNS neoplastic (57.1%, 2.0%). Errors for each etiologic category occurred at different points in the diagnostic pathway. For example, in the cases with a final diagnosis of ischemic stroke, 20.7% of cases were incorrect. Among all ischemic stroke cases, 4.2% were missed at the neurologic vs non-neurologic level, 3.0% missed at the CNS/PNS level, and 13.5% missed at the etiologic level (figure 2B). For comparison, in the primary headache group, 2.3% were missed at the neurologic/non-neurologic level, 2.3% at CNS/PNS level, and 33.0% of all headache cases were misdiagnosed among specific CNS etiologies (figure 2C).
Discussion
Morning report is an essential venue for clinical learning in residency. Over 6 academic years, we have used this forum to categorize cases assessed by neurology residents and to critically evaluate patterns of diagnostic acumen. Nearly one-third of cases presented by neurology residents were ultimately deemed to be non-neurologic, most of which were medical or psychiatric in etiology. The majority of neurologic cases localized to the CNS, with the most common diagnoses being stroke and seizure. This is similar to prior work in which single residents tracked all cases seen during their residency training.1,3,4
Overall junior neurology resident initial diagnostic accuracy when presenting their initial impressions at morning report was 64%, which is congruent with 2 prior studies.5,6 Residents tended to overcall cases presented in morning report as neurologic but rarely missed true neurologic disease. Despite the significantly higher quantity of CNS disorders presented, resident accuracy in morning report cases was similar between CNS and PNS localizations. Patterns where error was likely to occur varied between etiologies. For example, headache was more likely than ischemic stroke to be missed at the more nuanced etiologic categorization, while stroke was more likely than headache to be missed as non-neurologic.
Several limitations of our study warrant consideration. First, a resident's true independent diagnostic assessment is difficult to capture, as residents frequently interact with coresidents, ED or other attendings, and radiologists in the course of initial consultations. We note also that the amount of supporting clinical data (e.g., laboratory studies, imaging) available at the time of first diagnostic impression varied across cases and the natural diagnostic evolution that occurs during a hospitalization may hinder determination of true accuracy. Furthermore, although we sought to determine diagnostic accuracy, there is often no gold standard for final diagnosis. These final determinations reflect best clinical judgment at the conclusion of the hospitalization or evaluation of each case. There is also challenge in categorizing complex cases into our prespecified diagnostic localizations and etiologies (e.g., vasculitis could be categorized as ischemic stroke, but also as a neuroinflammatory or headache syndrome). Furthermore, not every diagnostic category could be parsed with identical levels of granularity. Mount Sinai is a large tertiary care hospital located in New York City and thus the cases seen by residents in this urban setting may not match those seen at community hospitals or in suburban or rural settings. Finally, our data include inpatient consultations only, thus diagnoses more often seen in our outpatient clinics are necessarily underrepresented.
Importantly, analyses based on morning report presentations are subject to selection biases, as the cases residents choose for presentation may be more complex and may overrepresent neurologic disease, as junior residents are presenting to neurology attendings with the aim of learning from their expertise in localization and management. Our results and conclusions need to be considered with this limitation in mind—the cases presented in morning report reflect only a subset of all cases seen and assessed—thus patterns of clinical judgement captured in this study may not reflect diagnostic acumen more broadly. Although our data do not capture all patients evaluated by residents, we have included the entirety of the cases used for clinical teaching in the morning report venue. The analysis spans 6 academic years and more than 1,000 cases with consistency across years, which supports the internal reliability of the results. Future studies could take our CTL approach forward to address the issue of selection bias, for instance by performing a shorter study of several months' duration closing the loop on all cases, seen by all residents, in every clinical venue. This approach would complement our current study, which evaluated longitudinally all the cases presented in one educational venue, morning report, over several years.
Systematically closing the loop on morning report cases has provided important short-term feedback for residents in the group setting of morning report, allowing for expanded educational discussion facilitated by faculty and senior residents and fostering a culture of continuity and diagnostic humility. Data on a per-resident basis are used in feedback sessions between individual residents and the resident program director to develop a personalized educational plan. More broadly, the CTL methodology in morning report was a component of our adult learning theory–based curriculum, which was found to have a positive effect on Residency In-service Training Examination scores.7 Further work is needed to assess both subjective (e.g., survey) and objective measures of improved diagnostic acumen in future postgraduate years. We also plan to analyze cases by specific etiology, patient demographics, time of year, patient disposition, and consult venue to pursue multivariate predictors of resident errors that could inform curricular changes. Qualitative examination of incorrect cases may also allow for assessment of the role of cognitive biases (e.g., anchoring or framing) on initial diagnostic accuracy on the group level. CTL methodology is simple to employ, low cost, and can be replicated and expanded upon in other programs. Our findings can guide educational and quality initiatives to improve clinical acumen among neurology residents.
Study Funding
No targeted funding reported.
Disclosure
Dr. Krieger reports consulting or advisory work with Biogen, EMD Serono, Genentech, Genzyme, Mallinckrodt, MedDay, Novartis, Teva, and TG Therapeutics, and nonpromotional speaking with Biogen, EMD Serono, and Novartis. Go to Neurology.org/N for full disclosures.
Acknowledgment
The authors thank Dr. Sarah Zubkov and Christine Hannigan; Prof. Emilia Bagiella for data analysis guidance; Ana Ferrer and Karin Cook for assistance with data visualization; the many Mount Sinai residents and attendings who contributed to the dataset; and the patients.
Appendix Authors
Footnotes
Go to Neurology.org/N for full disclosures. Funding information and disclosures deemed relevant by the authors, if any, are provided at the end of the article.
- © 2020 American Academy of Neurology
Disputes & Debates: Rapid online correspondence
REQUIREMENTS
If you are uploading a letter concerning an article:
You must have updated your disclosures within six months: http://submit.neurology.org
Your co-authors must send a completed Publishing Agreement Form to Neurology Staff (not necessary for the lead/corresponding author as the form below will suffice) before you upload your comment.
If you are responding to a comment that was written about an article you originally authored:
You (and co-authors) do not need to fill out forms or check disclosures as author forms are still valid
and apply to letter.
Submission specifications:
- Submissions must be < 200 words with < 5 references. Reference 1 must be the article on which you are commenting.
- Submissions should not have more than 5 authors. (Exception: original author replies can include all original authors of the article)
- Submit only on articles published within 6 months of issue date.
- Do not be redundant. Read any comments already posted on the article prior to submission.
- Submitted comments are subject to editing and editor review prior to posting.