Abstract
Objective To determine whether NeuroBytes is a helpful e-Learning tool in neurology through usage, viewer type, estimated time and cost of development, and postcourse survey responses.
Background A sustainable Continuing Professional Development (CPD) system is vital in neurology due to the field's expanding therapeutic options and vulnerable patient populations. In an effort to offer concise, evidence-based updates to a wide range of neurology professionals, the American Academy of Neurology (AAN) launched NeuroBytes in 2018. NeuroBytes are brief (<5 minutes) videos that provide high-yield updates to AAN members.
Methods NeuroBytes was beta tested from August 2018 to December 2018 and launched for pilot circulation from January 2019 to April 2019. Usage was assessed by quantifying course enrollment and completion rates; feasibility by cost and time required to design and release a module; appeal by user satisfaction; and effect by self-reported change in practice.
Results A total of 5,130 NeuroBytes enrollments (1,026 ± 551/mo) occurred from January 11, 2019, to May 28, 2019, with a median of 588 enrollments per module (interquartile range, 194–922) and 37% course completion. The majority of viewers were neurologists (54%), neurologists in training (26%), and students (8%). NeuroBytes took 59 hours to develop at an estimated $77.94/h. Of the 1,895 users who completed the survey, 82% were “extremely” or “very likely” to recommend NeuroBytes to a colleague and 60% agreed that the depth of educational content was “just right.”
Conclusions NeuroBytes is a user-friendly, easily accessible CPD product that delivers concise updates to a broad range of neurology practitioners and trainees. Future efforts will explore models where NeuroBytes combines with other CPD programs to affect quality of training and clinical practice.
Glossary
- AAN=
- American Academy of Neurology;
- CPD=
- Continuing Professional Development;
- CME=
- Continuing Medical Education
With the rapid expansion of diagnostic procedures, translational discoveries, and novel therapeutics in the field of medicine comes an acute need for relevant, high-yield updates in Continuing Professional Development (CPD).1,2 There is large variability in how CPD is delivered, ranging from annual meetings to virtual webinars, each seeking to close gaps between personal competencies and health care standards.3 These learning gaps become evident through needs assessments, quality management, and inconsistent patient care.4,5 CPD systems should aim to deliver cohesive, timely information to health care providers with the goal of improving patient safety and advancing evidence-based practices.6,7 A sustainable, cohesive CPD system is vital in neurology due to the field's quickly evolving practices, expanding therapeutic landscape, and increasing number of subspecialties. Despite this, there is limited research that evaluates effective, sustainable CPD practices in neurology.8
The American Academy of Neurology (AAN) launched a series of NeuroLearn programs in 2012 to provide evidence-based e-Learning modules featuring a variety of clinical, professional, and practice-based topics. Each NeuroLearn course requires approximately 2 hours to complete, provides 2 Continuing Medical Education (CME) credits, and consists of 3 multimedia video modules (10–25 minutes each) with a summative postcourse assessment. The courses can be completed over one or multiple sessions; however, the dedicated time required for module completion limits the program's usability and audience.9 In addition, NeuroLearn modules cannot be rapidly revised to reflect new changes in the field due to the long and detailed process required for module development.
In an effort to offer more concise, readily available educational videos to a broader population of learners, a new series, NeuroBytes, was developed. NeuroBytes are brief (<5 minutes) videos that provide focused, clinically relevant topic updates or reviews. Summative assessments are not included and no CME credit is offered. NeuroBytes can be easily accessed by AAN members on any device (e.g., smartphone, iPad, computer) and completed in one sitting. Few studies have explored the usefulness of updated CPD practices in neurology and even fewer studies have reviewed the effectiveness of training compared to returns on investing in training programs. In this study, we aimed to (1) assess the usability of NeuroBytes, (2) describe the target audience, (3) estimate the time and cost required for module development, and (4) evaluate its impact as an e-Learning educational tool.
Methods
Ethics
This study was reviewed and approved by the institutional review board at Weill Cornell Medical College.
Program Description
NeuroBytes Beta Test (August 2018–December 2018)
NeuroBytes was developed in response to (1) the successes seen with online delivery of NeuroLearn, (2) member feedback requesting shorter modules, and (3) concerns about the expense and burden of NeuroLearn course development. NeuroBytes are concise (<5 minutes) multimedia updates that review specific clinical, professional, or advocacy topics in neurology. NeuroBytes are readily available to registered AAN members but do not offer CME accreditation or postcourse assessments. A beta testing period was planned from August 2018 to December 2018 to assess the feasibility of this new method of CPD. Initial NeuroBytes included both repurposed content from expiring NeuroLearn courses as well as de novo NeuroBytes. To ensure high quality of the content produced, a standardized script was used for developing each module outlining (1) the module learning objective, (2) narration including introduction, topic content, clinical explanation, and call to action, (3) suggested graphics, and (4) technical aspects to align with best practices in e-Learning. For each NeuroByte, a project champion was selected from the NeuroBytes Working Group. This individual worked with a faculty expert to develop content and received training in e-Learning delivery. The completed NeuroBytes underwent iterative peer review by a faculty expert, editorial review by the project champion, and technical review by an expert in online learning.
NeuroBytes Pilot (January 2019–April 2019)
Following beta testing, a pilot NeuroBytes program, “Fundamental Concepts of Alzheimer's Risk Reduction,” was launched on January 11, 2019. Content followed a structured curriculum, which aimed to (1) deliver content at relevant times and (2) ensure that the NeuroBytes product included the breadth of clinical neurology, subspecialty areas, practice management, and advocacy required for evidence-based practice. Each month the specific titles and topics for NeuroBytes modules were solicited, proposed, and selected by the NeuroBytes Working Group. Topics were chosen based on AAN's curriculum calendar to ensure major specialty and subspecialty areas in neurology were addressed over the course of a year. The site released 8 subsequent videos based on user recommendations through April 9, 2019 (table 1). Video topics included hyperacute management of ischemic stroke, clinical approach to localization and diagnosis of ptosis, and neurologic complications of immune checkpoint inhibitors. NeuroBytes were distributed broadly through email, social media, and the AAN Learning and Management System.
NeuroBytes Program Data Including Course Enrollment, Usage, and User Satisfaction
Data Collection and Analysis
Usage data included NeuroBytes beta testing between August 2018 and December 2018 and NeuroBytes pilot programs between January 2019 and April 2019 (n = 9). NeuroBytes data were collected from January 11, 2019, to May 28, 2019. Usability was determined by the total number of enrollments and proportion of users who completed the course. An enrollment was defined as a single user launching an online course; users have the ability to review NeuroBytes videos multiple times, which was recorded as additional enrollments. Completion of a NeuroBytes course required the user to watch a portion of the module and submit a postmodule evaluation. Total video observation time was monitored for each user and observation times that either (1) were recorded as 00:00 or (2) exceeded 15 minutes were considered outliers and not included in analysis. Users “in progress” watched a portion of the module, clicked around within the module, or watched the entire module without completing the postcourse survey. The remainder of the users who launched the video without watching were categorized as “not started.” Target audience was assessed by user membership. Feasibility was assessed through the estimated cost for module design and release.
Impact was evaluated through postvideo surveys and self-reported impact on viewer practice.10 The NeuroBytes survey consisted of 3 multiple choice questions and one optional free response. The 3 multiple choice questions included the following: (1) “How likely would you be to recommend NeuroBytes to a colleague?” (extremely likely, very likely, somewhat likely, slightly likely, not at all); (2) “Was the information in this video helpful for your practice?” (this will help change how I manage many of my patients, this will help change how I manage a few of my patients, this was helpful but won't change my practice, this would be helpful if improved, this information was not helpful); and (3) “The depth of the educational content in this video was:” (far too basic for my level of training, a little more basic than I had hoped for, just right, somewhat more detailed than I had liked, far too detailed for my level of training). The optional free response asked users “What other topics would you like to see covered as a NeuroBytes video?”
Data Availability
Anonymized data were shared with qualified investigators.
Results
Aim 1: Assess the Usability of NeuroBytes
A total of 5,130 individuals enrolled in the NeuroBytes online learning program from January 11, 2019, to May 28, 2019 (1,026/month). The median enrollment was 588 per module (interquartile range, 194–922). Of the 5,130 enrollments, 1,884 individuals completed the course and postcourse survey (37%), 3,043 remained in progress (59%), and 203 launched, but did not watch, the video (4%). The 59% of participants who remained in progress clicked around within the module without starting the video, watched a portion of the module, or watched the entire module without completing the postcourse survey. These in progress users spent an average of 1 minute and 28 seconds on a video (range 1:14–1:39), which equated to a mean of 40% of the total length of the video (range 28%–62%). Completion rates were highest for advance practice providers (54%), followed by retired senior neurologists (47%), neurologists (42%), residents (37%), students (25%), and nonmembers (25%) (table 1).
Aim 2: Describe the Target Audience of NeuroBytes
The majority of NeuroBytes viewers were neurologists (54%), followed by junior neurologists in training (26%) and students (8%). The remainder of viewers included advance practice providers (4%), retired senior neurologists (3%), interns (2%), nonmembers (2%), researchers (<1%), physician affiliates (<1%), business administrators (<1%), and honorary members (<1%; table 1). No significant difference in target audience was observed when comparing the type of viewer among each of the 9 NeuroBytes modules (table 2).
Enrollment Breakdown by Title for NeuroBytes
Aim 3: Estimate the Time and Cost Required for NeuroBytes Module Development
The average time and cost required to launch a NeuroBytes product was 59 hours and $4,618: 8 hours and $713 for design (phase I), 31 hours and $2,450 for development (phase II), 19 hours and $1,360 for production (phase III), and 1 hour and $95 for course project closure (phase IV). Faculty time was not compensated for and was not included as part of the total cost. This totals to approximately $77.94/h and $0.90/user for module development (table 1).
Aim 4: Evaluate the Impact of NeuroBytes as an e-Learning Educational Tool
Of the 1,895 users who completed the NeuroBytes post-video survey, 82% of respondents were “extremely likely” or “very likely” to recommend NeuroBytes to a colleague. Sixty percent of completed users agreed that the depth of educational content in the video was “just right,” with 29% responding that the content was “a little too basic” and 8% reporting that it was “far too basic” (table 1 and supplemental table at doi.org/10.5061/dryad.0cfxpnw1b). A total of 29% of viewers responded that NeuroBytes would change how they managed “many” of their patients, compared to 35% of viewers who responded that NeuroBytes would change how they managed “a few” patients and 31% who responded that NeuroBytes was helpful but “won't change” their practice (figure). No significant difference in satisfaction scores was observed when comparing results among NeuroBytes modules.
User responses to the question: “Was the information in this video helpful for your practice?” Total 1,894 responses.
Discussion
E-learning is considered an integral method of continuing education for many health professionals. In this study, we report 3 important findings: (1) while NeuroBytes' target audience consisted mostly of neurologists, the program also appealed to junior neurologists in training and students; (2) despite consistently high enrollment rates, course completion was less than half of the total users; and (3) although users were satisfied with the depth of content and likely to recommend NeuroBytes to a colleague, they reported variable impact on daily clinical practice.
While national and regional conferences, local workshops, and Grand Rounds continue to provide CPD opportunities, online learning is changing how professionals train, learn, and practice. NeuroLearn was initially developed by the AAN e-Learning Subcommittee to respond to the needs of neurologists by providing asynchronous, CME-certified online learning to optimize clinical practice. From 2012 to 2018, 17 courses were developed and achieved initial program goals, although user engagement remained low and the product appealed primarily to practicing neurologists. NeuroBytes was developed in 2019 to further respond to changing needs of neurologists by offering short modules that can be delivered in real-time and appeal to a more diverse audience, including advance practice providers and an expanded cohort of health care professionals.
Our study found that in the first 4 months, NeuroBytes enrolled 5,130 users; this can be compared to its counterpart, NeuroLearn, which enrolled 8,911 users in over 4 years. This may be due in part to the increased accessibility of NeuroBytes through the AAN's new Learning Management System as well as the shorter duration, focused topics, and frequent release of the modules every 2 weeks. In addition, as one user can have multiple enrollments; this may have contributed to the higher number of enrollments seen with NeuroBytes. We found that module enrollment ranged from 194 views with “Increasing Patient Engagement Series” to 922 views with “Clinical Approach to the Localization & Diagnosis of Ptosis.” We saw increased views with more clinically relevant modules, including topics such as Alzheimer disease, ischemic stroke, and acute vestibular syndrome. This highlights the continued need for clinically based, high-yield reviews that can provide information to users with varying levels of training.
Course completion rates for NeuroBytes (37%) were lower than those historically reported for NeuroLearn modules (71%). The lower completion rates seen with NeuroBytes may be due to the fact that CME credit was not offered or linked to completion. Previous studies show that CME designation and course relevance directly correlate with increased user engagement, implying that the drop in NeuroBytes course completion may be partly explained by an absence of CME credits.12,-,14 These findings underscore the benefits of aligning e-Learning directly with clinical practice (e.g., intrinsic motivation) or offering incentives such as monetary rewards, accreditation, or networking (e.g., extrinsic motivation) to encourage course completion.15
Moreover, NeuroBytes course completion required the user to spend additional time filling out the postcourse survey. Since one of the main appeals of NeuroBytes is its succinct nature, it is unsurprising that many users did not devote additional time to reach “completion” status. Current research supports the idea that brief (2–5 minutes), interactive videos have higher rates of engagement due to the preferred rapid pace and increasing time constraints of the modern learner.16,-,19 For the small portion of users who launched but did not watch the module, we suspect this is due to time restrictions in addition to minimal investment in the video. This theory is supported by the higher completion rates seen with user populations that are directly involved in patient care, including advance practice providers and neurologists. Future research should continue to explore why learner populations, including residents and students, had lower completion rates in comparison to their senior counterparts.
NeuroBytes appealed to a wider audience with a higher proportion of student, intern, and junior trainee viewers. Compared to NeuroLearn, the NeuroBytes audience was less likely to include neurologists (54% vs 73%) and more likely to include house staff (26% vs 17%) and students (8% vs <1%). Trainees often supplement existing teaching with materials that reinforce their learning, such as the consolidated instructional design of NeuroBytes modules.20,21 Conversely, NeuroLearn covers an entire topic area in-depth and is intended to be completed over time to address specific practice gaps. This style of learning may be more appealing to practicing clinicians who also desire CME credit. While the difference in NeuroBytes and NeuroLearn audiences may be explained by a shift toward blended learning styles among younger learners,22,-,24 it may also reflect differing needs for students and residents compared to practicing neurologists. This variability is seen through the 29% of users who found the depth of NeuroBytes content to be “a little too basic,” suggesting that providers with more training may be looking for more advanced material. Ultimately, our findings suggest that NeuroBytes appeals to a wider audience than its counterpart, NeuroLearn, and may complement current CPD offerings to provide a comprehensive menu of continuing education options for neurologists.
More concise, focused e-Learning programs such as NeuroBytes have been shown to be effective in supplementing outside clinical training and experiences.25 NeuroLearn, on the other hand, offers lengthier, CME-certified modules that appear to have a more tangible effect on clinical practice, which is supported by previous studies that emphasize the role of CME programs in improving patient care.26-27 Despite NeuroLearn's stronger translation to clinical practice, both online programs offer a range of educational benefits to their users and are viable options for CPD in neurology. Strengths of this study included a high number of enrollments in the NeuroBytes program, which provided a generalizable sample of continuing education needs in neurology. The high number of users allowed for a detailed enrollment breakdown and offered the opportunity to apply findings to e-Learning populations moving forward. However, when considering limitations of the study, resources required for program development varied between modules, and the estimated time and cost are averages. In addition, faculty time was not compensated for and was therefore not included in the total summation of costs, meaning the calculated cost for module development may be an underestimate. Future research should continue to explore and define resource requirements for e-Learning program development. Finally, this study was limited in its ability to fully interpret NeuroBytes completion rates, as the majority of users remained in progress, meaning they watched a portion of the video or watched the entire video without submitting the postcourse survey. Usage statistics revealed that these in progress users completed an average of 40% of the video before navigating away. Future efforts should aim to better define user groups and further explore how many users watch the entire module without completing the evaluation. In addition, it would be beneficial to capture repeated views independently and calculate how many users return and rewatch modules.
This study shows that using high-yield video modules, such as NeuroBytes, is a helpful tool in promoting CPD within the ever-evolving field of neurology. In moving forward, our findings and lessons learned can be applied to future development of e-Learning programs, such as the medical student NeuroBytes series that is being developed. This work suggests that while bite-sized modules can have high levels of usability with positive user survey responses, this does not necessarily translate into high completion rates. In addition, while users may be likely to recommend e-Learning tools to their peers and colleagues, it may take lengthier, more detailed educational tools to result in significant changes in clinical practice. Our findings suggest that NeuroBytes is a platform that could be easily expanded and replicated in programs that show a need for more reliable CPD programs or desire an enhanced global understanding of up-to-date practices.
Study Funding
This study was not industry-sponsored.
Disclosure
V. Santini has received research/grant support from Insightec, Inc., Biogen, Inc., and Roche/Genentech, Inc. R.E. Strowd serves a consultant for Monteris Medical Inc, Novocure, and Nanobiotix; he receives an editorial stipend as Deputy Section Editor of the Resident & Fellow Section of Neurology® and has received research/grant support from the American Academy of Neurology, American Society for Clinical Oncology, Southeastern Brain Tumor Foundation, and Jazz Pharmaceuticals. All authors except L. Lavette serve or have served on the e-Learning Subcommittee of the American Academy of Neurology. Otherwise, these authors report no disclosures relevant to the manuscript. Go to Neurology.org/N for full disclosures.
Appendix Authors
Footnotes
Go to Neurology.org/N for full disclosures. Funding information and disclosures deemed relevant by the authors, if any, are provided at the end of the article.
- © 2021 American Academy of Neurology
References
Disputes & Debates: Rapid online correspondence
REQUIREMENTS
If you are uploading a letter concerning an article:
You must have updated your disclosures within six months: http://submit.neurology.org
Your co-authors must send a completed Publishing Agreement Form to Neurology Staff (not necessary for the lead/corresponding author as the form below will suffice) before you upload your comment.
If you are responding to a comment that was written about an article you originally authored:
You (and co-authors) do not need to fill out forms or check disclosures as author forms are still valid
and apply to letter.
Submission specifications:
- Submissions must be < 200 words with < 5 references. Reference 1 must be the article on which you are commenting.
- Submissions should not have more than 5 authors. (Exception: original author replies can include all original authors of the article)
- Submit only on articles published within 6 months of issue date.
- Do not be redundant. Read any comments already posted on the article prior to submission.
- Submitted comments are subject to editing and editor review prior to posting.