Skip to main content
Advertisement
  • Neurology.org
  • Journals
    • Neurology
    • Clinical Practice
    • Genetics
    • Neuroimmunology & Neuroinflammation
  • Specialty Sites
    • COVID-19
    • Practice Current
    • Practice Buzz
    • Without Borders
    • Equity, Diversity and Inclusion
    • Innovations in Care Delivery
  • Collections
    • Topics A-Z
    • Residents & Fellows
    • Infographics
    • Patient Pages
    • Null Hypothesis
    • Translations
  • Podcast
  • CME
  • About
    • About the Journals
    • Contact Us
    • Editorial Board
  • Authors
    • Submit a Manuscript
    • Author Center

Advanced Search

Main menu

  • Neurology.org
  • Journals
    • Neurology
    • Clinical Practice
    • Genetics
    • Neuroimmunology & Neuroinflammation
  • Specialty Sites
    • COVID-19
    • Practice Current
    • Practice Buzz
    • Without Borders
    • Equity, Diversity and Inclusion
    • Innovations in Care Delivery
  • Collections
    • Topics A-Z
    • Residents & Fellows
    • Infographics
    • Patient Pages
    • Null Hypothesis
    • Translations
  • Podcast
  • CME
  • About
    • About the Journals
    • Contact Us
    • Editorial Board
  • Authors
    • Submit a Manuscript
    • Author Center
  • Home
  • Latest Articles
  • Current Issue
  • Past Issues
  • Residents & Fellows

User menu

  • Subscribe
  • My Alerts
  • Log in

Search

  • Advanced search
Neurology
Home
The most widely read and highly cited peer-reviewed neurology journal
  • Subscribe
  • My Alerts
  • Log in
Site Logo
  • Home
  • Latest Articles
  • Current Issue
  • Past Issues
  • Residents & Fellows

Share

August 04, 2020; 95 (5) Article

Wisdom of the expert crowd prediction of response for 3 neurology randomized trials

Pavel Atanasov, Andreas Diamantaras, Amanda MacPherson, Esther Vinarov, Daniel M. Benjamin, Ian Shrier, Friedemann Paul, Ulrich Dirnagl, View ORCID ProfileJonathan Kimmelman
First published June 16, 2020, DOI: https://doi.org/10.1212/WNL.0000000000009819
Pavel Atanasov
From Pytho LLC (P.A.), Brooklyn, NY; Department of Neurology (A.D.), Inselspital, Bern University Hospital, University of Bern, Switzerland; Biomedical Ethics Unit, Department of Social Studies of Medicine (A.M., E.V., D.M.B., J.K.), and Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital (I.S.), McGill University, Montreal, Canada; Max Delbrueck Center for Molecular Medicine (F.P.), Berlin; Department of Neurology (F.P.), NeuroCure Clinical Research Center and Experimental and Clinical Research Center, Charité-Universitätsmedizin Berlin; Humboldt-Universität zu Berlin (U.D.), Charité-Universitätsmedizin Berlin, corporate member of Freie Universität Berlin; and Department of Experimental Neurology and Center for Stroke Research Berlin and QUEST Center for Transforming Biomedical Research (U.D.), Berlin Institute of Health, Germany.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Andreas Diamantaras
From Pytho LLC (P.A.), Brooklyn, NY; Department of Neurology (A.D.), Inselspital, Bern University Hospital, University of Bern, Switzerland; Biomedical Ethics Unit, Department of Social Studies of Medicine (A.M., E.V., D.M.B., J.K.), and Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital (I.S.), McGill University, Montreal, Canada; Max Delbrueck Center for Molecular Medicine (F.P.), Berlin; Department of Neurology (F.P.), NeuroCure Clinical Research Center and Experimental and Clinical Research Center, Charité-Universitätsmedizin Berlin; Humboldt-Universität zu Berlin (U.D.), Charité-Universitätsmedizin Berlin, corporate member of Freie Universität Berlin; and Department of Experimental Neurology and Center for Stroke Research Berlin and QUEST Center for Transforming Biomedical Research (U.D.), Berlin Institute of Health, Germany.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Amanda MacPherson
From Pytho LLC (P.A.), Brooklyn, NY; Department of Neurology (A.D.), Inselspital, Bern University Hospital, University of Bern, Switzerland; Biomedical Ethics Unit, Department of Social Studies of Medicine (A.M., E.V., D.M.B., J.K.), and Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital (I.S.), McGill University, Montreal, Canada; Max Delbrueck Center for Molecular Medicine (F.P.), Berlin; Department of Neurology (F.P.), NeuroCure Clinical Research Center and Experimental and Clinical Research Center, Charité-Universitätsmedizin Berlin; Humboldt-Universität zu Berlin (U.D.), Charité-Universitätsmedizin Berlin, corporate member of Freie Universität Berlin; and Department of Experimental Neurology and Center for Stroke Research Berlin and QUEST Center for Transforming Biomedical Research (U.D.), Berlin Institute of Health, Germany.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Esther Vinarov
From Pytho LLC (P.A.), Brooklyn, NY; Department of Neurology (A.D.), Inselspital, Bern University Hospital, University of Bern, Switzerland; Biomedical Ethics Unit, Department of Social Studies of Medicine (A.M., E.V., D.M.B., J.K.), and Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital (I.S.), McGill University, Montreal, Canada; Max Delbrueck Center for Molecular Medicine (F.P.), Berlin; Department of Neurology (F.P.), NeuroCure Clinical Research Center and Experimental and Clinical Research Center, Charité-Universitätsmedizin Berlin; Humboldt-Universität zu Berlin (U.D.), Charité-Universitätsmedizin Berlin, corporate member of Freie Universität Berlin; and Department of Experimental Neurology and Center for Stroke Research Berlin and QUEST Center for Transforming Biomedical Research (U.D.), Berlin Institute of Health, Germany.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Daniel M. Benjamin
From Pytho LLC (P.A.), Brooklyn, NY; Department of Neurology (A.D.), Inselspital, Bern University Hospital, University of Bern, Switzerland; Biomedical Ethics Unit, Department of Social Studies of Medicine (A.M., E.V., D.M.B., J.K.), and Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital (I.S.), McGill University, Montreal, Canada; Max Delbrueck Center for Molecular Medicine (F.P.), Berlin; Department of Neurology (F.P.), NeuroCure Clinical Research Center and Experimental and Clinical Research Center, Charité-Universitätsmedizin Berlin; Humboldt-Universität zu Berlin (U.D.), Charité-Universitätsmedizin Berlin, corporate member of Freie Universität Berlin; and Department of Experimental Neurology and Center for Stroke Research Berlin and QUEST Center for Transforming Biomedical Research (U.D.), Berlin Institute of Health, Germany.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Ian Shrier
From Pytho LLC (P.A.), Brooklyn, NY; Department of Neurology (A.D.), Inselspital, Bern University Hospital, University of Bern, Switzerland; Biomedical Ethics Unit, Department of Social Studies of Medicine (A.M., E.V., D.M.B., J.K.), and Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital (I.S.), McGill University, Montreal, Canada; Max Delbrueck Center for Molecular Medicine (F.P.), Berlin; Department of Neurology (F.P.), NeuroCure Clinical Research Center and Experimental and Clinical Research Center, Charité-Universitätsmedizin Berlin; Humboldt-Universität zu Berlin (U.D.), Charité-Universitätsmedizin Berlin, corporate member of Freie Universität Berlin; and Department of Experimental Neurology and Center for Stroke Research Berlin and QUEST Center for Transforming Biomedical Research (U.D.), Berlin Institute of Health, Germany.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Friedemann Paul
From Pytho LLC (P.A.), Brooklyn, NY; Department of Neurology (A.D.), Inselspital, Bern University Hospital, University of Bern, Switzerland; Biomedical Ethics Unit, Department of Social Studies of Medicine (A.M., E.V., D.M.B., J.K.), and Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital (I.S.), McGill University, Montreal, Canada; Max Delbrueck Center for Molecular Medicine (F.P.), Berlin; Department of Neurology (F.P.), NeuroCure Clinical Research Center and Experimental and Clinical Research Center, Charité-Universitätsmedizin Berlin; Humboldt-Universität zu Berlin (U.D.), Charité-Universitätsmedizin Berlin, corporate member of Freie Universität Berlin; and Department of Experimental Neurology and Center for Stroke Research Berlin and QUEST Center for Transforming Biomedical Research (U.D.), Berlin Institute of Health, Germany.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Ulrich Dirnagl
From Pytho LLC (P.A.), Brooklyn, NY; Department of Neurology (A.D.), Inselspital, Bern University Hospital, University of Bern, Switzerland; Biomedical Ethics Unit, Department of Social Studies of Medicine (A.M., E.V., D.M.B., J.K.), and Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital (I.S.), McGill University, Montreal, Canada; Max Delbrueck Center for Molecular Medicine (F.P.), Berlin; Department of Neurology (F.P.), NeuroCure Clinical Research Center and Experimental and Clinical Research Center, Charité-Universitätsmedizin Berlin; Humboldt-Universität zu Berlin (U.D.), Charité-Universitätsmedizin Berlin, corporate member of Freie Universität Berlin; and Department of Experimental Neurology and Center for Stroke Research Berlin and QUEST Center for Transforming Biomedical Research (U.D.), Berlin Institute of Health, Germany.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jonathan Kimmelman
From Pytho LLC (P.A.), Brooklyn, NY; Department of Neurology (A.D.), Inselspital, Bern University Hospital, University of Bern, Switzerland; Biomedical Ethics Unit, Department of Social Studies of Medicine (A.M., E.V., D.M.B., J.K.), and Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital (I.S.), McGill University, Montreal, Canada; Max Delbrueck Center for Molecular Medicine (F.P.), Berlin; Department of Neurology (F.P.), NeuroCure Clinical Research Center and Experimental and Clinical Research Center, Charité-Universitätsmedizin Berlin; Humboldt-Universität zu Berlin (U.D.), Charité-Universitätsmedizin Berlin, corporate member of Freie Universität Berlin; and Department of Experimental Neurology and Center for Stroke Research Berlin and QUEST Center for Transforming Biomedical Research (U.D.), Berlin Institute of Health, Germany.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Jonathan Kimmelman
Full PDF
Short Form
Citation
Wisdom of the expert crowd prediction of response for 3 neurology randomized trials
Pavel Atanasov, Andreas Diamantaras, Amanda MacPherson, Esther Vinarov, Daniel M. Benjamin, Ian Shrier, Friedemann Paul, Ulrich Dirnagl, Jonathan Kimmelman
Neurology Aug 2020, 95 (5) e488-e498; DOI: 10.1212/WNL.0000000000009819

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Permissions

Make Comment

See Comments

Downloads
106

Share

  • Article
  • Figures & Data
  • Info & Disclosures
Loading

This article requires a subscription to view the full text. If you have a subscription you may use the login form below to view the article. Access to this article can also be purchased.

Abstract

Objective To explore the accuracy of combined neurology expert forecasts in predicting primary endpoints for trials.

Methods We identified one major randomized trial each in stroke, multiple sclerosis (MS), and amyotrophic lateral sclerosis (ALS) that was closing within 6 months. After recruiting a sample of neurology experts for each disease, we elicited forecasts for the primary endpoint outcomes in the trial placebo and treatment arms. Our main outcome was the accuracy of averaged predictions, measured using ordered Brier scores. Scores were compared against an algorithm that offered noncommittal predictions.

Results Seventy-one neurology experts participated. Combined forecasts of experts were less accurate than a noncommittal prediction algorithm for the stroke trial (pooled Brier score = 0.340, 95% subjective probability interval [sPI] 0.340 to 0.340 vs 0.185 for the uninformed prediction), and approximately as accurate for the MS study (pooled Brier score = 0.107, 95% confidence interval [CI] 0.081 to 0.133 vs 0.098 for the noncommittal prediction) and the ALS study (pooled Brier score = 0.090, 95% CI 0.081 to 0.185 vs 0.090). The 95% sPIs of individual predictions contained actual trial outcomes among 44% of experts. Only 18% showed prediction skill exceeding the noncommittal prediction. Independent experts and coinvestigators achieved similar levels of accuracy.

Conclusion In this first-of-kind exploratory study, averaged expert judgments rarely outperformed noncommittal forecasts. However, experts at least anticipated the possibility of effects observed in trials. Our findings, if replicated in different trial samples, caution against the reliance on simple approaches for combining expert opinion in making research and policy decisions.

Glossary

ACTION=
Effect of Natalizumab on Infarct Volume in Acute Ischemic Stroke;
ALS=
amyotrophic lateral sclerosis;
ALS-Rag=
Efficacy, Safety and Tolerability Study of 1 mg Rasagiline in Patients With Amyotrophic Lateral Sclerosis (ALS) Receiving Standard Therapy (Riluzole): An AMG Trial With a Market Authorized Substance;
CI=
confidence interval;
MS=
multiple sclerosis;
SOLAR=
Supplementation of VigantOL® Oil Versus Placebo as Add-on in Patients With Relapsing Remitting Multiple Sclerosis Receiving Rebif® Treatment;
sPI=
subjective probability interval

Footnotes

  • Go to Neurology.org/N for full disclosures. Funding information and disclosures deemed relevant by the authors, if any, are provided at the end of the article.

  • ↵* These authors contributed equally to this work.

  • Podcast: NPub.org/3xoxf9

  • Received August 7, 2019.
  • Accepted in final form January 7, 2020.
  • © 2020 American Academy of Neurology
View Full Text

AAN Members

We have changed the login procedure to improve access between AAN.com and the Neurology journals. If you are experiencing issues, please log out of AAN.com and clear history and cookies. (For instructions by browser, please click the instruction pages below). After clearing, choose preferred Journal and select login for AAN Members. You will be redirected to a login page where you can log in with your AAN ID number and password. When you are returned to the Journal, your name should appear at the top right of the page.

Google Safari Microsoft Edge Firefox

Click here to login

AAN Non-Member Subscribers

Click here to login

Purchase access

For assistance, please contact:
AAN Members (800) 879-1960 or (612) 928-6000 (International)
Non-AAN Member subscribers (800) 638-3030 or (301) 223-2300 option 3, select 1 (international)

Sign Up
Information on how to subscribe to Neurology and Neurology: Clinical Practice can be found here 

Purchase
Individual access to articles is available through the Add to Cart option on the article page.  Access for 1 day (from the computer you are currently using) is US$ 39.00.  Pay-per-view content is for the use of the payee only, and content may not be further distributed by print or electronic means.  The payee may view, download, and/or print the article for his/her personal, scholarly, research, and educational use.  Distributing copies (electronic or otherwise) of the article is not allowed.

Disputes & Debates: Rapid online correspondence

No comments have been published for this article.
Comment

NOTE: All authors' disclosures must be entered and current in our database before comments can be posted. Enter and update disclosures at http://submit.neurology.org. Exception: replies to comments concerning an article you originally authored do not require updated disclosures.

  • Stay timely. Submit only on articles published within 6 months of issue date.
  • Do not be redundant. Read any comments already posted on the article prior to submission.
  • 200 words maximum.
  • 5 references maximum. Reference 1 must be the article on which you are commenting.
  • 5 authors maximum. Exception: replies can include all original authors of the article.
  • Submitted comments are subject to editing and editor review prior to posting.

More guidelines and information on Disputes & Debates

Compose Comment

More information about text formats

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
Author Information
NOTE: The first author must also be the corresponding author of the comment.
First or given name, e.g. 'Peter'.
Your last, or family, name, e.g. 'MacMoody'.
Your email address, e.g. higgs-boson@gmail.com
Your role and/or occupation, e.g. 'Orthopedic Surgeon'.
Your organization or institution (if applicable), e.g. 'Royal Free Hospital'.
Publishing Agreement
NOTE: All authors, besides the first/corresponding author, must complete a separate Disputes & Debates Submission Form and provide via email to the editorial office before comments can be posted.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.

Vertical Tabs

You May Also be Interested in

Back to top
  • Article
    • Abstract
    • Glossary
    • Methods
    • Results
    • Discussion
    • Study funding
    • Disclosure
    • Acknowledgment
    • Appendix Authors
    • Footnotes
    • References
  • Figures & Data
  • Info & Disclosures
Advertisement

Related Articles

  • No related articles found.

Topics Discussed

  • All Cerebrovascular disease/Stroke
  • All Ethics in Neurology/Legal issues
  • Multiple sclerosis
  • Amyotrophic lateral sclerosis
  • Decision analysis

Alert Me

  • Alert me when eletters are published
Neurology: 96 (15)

Articles

  • Ahead of Print
  • Current Issue
  • Past Issues
  • Popular Articles
  • Translations

About

  • About the Journals
  • Ethics Policies
  • Editors & Editorial Board
  • Contact Us
  • Advertise

Submit

  • Author Center
  • Submit a Manuscript
  • Information for Reviewers
  • AAN Guidelines
  • Permissions

Subscribers

  • Subscribe
  • Activate a Subscription
  • Sign up for eAlerts
  • RSS Feed
Site Logo
  • Visit neurology Template on Facebook
  • Follow neurology Template on Twitter
  • Visit Neurology on YouTube
  • Neurology
  • Neurology: Clinical Practice
  • Neurology: Genetics
  • Neurology: Neuroimmunology & Neuroinflammation
  • AAN.com
  • AANnews
  • Continuum
  • Brain & Life
  • Neurology Today

Wolters Kluwer Logo

Neurology | Print ISSN:0028-3878
Online ISSN:1526-632X

© 2021 American Academy of Neurology

  • Privacy Policy
  • Feedback
  • Advertise