Statistics and the detection of scientific misconduct

The paper by Bolland et al.¹ in this issue had a longer “gestation” than usual at Neurology®; it was received on December 4, 2015. We usually review on a tight timeline and publish important work quickly. But this manuscript has a different focus as it presents a statistical analysis that demonstrates probable scientific misconduct (fraud) on a large scale. It is also unique in that its analysis uses complicated methods that may be beyond what most readers are willing to tackle. However, the bottom line is simple: there are statistical methods that, when properly applied, can detect fraudulent behavior by investigators. Clinical trial data can thereby be validated or called into question. In this instance, analysis of a collection of studies by a large research group suggested that at least some of the studies were likely to be fraudulent.

Why did our review process take nearly a year? The findings in this report had implications for the authors and their institutions, for other journals and editors who published work by these authors, and for clinicians, guideline committees, and policymakers who rely on the validity of these findings. In the interests of all involved, we sought input from other editors, coauthors, and institutions. We asked 3 statisticians who regularly review for Neurology® to provide feedback on the manuscript. They validated the work and suggested important changes (as did the editorial team) that resulted in revisions of the article and the final version that is now published.

The present report analyzes 33 studies, 3 of which were published by Neurology. In the midst of our investigation and during the review process of the Bolland et al. manuscript, we heard from the main author of the examined studies (Sato), who admitted that the work reported in Neurology was fraudulent, relieved those he listed as coauthors of any wrongdoing, and requested retraction of the studies.

Fraud in an individual paper may be difficult to detect. Peer review, being a human process, can never be perfect; furthermore, the detection of likely fraudulent activity in the Bolland et al. paper required analysis of many trials. As such, it is unlikely that detection of fraud could have occurred during the review of single papers. Furthermore, simply because this group analysis suggested the likelihood of fraudulent activity, one cannot conclude that any one study—among those in the analysis—is, or is not, fraudulent. The authors were careful to avoid any deductions about individual studies, especially as there were multiple authors within the research group.

As part of our due process, we have notified other editors whose journals published papers by Sato et al., communicated with Sato's institution, and published retractions of the 3 papers and a letter published in Neurology.^2–4 Each step involved our scientific integrity advisor, former editor-in-chief Robert Daroff. The final outcome is the publication of the Bolland et al. article. Our retractions help to correct the literature and any meta-analyses or guidelines that relied on those data, while publication of this paper shows the method and approach that led to the retractions. We also alert readers and authors that we will continue to use rigorous statistical review to detect fraud with the goal of maintaining the highest possible standards in publishing.

• © 2016 American Academy of Neurology