Synopsis

Welcome to the second talk from Prof Ioannidis at SSAI2019.

Peer review has been a fundamental principle of scientific advancement for centuries. If a researcher wants to publish an article detailing scientific findings, the manuscript will have to pass the scrutiny of experts in the field before it will be published in a reputable journal. That way, some of the critical appraisal of evidence is done for the reader, sorting wheat from the chaff and guaranteeing a certain level of relevance and trustworthiness. However compelling this idea seems on paper, peer review as manifested in the real world of today’s publishing industry and academic incentive structure may not hold up well to scrutiny of its own.

I can think of no-one better to deliver an expert assessment of the developing state of peer review in scientific publishing than Prof John Ioannidis.

Video

Take me further

Please see our first talk by Prof Ioannidis from SSAI2019 for more details on the general difficulty of doing science properly.

Also, as always, please see some of the primary literature in full for a more in-depth look at the evidence. I’ve collated the articles referenced by Ioannidis in the talk below.

Ioannidis addresses a wide ranging issues and solutions, many of which have also been hotly debated in the FOAM community. The current state of publishing and peer review and the future of knowledge dissemination was notably discussed at smaccDUB at what was for me perhaps the most interesting session in a fiercely competitive field. Simon Finfer moderated a panel discussion on these topics with high profile names like Kate Maitland, Flavia Machado, Rob MacSweeney, Richard Smith, Simon Carly, Kathy Rowan, John Myburgh, Ryan Radecki and Jeff Drazen.

You can watch the smaccDUB panel below.

I think Prof Ioannidis adds a level of quantitative analysis to this discussion that is very worthwhile.

Two of the SMACCDUB panelists also gave isolated talks with a pro et con angle on pre-publication peer review

The rise of medical evidence by Jeff Drazen, editor-in-chief of NEJM, a logical proponent of expert peer review
The perils of peer review by Richard Smith, previous editor-in-chief of BMJ, now a strong proponent of abandoning pre-publication peer review

More sobering viewpoints on traditional peer review from Richard Smith can be found here

Peer review: a flawed process at the heart of science and journals (open access here)
The optimal peer review system (BMJ blog)

Articles referenced in the talk

Ioannidis JP a. Why most published research findings are false. PLoS Med. 2005 Aug 30;2(8):e124. Available from: http://www.ncbi.nlm.nih.gov/pubmed/27328301
Ioannidis JPA. How to Make More Published Research True. PLoS Med. 2014;11(10). Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204808/
Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, Percie du Sert N, et al. A manifesto for reproducible science. Nat Hum Behav. 2017 Jan 10;1(1):0021. Available from: http://dx.doi.org/10.1038/s41562-016-0021
Hopewell S, Collins GS, Boutron I, Yu L-M, Cook J, Shanyinde M, et al. Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study. BMJ. 2014 Jul 1;349(July):g4145. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24986891
Goldacre B, Drysdale H, Dale A, Milosevic I, Slade E, Hartley P, et al. COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time. Trials. 2019 Feb 14;20(1):118. Available from: http://www.ncbi.nlm.nih.gov/pubmed/30760329
Abdill RJ, Blekhman R. Tracking the popularity and outcomes of all bioRxiv preprints. Elife. 2019;8:515643. Available from: http://www.ncbi.nlm.nih.gov/pubmed/31017570
Avidan MS, Ioannidis JPA, Mashour GA. Independent discussion sections for improving inferential reproducibility in published research. Br J Anaesth. 2019;122(4):413–20. Available from: https://doi.org/10.1016/j.bja.2018.12.010
Vlisides PE, Ioannidis JPA, Avidan MS. Hypnotic depth and postoperative death: a Bayesian perspective and an Independent Discussion of a clinical trial. Br J Anaesth. 2019;122(4):421–7. Available from: https://doi.org/10.1016/j.bja.2019.01.012
Maraganore DM, Lesnick TG, Elbaz A, Chartier-Harlin M-C, Gasser T, Krüger R, et al. UCHL1 Is a Parkinson’s Disease Susceptibility Gene. Ann Neurol. 2004 Apr;55(4):512–21. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15048890
Maraganore DM, Lesnick TG, Elbaz A, Chartier-Harlin M-C, Gasser T, Krüger R, et al. UCHL-1 is not a Parkinson’s disease susceptibility gene. Ann Neurol. 2004 Apr;55(4):512–21. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15048890
Doshi P, Goodman SN, Ioannidis JPA. Raw data from clinical trials: within reach? Trends Pharmacol Sci. 2013 Dec;34(12):645–7. Available from: http://dx.doi.org/10.1016/j.tips.2013.10.006
Ioannidis JPA, Allison DB, Ball CA, Coulibaly I, Cui X, Culhane AC, et al. Repeatability of published microarray gene expression analyses. Nat Genet. 2009;41(2):149–55.
Open Science Collaboration. PSYCHOLOGY. Estimating the reproducibility of psychological science. Science. 2015 Aug 28;349(6251):aac4716. Available from: http://www.ncbi.nlm.nih.gov/pubmed/26315443
Ioannidis JPA. The Reproducibility Wars: Successful, Unsuccessful, Uninterpretable, Exact, Conceptual, Triangulated, Contested Replication. Clin Chem. 2017;63(5):943–5. Available from: http://www.ncbi.nlm.nih.gov/pubmed/28298413
Ioannidis JPA. The importance of potential studies that have not existed and registration of observational data sets. JAMA. 2012 Aug 8;308(6):575–6. Available from: http://www.ncbi.nlm.nih.gov/pubmed/22871867
Vanden Heuvel JP, Bullenkamp J, Reproducibility Project: Cancer Biology. Registered report: Systematic identification of genomic markers of drug sensitivity in cancer cells. Elife. 2016;5(JUNE2016):1–19. Available from: http://www.ncbi.nlm.nih.gov/pubmed/27336789
Hardwicke TE, Ioannidis JPA. Mapping the universe of registered reports. Nat Hum Behav. 2018;2(11):793–6. Available from: http://www.ncbi.nlm.nih.gov/pubmed/31558810
Ioannidis JPA. How to make more published research true. PLoS Med. 2014 Oct;11(10):e1001747. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204808/

Before you go, don’t forget to

share widely, options below
leave a comment and join the #FOAMed conversation in general
connect with the scanFOAM project and team on Twitter, Facebook, Instagram or write us an old fashioned email
subscribe to thenew stuff.

Mads Astvad

Scandinavian paediatric anaesthetist / intensivist.
PHARM, ED, OR, ICU.
Digital MedEd
Co-founder scanFOAM.org
Co-organiser CphCC & TBS-Zermatt (aka The Big Sick)
Medical lead REPEL (resilience in pediatric emergency life support)
Web dev SSAI.info