Welcome to the second talk from Prof Ioannidis at SSAI2019.
Peer review has been a fundamental principle of scientific advancement for centuries. If a researcher wants to publish an article detailing scientific findings, the manuscript will have to pass the scrutiny of experts in the field before it will be published in a reputable journal. That way, some of the critical appraisal of evidence is done for the reader, sorting wheat from the chaff and guaranteeing a certain level of relevance and trustworthiness. However compelling this idea seems on paper, peer review as manifested in the real world of today’s publishing industry and academic incentive structure may not hold up well to scrutiny of its own.
I can think of no-one better to deliver an expert assessment of the developing state of peer review in scientific publishing than Prof John Ioannidis.
Take me further
Please see our first talk by Prof Ioannidis from SSAI2019 for more details on the general difficulty of doing science properly.
Also, as always, please see some of the primary literature in full for a more in-depth look at the evidence. I’ve collated the articles referenced by Ioannidis in the talk below.
Ioannidis addresses a wide ranging issues and solutions, many of which have also been hotly debated in the FOAM community. The current state of publishing and peer review and the future of knowledge dissemination was notably discussed at smaccDUB at what was for me perhaps the most interesting session in a fiercely competitive field. Simon Finfer moderated a panel discussion on these topics with high profile names like Kate Maitland, Flavia Machado, Rob MacSweeney, Richard Smith, Simon Carly, Kathy Rowan, John Myburgh, Ryan Radecki and Jeff Drazen.
You can watch the smaccDUB panel below.
I think Prof Ioannidis adds a level of quantitative analysis to this discussion that is very worthwhile.
Two of the SMACCDUB panelists also gave isolated talks with a pro et con angle on pre-publication peer review
- The rise of medical evidence by Jeff Drazen, editor-in-chief of NEJM, a logical proponent of expert peer review
- The perils of peer review by Richard Smith, previous editor-in-chief of BMJ, now a strong proponent of abandoning pre-publication peer review
More sobering viewpoints on traditional peer review from Richard Smith can be found here
- Peer review: a flawed process at the heart of science and journals (open access here)
- The optimal peer review system (BMJ blog)
Articles referenced in the talk
- Ioannidis JP a. Why most published research findings are false. PLoS Med. 2005 Aug 30;2(8):e124. Available from: http://www.ncbi.nlm.nih.gov/pubmed/27328301
- Ioannidis JPA. How to Make More Published Research True. PLoS Med. 2014;11(10). Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204808/
- Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, Percie du Sert N, et al. A manifesto for reproducible science. Nat Hum Behav. 2017 Jan 10;1(1):0021. Available from: http://dx.doi.org/10.1038/s41562-016-0021
- Hopewell S, Collins GS, Boutron I, Yu L-M, Cook J, Shanyinde M, et al. Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study. BMJ. 2014 Jul 1;349(July):g4145. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24986891
- Goldacre B, Drysdale H, Dale A, Milosevic I, Slade E, Hartley P, et al. COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time. Trials. 2019 Feb 14;20(1):118. Available from: http://www.ncbi.nlm.nih.gov/pubmed/30760329
- Abdill RJ, Blekhman R. Tracking the popularity and outcomes of all bioRxiv preprints. Elife. 2019;8:515643. Available from: http://www.ncbi.nlm.nih.gov/pubmed/31017570
- Avidan MS, Ioannidis JPA, Mashour GA. Independent discussion sections for improving inferential reproducibility in published research. Br J Anaesth. 2019;122(4):413–20. Available from: https://doi.org/10.1016/j.bja.2018.12.010
- Vlisides PE, Ioannidis JPA, Avidan MS. Hypnotic depth and postoperative death: a Bayesian perspective and an Independent Discussion of a clinical trial. Br J Anaesth. 2019;122(4):421–7. Available from: https://doi.org/10.1016/j.bja.2019.01.012
- Maraganore DM, Lesnick TG, Elbaz A, Chartier-Harlin M-C, Gasser T, Krüger R, et al. UCHL1 Is a Parkinson’s Disease Susceptibility Gene. Ann Neurol. 2004 Apr;55(4):512–21. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15048890
- Maraganore DM, Lesnick TG, Elbaz A, Chartier-Harlin M-C, Gasser T, Krüger R, et al. UCHL-1 is not a Parkinson’s disease susceptibility gene. Ann Neurol. 2004 Apr;55(4):512–21. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15048890
- Doshi P, Goodman SN, Ioannidis JPA. Raw data from clinical trials: within reach? Trends Pharmacol Sci. 2013 Dec;34(12):645–7. Available from: http://dx.doi.org/10.1016/j.tips.2013.10.006
- Ioannidis JPA, Allison DB, Ball CA, Coulibaly I, Cui X, Culhane AC, et al. Repeatability of published microarray gene expression analyses. Nat Genet. 2009;41(2):149–55.
- Open Science Collaboration. PSYCHOLOGY. Estimating the reproducibility of psychological science. Science. 2015 Aug 28;349(6251):aac4716. Available from: http://www.ncbi.nlm.nih.gov/pubmed/26315443
- Ioannidis JPA. The Reproducibility Wars: Successful, Unsuccessful, Uninterpretable, Exact, Conceptual, Triangulated, Contested Replication. Clin Chem. 2017;63(5):943–5. Available from: http://www.ncbi.nlm.nih.gov/pubmed/28298413
- Ioannidis JPA. The importance of potential studies that have not existed and registration of observational data sets. JAMA. 2012 Aug 8;308(6):575–6. Available from: http://www.ncbi.nlm.nih.gov/pubmed/22871867
- Vanden Heuvel JP, Bullenkamp J, Reproducibility Project: Cancer Biology. Registered report: Systematic identification of genomic markers of drug sensitivity in cancer cells. Elife. 2016;5(JUNE2016):1–19. Available from: http://www.ncbi.nlm.nih.gov/pubmed/27336789
- Hardwicke TE, Ioannidis JPA. Mapping the universe of registered reports. Nat Hum Behav. 2018;2(11):793–6. Available from: http://www.ncbi.nlm.nih.gov/pubmed/31558810
- Ioannidis JPA. How to make more published research true. PLoS Med. 2014 Oct;11(10):e1001747. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204808/