Peer review – pre or post? | John PA Ioannidis | SSAI2019

Synopsis

Welcome to the second talk from Prof Ioannidis at SSAI2019.

Peer review has been a fundamental principle of scientific advancement for centuries. If a researcher wants to publish an article detailing scientific findings, the manuscript will have to pass the scrutiny of experts in the field before it will be published in a reputable journal. That way, some of the critical appraisal of evidence is done for the reader, sorting wheat from the chaff and guaranteeing a certain level of relevance and trustworthiness. However compelling this idea seems on paper, peer review as manifested in the real world of today’s publishing industry and academic incentive structure may not hold up well to scrutiny of its own.

I can think of no-one better to deliver an expert assessment of the developing state of peer review in scientific publishing than Prof John Ioannidis.

Video

Take me further

Please see our first talk by Prof Ioannidis from SSAI2019 for more details on the general difficulty of doing science properly.

Also, as always, please see some of the primary literature in full for a more in-depth look at the evidence. I’ve collated the articles referenced by Ioannidis in the talk below.

Ioannidis addresses a wide ranging issues and solutions, many of which have also been hotly debated in the FOAM community. The current state of publishing and peer review and the future of knowledge dissemination was notably discussed at smaccDUB at what was for me perhaps the most interesting session in a fiercely competitive field. Simon Finfer moderated a panel discussion on these topics with high profile names like Kate Maitland, Flavia Machado, Rob MacSweeney, Richard Smith, Simon Carly, Kathy Rowan, John Myburgh, Ryan Radecki and Jeff Drazen.

You can watch the smaccDUB panel below.

I think Prof Ioannidis adds a level of quantitative analysis to this discussion that is very worthwhile.

Two of the SMACCDUB panelists also gave isolated talks with a pro et con angle on pre-publication peer review

More sobering viewpoints on traditional peer review from Richard Smith can be found here

  • Peer review: a flawed process at the heart of science and journals (open access here)
  • The optimal peer review system (BMJ blog)

Articles referenced in the talk

  1. Ioannidis JP a. Why most published research findings are false. PLoS Med. 2005 Aug 30;2(8):e124. Available from: http://www.ncbi.nlm.nih.gov/pubmed/27328301
  2. Ioannidis JPA. How to Make More Published Research True. PLoS Med. 2014;11(10). Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204808/
  3. Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, Percie du Sert N, et al. A manifesto for reproducible science. Nat Hum Behav. 2017 Jan 10;1(1):0021. Available from: http://dx.doi.org/10.1038/s41562-016-0021
  4. Hopewell S, Collins GS, Boutron I, Yu L-M, Cook J, Shanyinde M, et al. Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study. BMJ. 2014 Jul 1;349(July):g4145. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24986891
  5. Goldacre B, Drysdale H, Dale A, Milosevic I, Slade E, Hartley P, et al. COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time. Trials. 2019 Feb 14;20(1):118. Available from: http://www.ncbi.nlm.nih.gov/pubmed/30760329
  6. Abdill RJ, Blekhman R. Tracking the popularity and outcomes of all bioRxiv preprints. Elife. 2019;8:515643. Available from: http://www.ncbi.nlm.nih.gov/pubmed/31017570
  7. Avidan MS, Ioannidis JPA, Mashour GA. Independent discussion sections for improving inferential reproducibility in published research. Br J Anaesth. 2019;122(4):413–20. Available from: https://doi.org/10.1016/j.bja.2018.12.010
  8. Vlisides PE, Ioannidis JPA, Avidan MS. Hypnotic depth and postoperative death: a Bayesian perspective and an Independent Discussion of a clinical trial. Br J Anaesth. 2019;122(4):421–7. Available from: https://doi.org/10.1016/j.bja.2019.01.012
  9. Maraganore DM, Lesnick TG, Elbaz A, Chartier-Harlin M-C, Gasser T, Krüger R, et al. UCHL1 Is a Parkinson’s Disease Susceptibility Gene. Ann Neurol. 2004 Apr;55(4):512–21. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15048890
  10. Maraganore DM, Lesnick TG, Elbaz A, Chartier-Harlin M-C, Gasser T, Krüger R, et al. UCHL-1 is not a Parkinson’s disease susceptibility gene. Ann Neurol. 2004 Apr;55(4):512–21. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15048890
  11. Doshi P, Goodman SN, Ioannidis JPA. Raw data from clinical trials: within reach? Trends Pharmacol Sci. 2013 Dec;34(12):645–7. Available from: http://dx.doi.org/10.1016/j.tips.2013.10.006
  12. Ioannidis JPA, Allison DB, Ball CA, Coulibaly I, Cui X, Culhane AC, et al. Repeatability of published microarray gene expression analyses. Nat Genet. 2009;41(2):149–55.
  13. Open Science Collaboration. PSYCHOLOGY. Estimating the reproducibility of psychological science. Science. 2015 Aug 28;349(6251):aac4716. Available from: http://www.ncbi.nlm.nih.gov/pubmed/26315443
  14. Ioannidis JPA. The Reproducibility Wars: Successful, Unsuccessful, Uninterpretable, Exact, Conceptual, Triangulated, Contested Replication. Clin Chem. 2017;63(5):943–5. Available from: http://www.ncbi.nlm.nih.gov/pubmed/28298413
  15. Ioannidis JPA. The importance of potential studies that have not existed and registration of observational data sets. JAMA. 2012 Aug 8;308(6):575–6. Available from: http://www.ncbi.nlm.nih.gov/pubmed/22871867
  16. Vanden Heuvel JP, Bullenkamp J, Reproducibility Project: Cancer Biology. Registered report: Systematic identification of genomic markers of drug sensitivity in cancer cells. Elife. 2016;5(JUNE2016):1–19. Available from: http://www.ncbi.nlm.nih.gov/pubmed/27336789
  17. Hardwicke TE, Ioannidis JPA. Mapping the universe of registered reports. Nat Hum Behav. 2018;2(11):793–6. Available from: http://www.ncbi.nlm.nih.gov/pubmed/31558810
  18. Ioannidis JPA. How to make more published research true. PLoS Med. 2014 Oct;11(10):e1001747. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204808/

Before you go, don’t forget to

  • share widely, options below
  • leave a comment and join the #FOAMed conversation in general
  • connect with the scanFOAM project and team on Twitter, Facebook, Instagram or write us an old fashioned email
  • subscribe to thenew stuff.
Spread the love
  • 25
  • 36
  •  
  •  
  •  
  •  
  •  
Mads Astvad
Scandinavian anaesthetist. PHARM, ED, OR, ICU. Special interest in wee patients.
Mads Astvad on LinkedinMads Astvad on TwitterMads Astvad on Youtube
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments