This was the opening keynote at the 2019 scientific assembly of the Scandinavian Society of Anaesthesia and Intensive care and so makes sense to be the first talk online of the select talks we’ll be releasing in the coming weeks.
Professor Ioannidis will be know to most for pioneering metascience, ie the study of the scientific process itself, and we feel extremely fortunate that he was able and willing to give 2 talks at the SSAI meeting.
This first talk goes over a number of systemic issues Prof Ioannidis and co-workers have described which keep us from reaching and applying sufficient meaningful insight from to the vast human and economic resources applied to biomedical science. Fortunately, Prof Ioannidis has ideas on how we can remedy the situation.
It will serve as a good introduction for the uninitiated to the importance of Prof Ioannidis’ work. The list of critical and fundamental problems in science is long and they are a concern for everyone in medicine, whether you are actively involved in a research capacity or work clinically and thus find yourself on the receiving end of the translation of studies to the bedside.
If you feel ready to go further down the rabbit hole find below more resources and a bibliography of articles referenced in the talk as a starting point.
It is hard to overestimate the importance and the reach of Prof Ioannidis’ work. He has been aptly described as “the scourge of sloppy science”.
He shot to fame in 2005 with his article “Why most published research findings are wrong” (open access here), to this day the most downloaded article from PLoS and one of the most cited articles in the biomedical literature ever.
Formal titels at Stanford include
- C.F. Rehnborg Chair in Disease Prevention
- Professor of Medicine, of Health Research and Policy, of Biomedical Data Science, and of Statistics
- co-Director, Meta-Research Innovation Center at Stanford
- Director of the PhD program in Epidemiology and Clinical Research.
If you’ve got a few weeks, pull up the full list of 1080 articles on Pubmed or check the 130 page CV on his profile page at Stanford. As important as this work is, though, and it truly is, I particularly admire him for these few lines in his CV which speak to a genuinely curious and humble mind as also confirmed by his public appearances
Current citation rates suggest that I am among the 10 scientists worldwide who are currently the most commonly cited, perhaps also the currently most-cited physician. This probably only proves that citation metrics are highly unreliable, since I estimate that I have been rejected over 1,000 times in my life. Regardless, I consider myself privileged to have learned and to continue to learn from interactions with students and young scientists (of all ages) from all over the world and I love to be constantly reminded that I know next to nothing.
Other talks from Prof Ioannidis
- Google, 2014: Reproducible Research: True or False? (youtube)
- Lown Institute, 2016: Evidence-Based Medicine Has Been Hijacked (youtube)
- CEBM Oxford, 2016: Why most clinical research is not useful (youtube)
If you’re more of the podcast type, try this 2018 STEM interview with Ioannidis about his work and views, recorded by the Florida Institute for Human & Machine Cognition.
Of course, Prof Ioannidis is by no account alone in his endeavours. Similar critiques of healthcare science and the lacklustre state of evidence based medicine are raised by many. Another notably popular researcher and communicator that comes to mind is Ben Goldacre (eg TED talk on battling bad science here, with more accounts of bad science to be found in his books).
For a short and accessible introduction to some of the issues like p-hacking and the reproducibility crisis I might recommend this video by Veratasium.
The FOAMed landscape of course is ripe with critical appraisal projects and frequently concerns itself with the checks and balances of good scientific conduct and translating science to the bedside.
Missed another favourite resource? Put it in the comments!
Articles referenced in the talk
- Ioannidis JPA. Why Most Clinical Research Is Not Useful. PLoS Med. 2016;13(6):1–10. Available from: https://www.ncbi.nlm.nih.gov/pubmed/27328301
- Ioannidis JPA. Evidence-based medicine has been hijacked: A report to David Sackett. J Clin Epidemiol [Internet]. 2016;73:82–6. Available from: http://dx.doi.org/10.1016/j.jclinepi.2016.02.012
- Fleming PS, Koletsi D, Ioannidis JPA, Pandis N. High quality of the evidence for medical and other health-related interventions was uncommon in Cochrane systematic reviews. J Clin Epidemiol [Internet]. 2016;78(2016):34–42. Available from: http://dx.doi.org/10.1016/j.jclinepi.2016.03.012
- Ioannidis JPA, Stuart ME, Brownlee S, Strite SA. How to survive the medical misinformation mess. Eur J Clin Invest. 2017;47(11):795–802. Available from: https://www.ncbi.nlm.nih.gov/pubmed/28881000
- Flacco ME, Manzoli L, Boccia S, Capasso L, Aleksovska K, Rosso A, et al. Head-to-head randomized trials are mostly industry sponsored and almost always favor the industry sponsor. J Clin Epidemiol [Internet]. 2015;68(7):811–20. Available from: http://dx.doi.org/10.1016/j.jclinepi.2014.12.016
- Ioannidis JPA. The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. Milbank Q [Internet]. 2016;94(3):485–514. Available from: http://www.ncbi.nlm.nih.gov/pubmed/27620683
- Lenzer J, Hoffman JR, Furberg CD, Ioannidis JPA, Guideline Panel Review Working Group. Ensuring the integrity of clinical practice guidelines: a tool for protecting patients. BMJ [Internet]. 2013 Sep 17;347(September):f5535. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24046286
- Goodman SN, Fanelli D, Ioannidis JPA. What does research reproducibility mean? Sci Transl Med [Internet]. 2016;8(341):341ps12. Available from: http://www.ncbi.nlm.nih.gov/pubmed/27252173
- Ioannidis JPA. How to make more published research true. PLoS Med [Internet]. 2014 Oct;11(10):e1001747. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204808/
- Patel CJ, Burford B, Ioannidis JPA. Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations. J Clin Epidemiol [Internet]. 2015 Sep;68(9):1046–58. Available from: http://www.ncbi.nlm.nih.gov/pubmed/26279400
- Vanden Heuvel JP, Bullenkamp J, Reproducibility Project: Cancer Biology. Registered report: Systematic identification of genomic markers of drug sensitivity in cancer cells. Elife [Internet]. 2016;5(JUNE2016):1–19. Available from: http://www.ncbi.nlm.nih.gov/pubmed/27336789
- Le Noury J, Nardo JM, Healy D, Jureidini J, Raven M, Tufanaru C, et al. Restoring Study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence. BMJ [Internet]. 2015 Sep 16;351:h4320. Available from: http://www.ncbi.nlm.nih.gov/pubmed/26376805
- Naudet F, Sakarovitch C, Janiaud P, Cristea I, Fanelli D, Moher D, et al. Data sharing and reanalysis of randomized controlled trials in leading biomedical journals with a full data sharing policy: survey of studies published in The BMJ and PLOS Medicine. BMJ [Internet]. 2018;360(January 2013):k400. Available from: http://www.ncbi.nlm.nih.gov/pubmed/29440066
- Hardwicke TE, Ioannidis JPA. Populating the Data Ark: An attempt to retrieve, preserve, and liberate data from the most highly-cited psychology and psychiatry articles. PLoS One [Internet]. 2018;13(8):e0201856. Available from: http://www.ncbi.nlm.nih.gov/pubmed/30071110
- Amann RI, Baichoo S, Blencowe BJ, Bork P, Borodovsky M, Brooksbank C, et al. Toward unrestricted use of public genomic data. Science [Internet]. 2019;363(6425):350–2. Available from: http://www.ncbi.nlm.nih.gov/pubmed/30679363
- Ioannidis JPA. Stealth research: is biomedical innovation happening outside the peer-reviewed literature? JAMA [Internet]. 2015 Feb 17;313(7):663–4. Available from: http://www.ncbi.nlm.nih.gov/pubmed/25688775
- Cristea IA, Cahan EM, Ioannidis JPA. Stealth research: Lack of peer-reviewed evidence from healthcare unicorns. Eur J Clin Invest [Internet]. 2019 Apr;49(4):e13072. Available from: http://www.ncbi.nlm.nih.gov/pubmed/30690709
- Iqbal SA, Wallach JD, Khoury MJ, Schully SD, Ioannidis JPA. Reproducible Research Practices and Transparency across the Biomedical Literature. PLoS Biol [Internet]. 2016 Jan;14(1):e1002333. Available from: http://www.ncbi.nlm.nih.gov/pubmed/26726926
- Wallach JD, Boyack KW, Ioannidis JPA. Reproducible research practices, transparency, and open access data in the biomedical literature, 2015-2017. PLoS Biol [Internet]. 2018;16(11):e2006930. Available from: http://www.ncbi.nlm.nih.gov/pubmed/30457984
- Stodden V, McNutt M, Bailey DH, Deelman E, Gil Y, Hanson B, et al. Enhancing reproducibility for computational methods. Science [Internet]. 2016;354(6317):1240–1. Available from: http://www.ncbi.nlm.nih.gov/pubmed/27940837
- Bzdok D, Ioannidis JPA. Exploration, Inference, and Prediction in Neuroscience and Biomedicine. Trends Neurosci [Internet]. 2019;42(4):251–62. Available from: https://doi.org/10.1016/j.tins.2019.02.001
- Chavalarias D, Wallach JD, Li AHT, Ioannidis JPA. Evolution of Reporting P Values in the Biomedical Literature, 1990-2015. JAMA [Internet]. 2016 Mar 15;315(11):1141–8. Available from: http://www.ncbi.nlm.nih.gov/pubmed/26978209
- Ioannidis JPA. The Proposal to Lower P Value Thresholds to .005. JAMA [Internet]. 2018;319(14):1429–30. Available from: http://www.ncbi.nlm.nih.gov/pubmed/29566133
- Wang MQ, Yan AF, Katz R V. Researcher Requests for Inappropriate Analysis and Reporting: A U.S. Survey of Consulting Biostatisticians. Ann Intern Med [Internet]. 2018;169(8):554–8. Available from: http://www.ncbi.nlm.nih.gov/pubmed/30304365
- Gall T, Ioannidis JPA, Maniadis Z. The credibility crisis in research: Can economics tools help? PLoS Biol [Internet]. 2017;15(4):e2001846. Available from: http://www.ncbi.nlm.nih.gov/pubmed/28445470
- Grimes DR, Bauch CT, Ioannidis JPA. Modelling science trustworthiness under publish or perish pressure. R Soc open Sci [Internet]. 2018 Jan;5(1):171511. Available from: http://www.ncbi.nlm.nih.gov/pubmed/29410855
- Ioannidis JPA, Khoury MJ. Assessing value in biomedical research: the PQRST of appraisal and reward. JAMA [Internet]. 2014 Aug 6;312(5):483–4. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24911291
- Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, Percie du Sert N, et al. A manifesto for reproducible science. Nat Hum Behav [Internet]. 2017 Jan 10;1(1):0021. Available from: http://dx.doi.org/10.1038/s41562-016-0021
Before you go, don’t forget to: