"We know that scientific papers regularly contain errors. One algorithm that ran through thousands of psychology papers found that, at worst, more than 50% had one specific statistical error, and more than 15% had an error serious enough to overturn the results. ... Imagine the number of errors that litter the scientific literature that haven’t been corrected because to do so is just too much hassle." -- The Guardian.
The Handbook of Research Synthesis and Meta-Analysis by Call Number: e-BookISBN: 9781610441384Publication Date: 2009-02-05
Errors in Evidence-Based Decision Making by Call Number: e-BookISBN: 9781475810820Publication Date: 2014-06-25Preface; Acknowledgments; Introduction; 1 Identifying Errors; 2 Dispositions and Ethics; 3 Problem Development; 4 Variables; 5 Measurement Foundations; 6 Data Collection Instruments; 7 Sampling Methods; 8 External and Internal Validity Threats to Evidence; 9 Research Design; 10 Data Analysis; 11 Statistical Probability and Interpretation; 12 Personal Biases in Interpretation; 13 The Age of Misinformation; 14 Religion and Science; Epilogue; Index.
The Essential Guide to Effect Sizes by Call Number: e-BookISBN: 9780511929793Publication Date: 2010-07-01
- Catalogue of Bias© 2021 Centre for Evidence-Based Medicine
- Gene name errors: Lessons not learnedAbeysooriya M, Soria M, Kasu MS, Ziemann M (2021) Gene name errors: Lessons not learned. PLoS Comput Biol 17(7): e1008984. https://doi.org/10.1371/journal.pcbi.1008984
Article discusses the problem of spreadsheet software converting gene symbols to dates and floating point numbers. - How scientists can stop fooling themselves over statisticsNature 584, 9 (2020), doi: 10.1038/d41586-020-02275-8
- Issues with data and analyses: Errors, underlying themes, and potential solutionsPNAS March 13, 2018 115 (11) 2563-2570; first published March 12, 2018 https://doi.org/10.1073/pnas.1708279115
- Letter to the editor Tanya M. Halliday, Diana M. Thomas, Cynthia O. Siu & David B. AllisonTanya M. Halliday, Diana M. Thomas, Cynthia O. Siu & David B. Allison (2018) Letter to the editor, Journal of Women & Aging, 30:1, 2-5, DOI: 10.1080/08952841.2017.1407575
- Meta-Research: Why we need to report more than 'Data were Analyzed by t-tests or ANOVA'eLife 2018;7:e36163 DOI: 10.7554/eLife.36163
- Regression to the Mean: Definition, ExamplesStephanie Glen. "Regression to the Mean: Definition, Examples" From StatisticsHowTo.com: Elementary Statistics for the rest of us! https://www.statisticshowto.com/regression-mean/
- Statistical Errors in Clinical StudiesJ Wrist Surg. 2013 Nov; 2(4): 285–287.
doi: 10.1055/s-0033-1359421
PMCID: PMC3826246
PMID: 24436830 - Threats of Bots and Other Bad Actors to Data Quality Following Research Participant Recruitment Through Social Media: Cross-Sectional QuestionnairePozzar R, Hammer MJ, Underhill-Blazey M, Wright AA, Tulsky JA, Hong F, Gundersen DA, Berry DL
Threats of Bots and Other Bad Actors to Data Quality Following Research Participant Recruitment Through Social Media: Cross-Sectional Questionnaire J Med Internet Res 2020;22(10):e23021 URL: https://www.jmir.org/2020/10/e23021
DOI: 10.2196/23021 PMID: 33026360 - The “regression to the mean project:” What researchers should know about a mistake many makeRetraction Watch, Posted on October 30, 2018, Ivan Oransky
- Common Flaws in Running Human Evaluation Experiments in NLPComputational Linguistics, 1-10, by Craig Thomson, Ehud Reiter, Anya Belz.
- Could machine learning fuel a reproducibility crisis in science?Nature.com, Elizabeth Gibney, 7/26/2022
- Healthcare Triage: Episode 4: Problems in Science PublishingIncludes links to podcast, transcript & lesson guide
- Three questions to address rigour and reproducibility concerns in your grant proposalNature.com, Jennifer L. Wilson & Crystal M. Botham, 8/19/2021
- ‘A display of extreme academic integrity’: A grad student who found a key error praises the original authorRetraction Watch, Paul Lodder, 10/11/2022
- The grad student who found a fatal error that may affect lots of papersRetraction Watch, Adam Marcus, 12/11/2020
- ‘I thought I had messed up my experiment’: How a grad student discovered an error that might affect hundreds of papersRetraction Watch, Adam Marcus, 12/23/2020
- Reproducibility: A tragedy of errorsNature. Author manuscript; available in PMC 2016 Aug 4. Published in final edited form as: Nature. 2016 Feb 4; 530(7588): 27–29. doi: 10.1038/530027a PMCID: PMC4831566
NIHMSID: NIHMS772869 PMID: 26842041 - Reproducibility vs. Replicability: A Brief History of a Confused TerminologyFront Neuroinform. 2017; 11: 76.
Published online 2018 Jan 18. doi: 10.3389/fninf.2017.00076
PMCID: PMC5778115 PMID: 29403370 - ‘Robot scientist’ Eve finds that less than one third of scientific results are reproducible"A successful experiment is one where another scientist, in a different laboratory under similar conditions, can achieve the same result. But more than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce some of their own experiments: this is the reproducibility crisis." -- DOI 10.1098/rsif.2021.0821
- Social psychology in the age of retractionRetraction Watch Augustine Brannigan, 6/9/2021
Summary: discusses replicability, hypothesizing after the results are known, and verification bias. - Software updates: the “unknown unknown” of the replication crisishttps://blogs.lse.ac.uk/ Anastasia Ershova, Gerald Schneider, 6/7/2018
- The Tadpole Paper Mill"... critical screening of images and other data should be an essential part of the pre-publication process of scientific papers. Scientists pay money to either publish or read these papers (depending on whether they are paywalled or open access), and they might spend months or years trying to replicate these results. " -- Science Integrity Digest blog entry by Elisabeth Bik, 2/21/2020
- Why “good PhD students are worth gold!” A grad student finds an errorRetractionWatch.com, Adam Marcus 2/8/2021
The Practice of Reproducible Research by Call Number: e-BookISBN: 9780520967779Publication Date: 2017-10-17
Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results by Call Number: e-BookISBN: 9780309392020Publication Date: 2016-03-29
Stepping in the Same River Twice by Call Number: e-BookISBN: 9780300228038Publication Date: 2018-10-08"A major issue in science has always been a "lack of reproducibility." At the heart of Stepping in the Same River Twice is a call for setting standards within and across disciplines in order to generate stronger and more "trustworthy" data sets."--Choice Review
- Investigating and preventing scientific misconduct using Benford’s LawEckhartt, G.M., Ruxton, G.D. Investigating and preventing scientific misconduct using Benford’s Law. Res Integr Peer Rev 8, 1 (2023). https://doi.org/10.1186/s41073-022-00126-w
- Not the First Digit! Using Benford's Law to Detect Fraudulent DataDiekmann, Andreas. “Not the First Digit! Using Benford’s Law to Detect Fraudulent Data.” Conference Papers -- American Sociological Association, Aug. 2005, pp. 1–26.
- Using Benford's Law to Detect FraudACFE (Association of Certified Fraud Examiners)
- Detecting Academic Fraud Using Benford Law: The Case of Professor James HuntonDiekmann, A. (2005). Not the First Digit! Using Benford’s Law to Detect Fraudulent Data. Conference Papers -- American Sociological Association, 1–26.
- Immortal time biasCatalogue of Bias Collaboration. Lee H, Nunan D. Immortal time bias. In: Catalogue of Bias 2020. https://catalogofbias.org/biases/immortaltimebias/
- Avoiding blunders involving ‘immortal time’Hanley, J. A., & Foster, B. J. (2014). Avoiding blunders involving 'immortal time'. International journal of epidemiology, 43(3), 949–961. https://doi.org/10.1093/ije/dyu105
- ENCePP Guide on Methodological Standards in Pharmacoepidemiology: 4.2.2.2.1. Immortal time biasENCPP (European Network of Centres for Pharmacoepidemiology and Pharmacovigilance
- An Example of How Immortal Time Bias Can Reverse the Results of an Observational Study (Letter to the Editor)Airaksinen, Jaakko; Pentti, Jaana; Suominen, Sakari; Vahtera, Jussi; Kivimäki, Mika An Example of How Immortal Time Bias Can Reverse the Results of an Observational Study, Epidemiology: March 2020 - Volume 31 - Issue 2 - p e19-e20 doi: 10.1097/EDE.0000000000001103
- ‘Immortal time bias’ fells JAMA journal asthma paperRetractionWatch.com, Adam marcus, 1/27/2021
- Immortal Time Bias in PharmacoepidemiologySamy Suissa, Immortal Time Bias in Pharmacoepidemiology, American Journal of Epidemiology, Volume 167, Issue 4, 15 February 2008, Pages 492–499, https://doi.org/10.1093/aje/kwm324
- Understanding immortal time bias in observational cohort studies (Editorial)Anaesthesia 2013, 68 Anaesthesia © 2012 The Association of Anaesthetists of Great Britain and Ireland doi:10.1111/anae.12120
- Immortal Time Bias Can Occur in the Analysis of Dose-Dependent Effect of Statin (Letter to the Editor)153 #6 CHEST JUNE 2018 American College of Chest Physicians. Published by Elsevier Inc. DOI: https://doi.org/10.1016/j.chest.2018.03.009
- Immortal Time Bias in Observational Studies of Time-to-Event OutcomesAgarwal, P., Moshier, E., Ru, M., Ohri, N., Ennis, R., Rosenzweig, K., & Mazumdar, M. (2018). Immortal Time Bias in Observational Studies of Time-to-Event Outcomes: Assessing Effects of Postmastectomy Radiation Therapy Using the National Cancer Database. Cancer control : journal of the Moffitt Cancer Center, 25(1), 1073274818789355. https://doi.org/10.1177/1073274818789355
- The Secret of Immortal Time Bias in Epidemiologic StudiesThe Secret of Immortal Time Bias in Epidemiologic Studies Salimah Z. Shariff, Meaghan S. Cuerden, Arsh K. Jain, Amit X. Garg JASN May 2008, 19 (5) 841-843; DOI: 10.1681/ASN.2007121354
- An unjustified benefit: immortal time bias in the analysis of time-dependent eventsGleiss, A., Oberbauer, R., & Heinze, G. (2018). An unjustified benefit: immortal time bias in the analysis of time-dependent events. Transplant International : Official Journal of the European Society for Organ Transplantation, 31(2), 125–130. https://doi.org/10.1111/tri.13081
- Simpson’s ParadoxStanford Encyclopedia of Philosophy, 3/24/2021
- Understanding Simpson’s ParadoxTechnical Report R-414, 12/2013, Judea Pearl, University of California, Los Angeles
"The vast majority of neuroimaging studies are underpowered and rarely produce results above noise. The odds that an average neuroscience study is true is 50-50 or lower, according to a 2013 review. And an estimated 50% of studies in biomedicine “have statistical power in the 0–10% or 11–20% range, well below the minimum of 80% that is often considered conventional.” P-hacking and hypothesis fishing are rewarded because they make for more remarkable, and therefore more publishable, results (Figure 2)." -- New Science blog post by Niko McCarty, 9/6/2022
- The Extent and Consequences of P-Hacking in ScienceHead ML, Holman L, Lanfear R, Kahn AT, Jennions MD (2015) The Extent and Consequences of P-Hacking in Science. PLoS Biol 13(3): e1002106. https://doi.org/10.1371/journal.pbio.1002106
- P-Hacking: A Talk and Further ThoughtsDiscovermagazine.com, 5/18/2015
- P-hacking in clinical trials and how incentives shape the distribution of results across phasesPNAS, 117 (24)13386-13392 Jérôme Adda, Christian Decker, Marco Ottaviani,
- P-Hacking and the Problem of Multiple ComparisonsAnderson, B. (n.d.). P-Hacking and the Problem of Multiple Comparisons [web log]. Retrieved April 28, 2022, from https://info.umkc.edu/drbanderson/p-hacking-and-the-problem-of-multiple-comparisons/.
- Statistical P-hacking explainedWorld Editors Forum. (2021). Statistical P-hacking explained [web log]. Retrieved April 28, 2022, from https://scienceinthenewsroom.org/resources/statistical-p-hacking-explained/.
- An Overview of Methods to Address the Multiple Comparison ProblemZheng, P. (2018, August 10). An Overview of Methods to Address the Multiple Comparison Problem [web log]. Retrieved April 28, 2022, from https://towardsdatascience.com/an-overview-of-methods-to-address-the-multiple-comparison-problem-310427b3ba92.
- How (not) to deal with missing data: An economist’s take on a controversial studyRetraction Watch, Gary Smith, 2/21/2024
- No data? No problem! Undisclosed tinkering in Excel behind economics paperRetraction Watch, by Frederik Joelving, 2/6/2024
- Prevention and handling of the missing dataKang H. The prevention and handling of the missing data. Korean J Anesthesiol. 2013 May;64(5):402-6. doi: 10.4097/kjae.2013.64.5.402. Epub 2013 May 24. PMID: 23741561; PMCID: PMC3668100.
- [98] Evidence of Fraud in an Influential Field Experiment About DishonestySimonsohn, U., Nelson, J., & Simmons, L. (2021, August 17). [98] Evidence of Fraud in an Influential Field Experiment About Dishonesty. http://datacolada.org/98.
- A Famous Honesty Researcher Is Retracting A Study Over Fake DataLee, S. M. (2021, August 20). A big study about honesty turns out to be based on fake data. BuzzFeed News. https://www.buzzfeednews.com/article/stephaniemlee/dan-ariely-honesty-study-retraction.
- Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the endLisa L. Shu, Nina Mazar, Francesca Gino, Dan Ariely, Max H. Bazerman
Proceedings of the National Academy of Sciences Sep 2012, 109 (38) 15197-15200; DOI: 10.1073/pnas.1209746109 - Misconduct and Questionable Research Practices: The Ethics of Quantitative Data Handling and Reporting in Applied Linguistics" To investigate the extent of misconduct and QRPs in quantitative applied linguistics research, we surveyed 351 applied linguists who conduct quantitative research about their practices related to data handling and reporting. We found that 17% of respondents (approximately 1 in 6) admitted to 1 or more forms of scientific misconduct and that 94% admitted to 1 or more QRPs relevant to quantitative research." -- https://doi.org/10.1111/modl.12760 [QRPs=Questionable Research Practices]
- Data is not available upon request"Many journals now require data sharing and require articles to include a Data Availability Statement. However, several studies over the past two decades have shown that promissory notes about data sharing are rarely abided by, and that data is generally not available upon request." -- Abstract. Hussey, I. (2023, May 8). Data is not available upon request. https://doi.org/10.31234/osf.io/jbu9r