• Book

Reproducibility: A primer on semantics and implications for research

Citation

Pellizzari, E., Lohr, K., Blatecky, A., & Creel, D. (2017). Reproducibility: A primer on semantics and implications for research. (RTI Press Publication No. BK-0020-1708). Research Triangle Park, NC: RTI Press. DOI: 10.3768/rtipress.2017.bk.0020.1708

Abstract

Science is allegedly in the midst of a reproducibility crisis, but questions of reproducibility and related principles date back nearly 80 years. Numerous controversies have arisen, especially since 2010, in a wide array of disciplines that stem from the failure to reproduce studies or their findings:biology, biomedical and preclinical research, business and organizational studies, computational sciences, drug discovery, economics, education, epidemiology and statistics, genetics, immunology, policy research, political science, psychology, and sociology.

This monograph defines terms and constructs related to reproducible research, weighs key considerations and challenges in reproducing or replicating studies, and discusses transparency in publications that can support reproducible researchgoals. It attempts to clarify reproducible research, with its attendant (andconfusing or even conflicting) lexicon and aims to provide useful background, definitions, and practical guidance for all readers.

Among its conclusions: First, researchers must become better educated about these issues, particularly the differences between the concepts and terms. The main benefit is being able to communicate clearly within their own fields and, more importantly, across multiple disciplines. In addition, scientists need to embrace these concepts as part of their responsibilities as good stewards of research funding and as providers of credible information for policy decision making across many areas of public concern. Finally, although focusing on transparency and documentation is essential, ultimately the goal is achieving the most rigorous, high-quality science possible given limitations on time, funding, or other resources.

Author Details

E Pellizzari

Edo D. Pellizzari, PhD, is Lead Fellow (Emeritus) in the RTI International Fellow Program. His nearly 40 years of research experience at RTI spans biochemistry, analytical chemistry, and environmental health. His research focused on exposure of children and adults to toxic chemicals such as volatile organic chemicals (e.g., benzene, chloroform, trichloroethylene, tetrachloroethylene, and bromodichloromethane), polynuclear aromatic hydrocarbons (e.g., benzopyrene), polychlorinated biphenyls, and heavy metals (e.g., lead, mercury, cadmium, arsenic, and manganese in air, food, and drinking water), and PM10 and PM2.5 particles in air. Dr. Pellizzari developed qualitative and quantitative analytical chemistry methods and questionnaires for measuring personal exposure. He applied these methods to 14 probability-based human population studies throughout the United States and Canada. Dr. Pellizzari has served on several National Research Council committees; as chair of the Environmental Health Sciences Committee for the National Institute of Environmental Health Sciences (NIEHS) from 2006 to 2008; as a member of the Health Effects Institute (HEI) from 1998 to 2007; and as a member of the US EPA Science Advisory Board’s Drinking Water Committee from 1987 to 1997. He has served on 40 scientific review panels for NIEHS, EPA, the Department of Energy, and the National Institute of Occupational Safety and Health. He has authored or coauthored more than 240 scientific publications. He received his AB in biology from California State University, Chico, and his PhD in biochemistry from Purdue University; he was also a Fulbright-Hays recipient and a Fellow of the US Public Health Service.

Kathleen Lohr

Kathleen N. Lohr, PhD, MPhil, MA, is a Distinguished Fellow at RTI International with more than 40 years of experience in health care and health care policy research. She was the founding director of (and is now senior advisor to) the RTI–University of North Carolina (UNC) Evidence-based Practice Center. Dr. Lohr was also the founding editor-in-chief of RTI Press, a corporate-wide initiative. In 2007, RTI honored her with the Margaret Elliott Knox Excellence Award. Before coming to RTI, her 9 years at the Institute of Medicine, National Academy of Sciences, entailed overseeing the Division of Health Care Services’ portfolio of studies in health care delivery, organization, financing, quality of care and clinical evaluation, practice guidelines, health workforce, public health, and related topics. During her 12 years at the RAND Corporation, she led or participated in numerous health care projects for the US Department of Health and Human Services, Department of Defense, and the congressional Office of Technology Assessment. She is the author or coauthor or more than 350 books, monographs, chapters, journal articles, and other publications. In 2005, she was awarded the Avedis Donabedian Outcomes Research Lifetime Achievement Award by the International Society of Pharmacoeconomics and Outcomes Research. She served a 3-year term (2008–2010) as a member of the National Advisory Council, Agency for Healthcare Research and Quality, and she continues to serve on several expert panels and advisory boards for national and international groups. For several years at UNC-Chapel Hill, Dr. Lohr held the rank of Research Professor, Health Policy and Administration; she remains an adjunct professor in Health Policy and Management in the Gillings School of Global Public Health and a Senior Research Fellow at the UNC Cecil G. Sheps Center for Health Services Research. In 2016, she was appointed Adjunct Research Professor in the Department of Medical Informatics and Clinical Epidemiology at Oregon Health & Science University in Portland.

Alan Blatecky

Alan Blatecky, MBA, MA, ThM, MDiv, is a Visiting Fellow at RTI International; he previously directed the NSF Office of Cyberinfrastructure at the National Science Foundation. Mr. Blatecky is also a member of the Steering Committee for the National Data Service and the US Research Data Alliance. He helped co-found the Renaissance Computing Institute and served as the Deputy Director; he was also the Executive Director of Research and Programs at the San Diego Supercomputing Center, and Vice President of Information Technology at MCNC. He played a central role in establishing the NC Research and Education Network and the North Carolina Supercomputing Center. His education includes an MBA from the Fuqua School of Business, an MA in Liberal Studies from Duke University, and a ThM and an MDiv from Princeton Theological Seminary.

Darryl Creel

Darryl V. Creel, MS, is a Senior Research Statistician at RTI International. He has 19 years of experience working in the statistical field with a focus on survey sampling and biostatistical research. His general statistical experience includes survey sampling, data visualization, data analysis, and statistical learning. His statistical experience has been applied to a wide spectrum of research projects, ranging from large national surveys to relatively small surveys of highly specific populations.

Praise

“The authors have written a nuanced and thoughtful primer on scientific reproducibility. By highlighting the social, political, and technical importance of reproducibility, together with a precise description of the related concepts of reproducibility, replicability, and repeatability, this primer provides a significant resource that all practicing researchers should read.”

--Daniel Reed, Vice President for Research and Economic Development, University of Iowa and former Corporate Vice President, Microsoft

“This is a well-written, clearly articulated, and timely primer on the developing and evolving rich terminology of reproducible research. The primer, put together by authors with deep experience and expertise in the topic area, focuses primarily on human-centric research in biomedicine, medicine, and the social sciences as well as reproducibility issues in analytics and computational science. The growing focus on reproducibility will open new vistas in research methodologies, meta analysis, comparative studies of research results, and reuse and adaptation of results from prior research. This primer provides an excellent overview of the subject area, and I would recommend it to anyone interested in coming up to speed on current issues in reproducible research.”

--Chaitan Baru, Distinguished Scientist and Associate Director for Data Initiatives, San Diego Supercomputing Center; current appointment as Senior Advisor for Data Science, Computer and Information Science and Engineering Directorate, National Science Foundation

“Pellizzari et al. have taken on the Herculean task of collecting, synthesizing, and relating the various interpretations of reproducibility used in the research community today, and turned the result into an accessible must-read guide. This important work provides a Rosetta Stone for various stakeholders to discuss and implement solutions that make real progress toward a research enterprise that routinely produces reproducible findings.”

--Victoria Stodden, Associate Professor at the School of Information Sciences, University of Illinois at Urbana Champaign and co-editor of the books Implementing Reproducible Research and Privacy, Big Data, and the Public Good: Frameworks for Engagement