US Government Resources Related to Research Rigor and Reproducibility

Alicia Kubas, Amy Riegelman, and Franklin Sayre, University of Minnesota

The reproducibility of scientific studies has recently come under increased scrutiny in both the popular and scientific press.1 Studies from various disciplines (e.g., psychology, health sciences) have revealed failures to reproduce and replicate research.2 This has led to declarations that science is experiencing a “reproducibility crisis” and that this crisis has negative consequences for science, the public, and public policy. Two of the authors have previously published on reproducibility and the services and expertise librarians and libraries offer that make the library community a key part of supporting reproducible research, and we direct you to these articles for more information on this broader topic.3

In this article we extend this work to government documents by outlining some of the government publications and information related to reproducibility. Funders, including government agencies, have responded with specific recommendations and guidelines for researchers and we include some of those materials here. This issue has also been politicalized by actors and groups looking to undermine science and evidence-based policy. Therefore these issues have increased relevance for all librarians and the public at large.

Briefly, we adopt a distinction between reproducibility and replicability where reproducibility is defined as “the ability of a researcher to duplicate the results of a prior study using the same materials and procedures as were used by the original investigator.” Whereas replicability is defined as “the ability of a researcher to duplicate the results of a prior study if the same procedures are followed but new data are collected.”4 Thus, reproducibility largely relates to accurate reporting and transparency and is especially relevant to the work of librarians as demand rises for guidance and assistance with reproducibility within the research lifecycle. This often entails assistance with managing data and data sharing requirements, finding repositories for pre-registering studies and analysis, and finding discipline-specific guidelines for what to report in a research publication to promote transparency and reproducibility. A call for data availability and transparency of data used for federal policy-making has grown out of the research publishing world where data transparency and sharing has become a best practice across disciplines. Libraries are uniquely situated to assist faculty, researchers, and graduate students in thinking and moving proactively through the research lifecycle with these tenets in mind.

A number of the government publications included below refer to “sound science” and “questionable research practices.” It is important to understand the context behind these phrases. The phrase sound science is sometimes used to create scientific doubt and therefore limit or delay government action, and examples include attempts to stall regulation related to secondhand smoke and climate change.5 Questionable research practices, often shortened to QRPs, are referenced by government officials (as noted in this article) but also widely referenced by researchers. QRPs could refer to criticized research practices; some examples include p-hacking (manipulating data in a certain way to induce a statistical significant p value during analysis), intentionally excluding data, and HARKing (hypothesis after results are known). The more blatant and flagrant practices would include falsification and fabrication of data and plagiarism.

With this topic appearing in the popular press, this source guide can be used by government information librarians and others to familiarize themselves with the government resources on this topic. The resources below give an idea of which agencies are discussing and responding to this issue as well as how Congress is looking at implementing laws to govern agency use of scientific data and what agencies should be doing to promote reproducibility and sound science.

Agency Initiatives and Grant Funding

A Framework for Ongoing and Future National Science Foundation Activities to Improve Reproducibility, Replicability, and Robustness in Funded Research, Office of Management and Budget, December 31, 2014

The Framework summarizes reproducibility, replicability, and robustness in relationship to confidence of published research results. It emphasizes that different practices exist across various fields and subfields. This framework notes that the key elements include scientific question formulation, data collection, data sharing, data curation, instruments, models, analysis of findings, and research outputs. Regarding data, the text emphasizes exploration of open access policy changes. Regarding interpretation of findings, “directorates will launch initiatives to introduce language into solicitations requesting that research plans describe how datasets would be assembled and analyzed” as to prevent data dredge (otherwise known as searching for statistical significance or HARKing). The Framework also mentions the need to disseminate studies with negative results stating, “NSF will also explore policy changes that would encourage researchers to include null findings and negative findings in project reports.” Access: https://www.nsf.gov/attachments/134722/public/Reproducibility_NSFPlanforOMB_Dec31_2014.pdf.

Rigor and Reproducibility, National Institutes of Health (NIH), Undated (Internet Archive first capture on October 31, 2015 shows identical information)

This is the portal for information related to rigor and reproducibility in research at the National Institutes of Health. It includes further information and links to reporting guidelines, grant application requirements, funding opportunities, training modules, announcements, meetings, workshops, publications, and resources. Access: https://www.nih.gov/research-training/rigor-reproducibility.

Social, Behavioral, and Economic Sciences Perspectives on Robust and Reliable Science, National Science Foundation, May 2015

This report was written by the Subcommittee on Replicability in Science Advisory Committee to the National Science Foundation (NSF) Directorate for Social, Behavioral, and Economic Sciences (SBE). This subcommittee was launched to investigate ways to promote “robust research practices.” One result of this committee was a workshop that occurred in February 2014 in which they attempted to “assess the scope and magnitude of the problem,” identify recommendations, and “position SBE to support research exploring the causes and consequences of scientific behavior that enhance the likelihood of generating nonreplicable findings and replicable findings, and into research practices to improve the validity of research findings.” This report suggests that NSF SBE promote the identification of questionable research practices, encourage attempts to replicate, reproduce, and generalize findings, discover impetus for QRPs, and propose interventions to reduce frequency of QRPs. Access: https://www.nsf.gov/sbe/AC_Materials/SBE_Robust_and_Reliable_Research_Report.pdf.

NSF 16-083 Dear Colleague Letter: Reproducibility and Robustness of Results, May 18, 2016

This “Dear Colleague” letter from Roger M. Wakimoto, Assistant Director, Directorate for Geosciences, National Science Foundation reaffirms support for reliability, reproducibility, and robustness and notes that the directorate continues “to welcome proposals related to enhancing the validity of the data and outcomes of research in all GEO programs.” Specifically mentioned are community approaches, including comparisons of techniques, instruments, and models, assessment and development of best practices, implementation of data management policies, and investments in infrastructure to make data and metadata available. Access: https://www.nsf.gov/pubs/2016/nsf16083/nsf16083.pdf.

Reproducibility and Rigor in REE’s Portfolio of Research, United States Department of Agriculture (USDA), September 20, 2016

REE in the title of this document refers to Research, Education, and Economics, a unit in the Department of Agriculture. This document covers the USDA’s Science Advisory Council examination of reproducibility in agriculture and nutrition related disciplines. The Science Advisory Council report includes definitions of reproducibility, replicability, and generalizability. The main question addressed was, “What actions can and should USDA take to foster reproducibility and rigor in USDA-supported research?” They refer to agriculture and nutrition research rigor issues also being identified in biomedical and social science disciplines. Topics explored included data dredging, null results as related to publication bias, underpowered studies, suboptimal reporting of methods, and weak study design and execution. Specific to agriculture research, underpowered studies may be a result of crop production season durations. Regarding human research, this report addresses ethical issues that may make it difficult to reproduce research. The following solutions were proposed and summarized: open data, pre-registration, collaboration (e.g., cooperative data sharing agreements), computer automation to reduce errors, open methods, and USDA reporting guidelines. They advise that funding requests and post-award management should address rigor and reproducibility similar to processes already in place at NIH. This report cites several existing documents that provide guidance on rigor (e.g., USDA Scientific Integrity Policy Handbook). Access: https://www.usda.gov/sites/default/files/documents/nareeeab-reproducibility-rigor-report.pdf.

NSF 17-022 Dear Colleague Letter: Encouraging Reproducibility in Computing and Communications Research, October 21, 2016

In this “Dear Colleague” letter from Jim Kurose, assistant director, Computer and Information Science and Engineering (CISE), recognition of concerns about standards of reproducibility and rigor in research led to the directorates intention to “support research that improves the level of reproducibility in research on computer systems and networking; modeling, analysis and simulation of computing and communication systems; and cybersecurity.” It encourages primary investigators to create rigorous protocols and make these and data available to other researchers. Also encouraged are proposals that “specifically seek to reproduce, verify and/or characterize recent results.” Access: https://www.nsf.gov/pubs/2017/nsf17022/nsf17022.pdf.

NIH Initiative on Enhancing Research Reproducibility and Transparency, National Institutes of Health, Last Revised January 2017

This initiative description cites a 2014 Nature article that focuses on self-correcting preclinical research.6 The article was written by NIH director Francis S. Collins and principal deputy director Lawrence A. Tabak. Collins and Tabak emphasized NIH commitment to systematic changes and encouraged engagement from the biomedical-research community. The initiative webpage cites NIH notices (e.g., NOT-MH-14-004) reflecting efforts to improve the quality of National Institute on Drug Abuse (NIDA) funded research by enhancing the reporting of methods and results. Access: https://www.drugabuse.gov/offices/office-nida-director-od/office-translational-initiatives-program-innovations-otipi/nih-initiative-enhancing-research-reproducibility-transparency.

Annual Report of the Office of Economic Research FY 2016, Small Business Administration (SBA), January 2017

This annual report summarizes updates made to the Office of Economic Research’s (OER) Small Business State Profiles in which the state profiles were described as reproducible products. This report defined reproducible research and explained the value to having OER reproducible products. This document also lists OER hosted small business Economic Research Forums. One session was called “Reproducibility,” and the speaker was Dr. Richard Schwinn, a Research Economist from the SBA Office of Advocacy. This session included an explanation of tools and languages (e.g., R, LATEX) that could be used to support transparency and reproducibility. Access: https://www.sba.gov/sites/default/files/OER_Annual_Report_FY2016.pdf.

NSF 18-053 Dear Colleague Letter: Achieving New Insights through Replicability and Reproducibility, March 9, 2018

This letter from Dr. Fay Lomax Cook, Assistant Director, Directorate for Social, Behavioral, and Economic Sciences encourages the submission of proposals that replicate, reproduce, or generalize studies utilizing neuroimaging or neuroelectric data. The letter states that “proposals should: (1) make a strong case for the studies chosen for replication, reproduction or generalization; (2) substantiate the chosen analytic strategy; and (3) present a plan or template for evaluating, documenting, and communicating the lessons learned during the work.” The letter provides further guidance on the traits of successful proposals in light of reproducibility and replicability. Access: https://www.nsf.gov/pubs/2018/nsf18053/nsf18053.pdf.

Rigor and Reproducibility, National Institutes of Health (NIH), updated May 30, 2018

This website provides information to extramural researchers on NIH expectations for addressing rigor and transparency in NIH grant applications and progress reports. It includes detailed information on the key areas of rigor and reproducibility that are expected in applications and assessed by reviewers, expectations for formal instruction in experimental design and transparency, and other available resources. Access: https://grants.nih.gov/reproducibility/index.htm.

Systematizing Confidence in Open Research and Evidence Grants Notice, Defense Advanced Research Projects Agency, Department of Defense, June 12, 2018

This Defense Advanced Research Projects Agency (DARPA) grant notice is for a funding opportunity titled, “Systematizing Confidence in Open Research and Evidence (SCORE).” The Defense Sciences Office within DARPA is seeking proposals to automate assignment of Confidence Scores (CSs) to Social and Behavioral Science (SBS) research. They define CSs as “quantitative measures that should enable someone to understand the degree to which a particular claim or result is likely to be reproducible and/or replicable.” The desired tools are expected to assign CSs with reliability that is equal to or improves upon human methods. The stated desired outcome is that consumers of SBS research products would quickly be able to calibrate “the level of confidence in the Reproducibility and Replicability (R&R) of a given SBS result or claim.” Access: https://www.grants.gov/web/grants/view-opportunity.html?oppId=306186.

Science Advisory Board (SAB) Consideration of EPA Proposed Rule: Strengthening Transparency in Regulatory Science, Environmental Protection Agency, June 28, 2018

This document is a report written by the Environmental Protection Agency’s (EPA) Science Advisory Board (SAB), a public advisory group authorized to advise the agency on scientific matters and review EPA research programs and plans. SAB reports like this one are then hosted at epa.gov/sab among other places. In this document, the SAB is responding to a proposed rule, Strengthening Transparency in Regulatory Science (RIN 2080-AA14 and see entry below). The SAB expresses concern over the time allotted to respond to the rule and how SAB was made aware of the proposed rule. This document summarizes SAB discussions during a teleconference meeting where many topics were discussed including the trend for scientific disciplines to allow for public access of data and analytic methods. Sensitive situations were explained and included confidentiality and privacy of human subject data, cost, effort, and Institutional Review Board requirements. They advise that the EPA define sound science concepts including “replication” and “validation.” Access: https://yosemite.epa.gov/sab/sabproduct.nsf/LookupWebReportsLastMonthBOARD/4ECB44CA28936083852582BB004ADE54/$File/EPA-SAB-18-003+Unsigned.pdf.

Congressional Publications

Subcommittee on Research, Committee on Science, Space, and Technology, House Hearing “Scientific Integrity and Transparency,” March 5, 2013

Through this hearing, committee members attempt to understand, “the scope of the problem with scientific integrity, especially how thorough researchers deal with underlying data” and how openness fosters replicability and reproducibility in research. The editor-in-chief of Science magazine discusses the issue of data availability and the crisis of reproducibility; those from academia discuss data and code sharing so that replication can lead to validation; the last witness exhorts Congress, funding agencies, and journal editors to “require data analysis strategies that demonstrate reproducibility.” Access: http://purl.fdlp.gov/GPO/gpo37400.

Committee on Science, Space, and Technology, House Hearing “Strengthening Transparency and Accountability Within the Environmental Protection Agency,” November 14, 2013

This hearing reviews the EPA’s policies regarding the use of science and research in creating regulations and policies as well as calling for additional transparency and openness in the name of replicability. The administrator of the EPA, Gina McCarthy, is the sole witness with almost fifty pages of discussion and questions with the committee members and almost seventy pages of post-hearing responses to questions submitted by the committee after the hearing concluded. These post-hearing questions from committee members of both parties focused in on specific EPA programs and studies, EPA regulations and rulemaking, and the actions of the agency around particular topics or issues, including the EPA hydraulic fracturing study, air pollution and air quality regulations, and conflicts of interest related to grant funding and participation on advisory groups, to name a few of the many areas examined. Access: http://purl.fdlp.gov/GPO/gpo54932.

Subcommittee on Environment, Committee on Science, Space, and Technology, House Hearing “Ensuring Open Science at EPA,” February 11, 2014

This hearing examines options to improve transparency and reproducibility of regulatory science used by the EPA and to receive testimony on the Secret Science Reform Act of 2014 (HR 4012), legislation that limits the research the EPA can use for making regulations to research that is publicly available and therefore usable in the name of reproducibility. Testimony from witnesses discussed why access to data underpinning regulations is so important, with one witness also calling upon more transparency of data by industry since this deference to industry privacy is part of why the EPA cannot make data it uses for regulatory purposes public. In addition, one piece of testimony focused on the impact of regulations on small businesses and how making this data available will help weed out regulations that are not legitimately needed. Access: http://purl.fdlp.gov/GPO/gpo52192.

Committee on Science, Space, and Technology, House Report 113-619 “Secret Science Reform Act of 2014,” November 12, 2014

This report outlines H.R. 4012, a bill that would prohibit the EPA from “proposing, finalizing, or disseminating regulations or assessments based upon science that is not transparent or reproducible,” and gives analysis for each section of the bill, why legislation was needed, proceedings of the committee markup, and letters submitted from the public in support of the legislation, among other information related to the bill. It also includes the reported bill text. The bill, sponsored by Rep. Smith (TX), chairman of the Committee on Science, Space, and Technology, passed the House but died in committee in the Senate. Access: http://purl.fdlp.gov/GPO/gpo53304.

Committee on Science, Space, and Technology, House Report 114-34 “Secret Science Reform Act of 2015,” March 2, 2015

This report outlines H.R. 1030, that would prohibit the EPA from “proposing, finalizing, or disseminating regulations or assessments based upon science that is not transparent or reproducible” and includes committee statements and views, explanations of amendments, minority views, and other relevant information to the bill. The bill, sponsored by Rep. Smith (TX), chairman of the Committee on Science, Space, and Technology, passed the House and died in committee in the Senate. See bill text as referred in the Senate at https://www.congress.gov/114/bills/hr1030/BILLS-114hr1030rfs.pdf. Access: http://purl.fdlp.gov/GPO/gpo55749.

Committee on Science, Space, and Technology, House Report 114-107, pt. 1, “America COMPETES Reauthorization Act of 2015,” May 8, 2015

This report outlines H.R. 1806 with amendments as well as majority and minority views of the bill. The bill is sponsored by Rep. Smith (TX), chairman of the Committee on Science, Space, and Technology. The first section of the bill seeks to revise NSF programs and includes a section about how NSF should police research misconduct and conduct a report “to assess research and data reproducibility and replicability issues in interdisciplinary research and to make recommendations on how to improve rigor and transparency in scientific research.” The bill passed the House but died in committee in the Senate. Access: http://purl.fdlp.gov/GPO/gpo57384.

Committee on Environment and Public Works, Senate Report 114-69 “Secret Science Reform Act of 2015,” June 22, 2015

This report outlines S. 544, that would prohibit the EPA from, “proposing, finalizing, or disseminating regulations or assessments based upon science that is not transparent or reproducible” and includes committee majority and minority statements and views, explanations of amendments, section-by-section analysis, costs of these changes in legislation, and other relevant information to the bill. The bill is sponsored by Sen. Barrasso (WY), chairman of the Committee on Environment and Public Works. The bill was reported favorably by the Committee but never made it to the Senate floor. See bill text as reported in the Senate at https://www.congress.gov/114/bills/s544/BILLS-114s544rs.pdf. Access: http://purl.fdlp.gov/GPO/gpo58645.

Committee on Science, Space, and Technology, House Hearing “Making EPA Great Again,” February 7, 2017

This hearing examines how the EPA evaluates and uses science in making policies and regulations. The hearing was a precursor to the HONEST Act (see below) introduced in the House by Rep. Smith (TX), the chairman of the committee for this hearing. Testimonies and statements offer possible steps for EPA reform (reforming Science Advisory Board, making all data public that it uses to make decisions, peer-review process for recommendations and regulations, etc.), but also an urging for Congress to not try to regulate science and the scientific process through which the EPA functions and relies. It also discusses the EPA’s over-exaggeration of risk related to public health and environmental hazards and the related lack of clarity within EPA procedures when determining hazards and risk. Access: http://purl.fdlp.gov/GPO/gpo80315.

Subcommittee on Research and Technology, Committee on Science, Space and Technology, House Hearing “National Science Foundation Part II: Future Opportunities and Challenges for Science,” March 21, 2017

This hearing explores research challenges and future opportunities related to the NSF and what the Foundation funds, including the issue of using taxpayer funds to encourage reproducible open science. Oral and written statements from the witnesses shed light on the complexity of data sharing and open science, what the NSF already does to support and promote “good science,” and some areas and initiatives to focus on for continuing federal monetary investment. Access: https://www.gpo.gov/fdsys/pkg/CHRG-115hhrg24672/pdf/CHRG-115hhrg24672.pdf.

Committee on Science, Space, and Technology, House Report 115-59 “HONEST and Open New EPA Science Treatment Act of 2017,” March 24, 2017

This report outlines H.R. 1430, that would prohibit the EPA from “proposing, finalizing, or disseminating regulations or assessments based upon science that is not transparent or reproducible” and includes committee statements and views, explanations of amendments, cost estimates, and other relevant information to the bill. This legislation is sponsored by Rep. Smith (TX), chairman of the Committee on Science, Space, and Technology. Note that it includes identical language to the Secret Science Reform Acts of 2014 and 2015. The bill passed the House and has been sitting with the Committee on Environment and Public Works since March 30, 2017. See bill text as referred in the Senate at https://www.congress.gov/115/bills/hr1430/BILLS-115hr1430rfs.pdf. Access: http://purl.fdlp.gov/GPO/gpo79049.

“HONEST and Open New EPA Science Treatment Act of 2017,” H.R. 1430. 115th Congress, 1st session, Congressional Record p. H2536-2547, March 29, 2017

A discussion of H.R. 1430 (HONEST Act, see above), which includes spirited debate for and against the bill covering various topics including costs associated with the bill, data privacy and sensitive data, economic effects of the bill, and the effect on the EPA’s practice and ability, among other important points. Also included are letters of support and opposition from various external groups and a motion to recommit the bill back to committee with an amendment that would allow the EPA administrator to use “the best available science, whether or not it is publicly available in any form, when responding to threats to public health,” which ultimately failed. Access: https://www.congress.gov/crec/2017/03/29/CREC-2017-03-29-pt1-PgH2536.pdf.

HONEST Act and Open New EPA Science Treatment Act of 2017, 115 S. 1794. September 12, 2017

Legislation introduced and sponsored by Sen. Rounds, majority member of the Committee on Environment and Public Works. The bill is sitting with the Committee on Environment and Public Works. It contains almost identical language to H.R. 1430 (see above). Access: https://www.congress.gov/115/bills/s1794/BILLS-115s1794is.pdf.

Subcommittee on Federal Spending Oversight and Emergency Management, Committee on Homeland Security and Governmental Affairs, Senate Hearing “Broken Beakers: Federal Support for Research,” October 18, 2017

This hearing examines the federal government’s role in funding research and discusses the worth and value of public investment in light of the perceived lack of transparency and/or reproducibility in funded research. Testimony and statements highlight a need for Congressional oversight in ensuring federal agencies make taxpayer funded research openly accessible to the public and a debate on the government’s involvement in funding scientific endeavors. One witness claims that it does little to advance economic growth, while another outlines how federal research funding positively influences innovation and fills a funding niche that industry does not. Access: https://purl.fdlp.gov/GPO/gpo91334.

Rules, Regulations, and Executive Orders and Documents

Executive Order 13777 of February 24, 2017, Enforcing the Regulatory Reform Agenda,” 82 Federal Register 12285

President Donald Trump’s Executive Order demands “lower regulatory burdens on the American people by implementing and enforcing regulatory reform.” Section 3 directs agencies to establish a Regulatory Reform Task Force that will evaluate existing regulations with a focus on “those regulations that rely in whole or in part on data, information, or methods that are not publicly available or that are insufficiently transparent to meet the standard for reproducibility.” Access: https://www.gpo.gov/fdsys/pkg/FR-2017-03-01/pdf/2017-04107.pdf.

Executive Order 13783 of March 28, 2017, Promoting Energy Independence and Economic Growth,” 82 Federal Register 16093

President Donald Trump’s Executive Order outlines his desire to avoid “regulatory burdens that unnecessarily encumber energy production, constrain economic growth, and prevent job creation.” He also emphasizes in Section 1e the development of environmental regulations “through transparent processes that employ the best available peer-reviewed science and economics,” which hearkens to some of the bills introduced in Congress around the idea of using open data in creating EPA regulations. Access: https://www.gpo.gov/fdsys/pkg/FR-2017-03-31/pdf/2017-06576.pdf.

Memorandum for the Heads of Executive Departments and Agencies, Executive Office of the President, August 17, 2017

This memorandum from Mick Mulvaney, the director of the Office of Management and Budget, as well as Michael Kratsios, the Deputy Assistant to the President in the Office of Science and Technology Policy, provides fiscal year 2019 research and development budget priorities. This memo includes Research and Development (R&D) Priority Practices, and one of the listed priority practices is “Increasing Government Accountability and Efficiency.” In the explanation for this priority practice, the text references proposed programs based on sound science. Further “quantitative metrics to evaluate R&D outcomes should be developed and utilized for all Federal R&D programs.” Access: https://www.whitehouse.gov/sites/whitehouse.gov/files/ostp/fy2019-administration-research-development-budget-priorities.pdf.

Strengthening Transparency in Regulatory Science, 83 Federal Register 18768 (proposed April 30, 2018) (to be codified at 40 C.F.R. pt. 30)

This EPA proposed regulation would require the “EPA [to] ensure that the data underlying [regulations] are publicly available in a manner sufficient for independent validation” including “regulations for which the public is likely to bear the cost of compliance.” This regulation would particularly focus on “pivotal regulatory science,” which is essentially the research and data that plays the largest role in creating significant regulations. Access: https://www.regulations.gov/docket?D=EPA-HQ-OA-2018-0259.

References

1. Michael Schulson, “Science’s ‘Reproducibility Crisis’ is Being Used as Political Ammunition,” Wired (April 20, 2018), https://www.wired.com/story/sciences-reproducibility-crisis-is-being-used-as-political-ammunition/; Stephan Lewandowsky and Dorothy Bishop, “Don’t Let Transparency Damage Science,” Nature 529 (January 28, 2016): 459–61, https://doi.org/10.1038/529459a; Joseph Hilgard and Kathleen Hall Jamieson, “Science as ‘Broken’ Versus Science as ‘Self-Correcting’: How Retractions and Peer-Review Problems Are Exploited to Attack Science,” in The Oxford Handbook of the Science of Science Communication, ed. Kathleen Hall Jamieson, Dan M. Kahan, and Dietram A. Scheufele (Oxord: Oxford University Press, 2017), https://doi.org/10.1093/oxfordhb/9780190497620.013.9; Jeff Tollefson, “US EPA Science Advisers Question ‘Secret Science’ Rule on Data Transparency,” Nature 558, no. 15 (2018), https://doi.org/10.1038/d41586-018-05319-2; Juliet Eilperin and Brady Dennis, “Pruitt Unveils Controversial ‘Transparency’ Rule Limiting What Research EPA Can Use,” Washington Post, April 24, 2018, https://www.washingtonpost.com/news/energy-environment/wp/2018/04/24/pruitt-to-unveil-controversial-transparency-rule-limiting-what-research-epa-can-use/?utm_term=.0a80453d507d.
2. Open Science Collaboration, “Estimating the Reproducibility of Psychological Science,” Science 349, no. 6251 (2015): aac4716-1-aac4716-8, https://doi.org/10.1126/science.aac4716; Center for Open Science and Science Exchange, “Reproducibility Project: Cancer Biology,” eLife (December 10, 2014), https://elifesciences.org/collections/9b1e83d1/reproducibility-project-cancer-biology.
3. Franklin Sayre and Amy Riegelman, “The Reproducibility Crisis and Academic Libraries,” College & Research Libraries 79, no. 1 (2018): 2, https://doi.org/10.5860/crl.79.1.2; Franklin Sayre and Amy Riegelman, “Replicable Services for Reproducible Research: A Model for Academic Libraries,” College & Research Libraries (2019 preprint), https://crl.acrl.org/index.php/crl/article/view/16993.
4. Kenneth Bollen et al., “Social, Behavioral, and Economic Sciences Perspectives on Robust and Reliable Science,” Report of the Subcommittee on Replicability in Science of the Advisory Committee to the National Science Foundation Directorate for Social, Behavioral, and Economic Sciences (2015), https://www.nsf.gov/sbe/AC_Materials/SBE_Robust_and_Reliable_Research_Report.pdf. Emphasis added.
5. Christie Aschwanden, “There’s No Such Thing as ‘Sound Science,’” FiveThirtyEight, December 6, 2017, https://fivethirtyeight.com/features/the-easiest-way-to-dismiss-good-science-demand-sound-science/.
6. Francis S. Collins and Lawrence A. Tabak, “NIH Plans to Enhance Reproducibility,” Nature 505, no. 7485 (2014): 612–13, https://doi.org/10.1038/505612a.