2 Reproducibility
One goal of good research data management is to improve the reproducibility of scientific findings, which is crucial for the trustworthiness of empirical research. This code of conduct discusses important steps towards improving reproducibility, such as adopting standards for organising data, providing documentation and metadata, and using interoperable file formats, controlled vocabularies, and version control. Throughout, the code will refer to different aspects of reproducibility, which we introduce in this chapter. Irrespective of exact definitions, we recommend researchers consider scientific reproducibility to be a continuum that benefits from any action and effort that improves transparency and reusability of their research outputs.
“Science is widely regarded as the most reliable way of turning observations into facts. But none of our scientific facts should be based on a single study, or one set of results. Whenever a study claims to have made a new discovery, it is the job of science to ask how we can verify and confirm these results.” (Held and Schwab 2020).
2.1 Terminology
(2019) Definitions by the US National Academy of Sciences
The National Academy of Sciences discussed the following concepts in their 2019 book on scientific reproducibility and replicability (National Academies of Sciences, Engineering and Medicine 2019):
Reproducibility or computational reproducibility: obtaining consistent computational results using the same input data, computational steps, methods, code, and conditions of analysis;
Replicability: obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data.
Generalizability: refers to the extent that results of a study apply in other contexts or populations that differ from the original one.
A single scientific study may entail one or more of these concepts.
(2022) Definitions by the Turing Way
The Turing way book (The Turing Way Community 2022) added the concept of ‘robustness’ and systematized the different aspects of reproducibility in the matrix shown in Figure 2.1.
(2023) Definitions by the iRISE project
The EU-funded project iRISE - improving Reproducibility In SciencE - Open Knowledge Base, in its glossary, highlights another aspect of reproducibility, which links the concept to open science practices: ‘the extent to which design, implementation, analysis, and reporting of a study enable a third party to repeat the study and assess its findings’. We recommend the iRISE glossary as a resource for those interested in recent in-depth discussion of the terminology around reproducibility. In this code of conduct we will use the earlier terminology from the Turing Way as it is standard and established in various areas of empirical research.