14  Version control

Version control refers to the record of changes made in a file or set of files over time. The record should allow collaborators to track history, to review and to revert changes. Versioning refers to the management of changes in a file or project (read more in https://book.the-turing-way.org/reproducible-research/vcs).

Versioning is very important for scientific reproducibility and transparency as it allows to follow provenance information, i.e., the record of how the results are derived from data. Data and code evolve and change over time and it can become really difficult to track which version of them generated a figure or a results table in a publication.

We recommend adopting Git, introduced below, to apply version control in documents, data and code. We suggest beginning with simple workflows; there is no need to master all of Git’s advanced features right away.

14.1 Version control in documents

There are many tools widely used tools for version control and collaborative editing of text documents. For example, MS Word, Google Docs and Overleaf. These tools have some limitations that need to be considered:

  • They require a master document (a single source document where we edit changes)

  • It does not scale up well to large number of collaborators

  • These tools do not show the complete history of changes

  • The changes and master document are stored in a company’s cloud

  • They only work for specific types of documents

In view of these limitations, these tools cannot provide a reliable and manageable version control system for many research outputs like metadata, documentation, tables and specially code.

14.2 Git version control system for data science

Git is a version control system that tracks changes in files, including timestamps. It is the de facto standard in code development and it is increasingly used in research and data science in general. It is open source, developed by Linus Torvalds in 2005. These are some characteristics of the Git system:

  • It is known to be fast, efficient, and reliable

  • Runs on all major operating systems

  • it is integrated in most IDEs (Integrated Development Environments): RStudio, Visual Studio Code, Eclipse, PyCharm, Spyder, MATLAB®…

Git is a distributed version control system (DVCS), which means collaborators create local copies of a project, make changes locally and then commit those changes to a remote repository (see Figure 14.1)

Figure 14.1: Centralized vs distributed version control systems

Main concepts in Git

  • Repository: A collection of files, typically organized as a project, managed with version control.

  • Commit: The action of recording changes in a repository. It is like a snapshot of the entire repository at a given time.

  • History: A registry (“log”) and collection of all the snapshots (“commits”) of a repository, allowing us to revert changes.

Platforms to host Git repositories online

  • GitLab: An open-source web-based tool that provides a Git repository manager (www.gitlab.com).
    Many institutions and organizations have their own GitLab instance (e.g., UZH https://gitlab.uzh.ch/).

  • GitHub: Another provider, owned by Microsoft (potential data governance issues) (www.github.com).

An overview of the Git system

The main areas and commands in Git are summarized in Figure 14.2.

Git ‘areas’:

  • Remote repository: the online repository with all files and history of changes, hosted in platforms like Gitlab or Github.

  • Workspace: this is the “visible” folder. In most cases this is a local folder with a copy of the online repository (including history of changes).

  • Index : (not directly visible) is a stagging area used to prepare changes before committing them to the project history

  • Local repository: (usually a hidden folder .git ) is a copy of the entire project’s history and codebase

Main actions:

  • Pull: Pull changes from the remote to update the local copy.

  • Commit & Push: Commit and push local changes into the remote.

  • Revert: Revert changes to a previous commit.

    Figure 14.2: Overview of Git commands

Workflows

Git can be used in the context of complex application development and in smaller settings. Although there may be a steep learning curve at the beginning, scientists can gradually incorporate Git into their routine workflow and it can be kept relatively simple without needing to use all the advanced features.

  • Branches: a repository can have multiple ‘branches’, which allows working in parallel. The same user or multiple collaborators can work on parallel versions of the repository. Some of the changes from the branches (e.g., new features into the main code) can be eventually merged into the ‘main’ code branch.
Figure 14.3: Illustration of git branches from the Turing Way

Publishing and sharing

  • Releases. The content of a Git repository is changing frequently. When scientists publish the code associated with a publication, the version of the repository with the files involved in that published analysis need to be also shared. That can be done by publishing a gitlab release, which is a snapshot of a repository with a version tag.
  • Persistent identifiers. Sharing the URL to a Gitlab or Github repository associated with a publication is not recommended. The URL may change in time and the code version become unavailable. It is recommended to publish a Git repository release in a platform that allows to assign a persistent identifier. For example, a digital object identifier or DOI, so that the repository can become a citable resource to be found in the future. Platforms like Zenodo, for example, offer direct integration with Github, assigning a DOI to every release of the repository. Another way is to simply upload in the platform a .zip folder with all the files in the repository (the ‘snapshot’ created when making a release) with a version tag.

Best practices

Setting up a repository

  • “Documentation is a love letter that you write to your future self.”

  • Add a README file in the main folder (root folder) and in other folders if needed.

  • Use a naming convention with a balance between concise and informative names. Avoid too cryptic or ambiguous folder or file names. For example: run_model_AB.r instead of run_scripts.r

Code writing

  • Consider numbering scripts that need to be run sequentially.

  • Use relative paths.

  • Log software libraries and system dependencies, e.g., sessionInfo(), and hardware when applicable.

  • Scripts should include comments (but concise).

  • Consider scripts to generate reports (dynamic reporting).

Sharing and Publishing

  • Describe the project in the main README file (dates, project ID, funding, affiliations, and DOI).

  • When the repository contains the code associated with a publication, include the DOI of the publication in the main README file.

  • Create a release, take a snapshot with a version tag .

  • Assign a DOI to the release.

  • Include a license file to explain reusability (agree with PI, e.g., GPL, MIT).

  • Specify the location of the source data.

14.3 Selected resources