As we explained, the programming language used in an analysis has a clear influence on the reproducibility of your analysis. It is not a characteristic of the language itself but rather a consequence of the development philosophy of the underlying community. For example C is a very stable language with a very clear specification designed by a committee (even though some compilers may not respect this norm).

On the other end of the spectrum, Python had a much more organic development based on a readability philosophy and valuing continuous improvement over backwards-compatibility. Furthermore, Python is commonly used as a wrapping language (e.g., to easily use C or FORTRAN libraries) and has its own packaging system. All these design choices tend to make reproducibility often a bit painful with Python, even though the community is slowly taking this into account. The transition from Python 2 to the not fully backwards compatible Python 3 has been a particularly painful process, not least because the two languages are so similar that is it not always easy to figure out if a given script or module is written in Python 2 or Python 3. It isn't even rare to see Python scripts that work under both Python 2 and Python 3, but produce different results due to the change in the behavior of integer division.

R, in comparison is much closer (in terms of developer community) to languages like SAS, which is heavily used in the pharmaceutical industry where statistical procedures need to be standardized and rock solid/stable. R is obviously not immune to evolutions that break old versions and hinder reproducibility/backward compatibility. Here is a relatively recent true story about this and some colleagues who worked on the statistics introductory course with R on FUN reported us several issues with a few functions (plotmeans from gplots, survfit from survival, or hclust) whose default parameters had changed over the years. It is thus probably good practice to give explicit values for all parameters (which can be cumbersome) instead of relying on default values, and to restrict your dependencies as much as possible.

This being said, the R development community is generally quite careful about stability. We (the authors of this MOOC) believe that open source (which allows to inspect how computation is done and to identify both mistakes and sources of non-reproducibility) is more important than the rock solid stability of SAS, which is proprietary software. Yet, if you really need to stay with SAS (similar solutions probably exist for other languages as well), you should know that SAS can be used within Jupyter using either the Python SASKernel or the Python SASPy package (step by step explanations about this are given here). Using such literate programming approach allied with systematic version and environment control will always help.

As we mentioned in the video sequences, there are several solutions to control your environment:

• The easy (preserve the mess) ones: CDE or ReproZip
• The more demanding (encourage cleanliness) where you start with a clean environment and install only what's strictly necessary (and document it):
  • The very well known Docker
  • Singularity or Spack, which are more targeted toward the specific needs of high performance computing users
  • Guix, Nix that are very clean (perfect?) solutions to this dependency hell and which we recommend

It may be hard to understand the difference between these different approaches and decide which one is better in your context.

Here is a webinar where some of these tools are demoed in a reproducible research context: Controling your environment (by Michael Mercier and Cristian Ruiz)

You may also want to have a look at the Popper conventions (webinar by Ivo Gimenez through google hangout) or at the presentation of Konrad Hinsen on Active Papers (http://www.activepapers.org/).

Ensuring software is properly archived, i.e, is safely stored so that it can be accessed in a perennial way, can be quite tricky. If you have never seen Roberto Di Cosmo presenting the Software Heritage project, this is a must see. https://www.softwareheritage.org/

For regular data, we highly recommend using https://www.zenodo.org/ whenever the data is not sensitive.

In the video sequences, we mentioned workflow managers (original application domain in parenthesis):

• Galaxy (genomics), Kepler (ecology), Taverna (bio-informatics), Pegasus (astronomy), Collective Knowledge (compiling optimization) , VisTrails (image processing)
• Light-weight: dask (python), drake (R), swift (molecular biology), snakemake (like make but more expressive and in python) …
• Hybrids: SOS-notebook, …

You may want to have a look at this webinar: Reproducible Science in Bio-informatics: Current Status, Solutions and Research Opportunities (by Sarah Cohen Boulakia, Yvan Le Bras and Jérôme Chopard).

We have mentioned these topics in our MOOC but we could by no way cover them properly. We only suggest here a few interesting talks about this.

• In this talk, Pierre Dragicevic provides a nice illustration of the consequences of statistical uncertainty and of how some concepts (e.G. p-values) are commonly badly understood.
• Nathalie Revol, Philippe Langlois and Stef Graillat present the main challenges encountered when trying to achieve numerical reproducibility and present recent research work on this topic.

You may want to have a look at the following two webinars:

• Enabling open and reproducible research at computer systems’ conferences (by Grigori Fursin). In particular, this talk discusses artifact evaluation that is becoming more and more popular.
• Publication Modes Favoring Reproducible Research (by Konrad Hinsen and Nicolas Rougier). In this talk, the motivation for the ReScience journal initiative are presented.
• Simine Vazire - When Should We be Skeptical of Scientific Claims?, which is discussing publication practices in social sciences and in particular HARKing (Hypothesizing After the Results are Known), p-hacking, etc.

Experimentation was not covered in this MOOC, although it is an essential part of science. The main reason is that practices and constraints can vary so wildly from one domain to another that it could not be properly covered in a first edition. We would be happy to gather references you consider as interesting in your domain so do not hesitate to provide us with such references by using the forum and we will update this page.

• A recent talk by Lucas Nussbaum on Experimental Testbeds in Computer Science.

When taking notes, it may be difficult to remember which version of the code or of a file was used. This is what version control is useful for. Here are a few useful commands that we typically insert at the top of our notebooks in shell cells

git log -1

commit 741b0088af5b40588493c23c46d6bab5d0adeb33
Author: Arnaud Legrand <arnaud.legrand@imag.fr>
Date:   Tue Sep 4 12:45:43 2018 +0200

    Fix a few typos and provide information on jupyter-git plugins.

git status -u

On branch master
Your branch is ahead of 'origin/master' by 4 commits.
  (use "git push" to publish your local commits)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   resources.org

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	../../module2/ressources/replicable_article/IEEEtran.bst
	../../module2/ressources/replicable_article/IEEEtran.cls
	../../module2/ressources/replicable_article/article.bbl
	../../module2/ressources/replicable_article/article.tex
	../../module2/ressources/replicable_article/data.csv
	../../module2/ressources/replicable_article/figure.pdf
	../../module2/ressources/replicable_article/logo.png
	.#resources.org

no changes added to commit (use "git add" and/or "git commit -a")

Note: the -u indicates that git should also display the contents of new directories it did not previously know about.

Then, we often include commands at the end of our notebook indicating how to commit the results (adding the new files, committing with a clear message and pushing). E.g.,

git add resources.org;
git commit -m "Completing the section on getting Git information"
git push

[master 514fe2c1 ] Completing the section on getting Git information
 1 file changed, 61 insertions(+)
Counting objects: 25, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (20/20), done.
Writing objects: 100% (25/25), 7.31 KiB | 499.00 KiB/s, done.
Total 25 (delta 11), reused 0 (delta 0)
To ssh://app-learninglab.inria.fr:9418/learning-lab/mooc-rr-ressources.git
   6359f8c..1f8a567  master -> master

Obviously, in this case you need to save the notebook before running this cell, hence the output of this final command (with the new git hash) will not be stored in the cell. This is not really a problem and is the price to pay for running git from within the notebook itself.