diff --git a/module2/ressources/emacs_orgmode.html b/module2/ressources/emacs_orgmode.html index d3410313554579113b2a863a1fe64809aed4b16e..8d7ce735f38c6545f3cf40299380273bc37316ac 100644 --- a/module2/ressources/emacs_orgmode.html +++ b/module2/ressources/emacs_orgmode.html @@ -4,46 +4,47 @@

Table of Contents

@@ -64,15 +65,15 @@ the -

Installing emacs, org-mode, ess, and auctex.

-
+
+

Installing emacs, org-mode, ess, and auctex.

+
-
-

Linux (Debian, Ubuntu)

-
+
+

Linux (Debian, Ubuntu)

+

-We provide here only instructions for debian-based distributions. Feel +We provide here only instructions for Debian-based distributions. Feel free to contribute to this document to provide up-to-date information for other distributions (e.g.n redhat, fedora).

@@ -132,9 +133,9 @@ which is now deprecated.

-
-

macOS

-
+
+

macOS

+

Note: macOS comes with a prehistoric command-line-only version of Emacs located at /usr/bin/emacs. It's best to forget about it.

@@ -189,9 +190,9 @@ in order to make Emacs accessible directly from /Applications.
-
-

Windows

-
+
+

Windows

+

Install the .exe file from Vincent Goulet: https://vigou3.github.io/emacs-modified-windows/. It ships with recent @@ -202,21 +203,13 @@ versions:

  • Org-mode 9.1.13
  • ESS 17.11
  • -

    -To have code pretty printing when exporting to html, it appears like -you should install the htmlize package, which is done by opening emacs -and typing the following command: -

    -
    -M-x package-install RET htmlize RET    # where M-x means pressing the "Esc" key then the "x" key
    -
    -
    -

    Directory naming conventions

    -
    +
    +

    Directory naming conventions

    +

    -In all the following instructions, it is common to refer to your home +In the following instructions, we refer to your home directory through the (UNIX) ~/ notation. On Windows, your home directory should be something like C:\Users\yourname. Therefore, whenever we mention the ~/org/ (resp. the ~/.emacs.d/) directory this @@ -225,17 +218,16 @@ means we are referring to C:\Users\yourname\org (resp.

    -
    -

    Making R and Python available to the console

    -
    +
    +

    Making R and Python available to the console

    +

    -When running a command, windows will look for the command in the +When running a command, Windows will look for the command in the directories indicated in the PATH environment variable. If none of these directories contains the command, Windows will stop and indicate -the command does not exist. To make sure R (, which may be in -something like C:/Program Files/R/R-3.5.1/bin/x64/) and Python (, -which may be in something like C:/Program Files/Python/Python37/) can -easily be run from emacs, you should thus configure the PATH variable +the command does not exist. To make sure R (which may be in +something like C:/Program Files/R/R-3.5.1/bin/x64/) and Python (which may be in something like C:/Program Files/Python/Python37/) can +easily be run from Emacs, you should thus configure the PATH variable accordingly.

    @@ -245,9 +237,9 @@ explained -

    Installing and configuring Matplotlib (graphic python library)

    -
    +
    +

    Installing and configuring Matplotlib (graphic python library)

    +

    Open an DOS console and type the following command:

    @@ -282,11 +274,25 @@ starting with backend, which amounts to use the default Agg
    + +
    +

    All platforms: pretty code in HTML export

    +
    +

    +To have code pretty printing when exporting to HTML, you should +install the htmlize package, which is done by opening emacs and +typing the following command: +

    +
    +M-x package-install RET htmlize RET    # where M-x means pressing the "Esc" key then the "x" key
    +
    +
    +
    -
    -

    A simple "reproducible research" emacs configuration

    -
    +
    +

    A simple "reproducible research" emacs configuration

    +

    This section is illustrated in a video tutorial ("Mise en place Emacs/Orgmode" in French). Watching it before following the @@ -305,14 +311,14 @@ adding a few org-mode specific configurations.

    -
    -

    Step 0: Backup and download our configuration

    -
    +
    +

    Step 0: Backup and download our configuration

    +

    -The procedure we propose will wipe your already existing custom emacs -configuration if you already have one. You should thus beforehand make -a backup of ~/.emacs and of ~/.emacs.d/init.el (if these file -exists). +The procedure we propose will wipe your already existing custom Emacs +configuration if you have one. You should thus beforehand make +a backup of ~/.emacs and of ~/.emacs.d/init.el (if these files +exist).

    @@ -328,9 +334,9 @@ rr_org/journal.org

    -
    -

    Step 1: Prepare your journal

    -
    +
    +

    Step 1: Prepare your journal

    +

    Create an org/ directory in the top of your home:

    @@ -342,14 +348,14 @@ Create an org/ directory in the top of your home: Then copy rr_org/journal.org file in your ~/org/ directory. This file will be your laboratory notebook and all the notes you will capture with C-c c will go automatically go in this file. The first -entry of this notebook is populated with many emacs shortcuts that you +entry of this notebook is populated with many Emacs shortcuts that you should give a try.

    -
    -

    Step 2: Set up Emacs configuration

    -
    +
    +

    Step 2: Set up Emacs configuration

    +

    Copy rr_org/init.el in your ~/.emacs.d/ directory.

    @@ -361,9 +367,9 @@ configuration with the following command: emacs -q -l rr_org/init.el
    -
    -

    Step 3: Adapt the configuration to your specific needs if required

    -
    +
    +

    Step 3: Adapt the configuration to your specific needs if required

    +

    There are two situations in which it might be necessary to modify init.el: @@ -392,9 +398,9 @@ beginning of the file for instructions.

    -
    -

    Step 4: Check whether the installation is working or not

    -
    +
    +

    Step 4: Check whether the installation is working or not

    +

    Open a new instance of Emacs and open a foo.org file. Copy the following lines in this file: @@ -425,22 +431,22 @@ is demonstrated in the -

    Step 5: Open and play with your journal:

    -
    +
    +

    Step 5: Open and play with your journal:

    +

    In step 1, you were told to create an journal in ~org/journal.org. First you probably want to make sure this file is -backed up in a revision control system like git. We leave it up to you +stored in a version control system like git. We leave it up to you to set this up but if you have any trouble, feel free to ask on the FUN forums.

    -
    -

    A stub of replicable article

    -
    +
    +

    A stub of a replicable article

    +

    This section is illustrated in a video tutorial ("Écrire un article réplicable avec Emacs/Orgmode" in French). Watching it before @@ -473,7 +479,7 @@ tar zxf replicable_article.tgz; cd repli

    • If the make command fails (especially on Mac), it may be because -emacs or something else is not correctly installed. In that case, +Emacs or something else is not correctly installed. In that case, open the article directly with the following command:

      @@ -504,33 +510,33 @@ chunk of code when exporting.
    -
    -

    Emacs Tips and tricks

    -
    +
    +

    Emacs tips and tricks

    +
    -
    -

    Cheat-sheets

    -
    +
    +

    Cheat-sheets

    +

    -Learning emacs and org-mode can be difficult as there is an inordinate +Learning Emacs and Org-Mode can be difficult as there is an inordinate amount of shortcuts. Many people have thus come up with cheat-sheats. Here is a selection in case it helps:

    -
    -

    Emacs

    -
    + -
    -

    Org-mode

    -
    +
    +

    Org-mode

    +
    -
    -

    Video tutorials

    -
    +
    +

    Video tutorials

    +

    For those of you who prefer video explanations, here is a Youtube channel with many step by step emacs tutorials.

    -
    -

    Additional useful emacs packages

    -
    +
    +

    Additional useful emacs packages

    +
    -
    -

    Company-mode

    -
    +
    +

    Company-mode

    +

    Company-mode is a text completion framework for Emacs. It allows to have smart completion in emacs for the most common languages. If you @@ -563,18 +569,18 @@ official Web page: http://company-mode.

    -
    -

    Magit

    -
    +
    +

    Magit

    +

    -Magit is an emacs interface for Git. Its usage is briefly illustrated +Magit is an Emacs interface for Git. Its usage is briefly illustrated in the context of this MOOC in a video tutorial ("Utilisation Emacs/git" in French).

    It is very powerful and we use it on a daily basis but you should -definitely understand what git does behind the scene beforehand. If +definitely understand what git does behind the scenes beforehand. If you feel this would be useful for you, you should follow this visual walk-through or this really short "crash course". If you installed the previous "reproducible research" emacs configuration, you can easily @@ -583,9 +589,9 @@ invoke magit by using C-x g.

    -
    -

    Other resources

    -
    +
    +

    Other resources

    +

    -This document is particularly important if you follow the Rstudio or -the Org-Mode course. If you follow the Jupyter course, it can be -ignored in a first step as we have closely integrated Jupyter and +This document is particularly important if you follow the RStudio or +the Org-Mode path. If you follow the Jupyter path, it can be +ignored at first as we have closely integrated Jupyter and GitLab in the context of this MOOC.

    @@ -61,17 +61,17 @@ Please read all these instructions carefully, in particular the one on "Configuring your password on GitLab".

    -
    -

    Installing Git

    -
    +
    +

    Installing Git

    +
    -
    -

    Linux (debian, ubuntu)

    -
    +
    +

    Linux (Debian, Ubuntu)

    +

    -We provide here only instructions for debian-based distributions. Feel +We provide here only instructions for Debian-based distributions. Feel free to contribute to this document to provide up-to-date information -for other distributions (e.g., redhat, fedora). +for other distributions (e.g., RedHat, Fedora).

    @@ -83,13 +83,13 @@ Run (as root):

    -
    -

    Mac OSX and Windows

    -
    +
    +

    Mac OSX and Windows

    +
    • Download and install Git from the Git website.
    • Optional Git clients (should not be needed if you work within -Rstudio): +RStudio):
    -
    -

    Configuring Git

    -
    +
    +

    Configuring Git

    +
    -
    -

    Telling Git who you are: Name and Email

    -
    +
    +

    Telling Git who you are: Name and Email

    +
      -
    1. Open Terminal.
    2. +
    3. Open terminal.
    4. Set a Git username and email:

      @@ -140,9 +140,9 @@ email@example.com
    -
    -

    Dealing with proxies

    -
    +
    +

    Dealing with proxies

    +

    You may be behind a proxy, in which case you may have trouble cloning or fetching from a remote repository or you may get an error like @@ -164,11 +164,14 @@ will be prompted for it every time it is needed.

    -
    -

    Getting your default password on GitLab (and possibly changing it)

    -
    +
    +

    Getting your default password on GitLab (and possibly changing it)

    +

    -Warning (Jupyter users) : changing your default Gitlab password will prevent you from committing in Jupyter. You will have to do the extra step of changing your Jupyter ~/.git-credentials +Warning (Jupyter users) : changing your default Gitlab password will +prevent you from committing in Jupyter. You will have to do the extra +step of changing your ~/.git-credentials in the Jupyter environment +(possibly several times).

      @@ -224,13 +227,13 @@ button).
  • -If you wish to modify your password, you shoul go to Account > Settings > Password +If you wish to modify your password, you should go to Account > Settings > Password and define your password using the default password you just retrieved. Again, if you use the Jupyter notebooks we have deployed for the MOOC, remember that changing your default Gitlab password will prevent you from committing in Jupyter. You will have to do the extra step of changing your -Jupyter ~/.git-credentials through a jupyter console (see next +Jupyter ~/.git-credentials through a Jupyter console (see next section).

    @@ -242,9 +245,9 @@ section).
  • -
    -

    Remembering your password locally

    -
    +
    +

    Remembering your password locally

    +

    If you clone your repository by simply pasting the GitLab URL, you will be prompted for your login and your password every time you want to @@ -278,9 +281,9 @@ Use it at your own risk.

    -
    -

    Optional: authenticating through SSH

    -
    +
    +

    Optional: authenticating through SSH

    +

    There are two ways of authenticating and synchronizing your local repository with GitLab: through HTTPS or through SSH. The first one is @@ -293,17 +296,17 @@ your SSH public key on GitLab. This section provides with information on how to do this.

    -
    -

    Installing SSH

    -
    +
    +

    Installing SSH

    +
      -
    • Linux (debian, ubuntu)
      -
      +
    • Linux (Debian, Ubuntu)
      +

      We provide here only instructions for debian-based distributions. Feel free to contribute to this document to provide up-to-date information -for other distributions (e.g., redhat, fedora). +for other distributions (e.g., RedHat, Fedora).

      @@ -315,15 +318,15 @@ Run (as root):

    -
  • Mac OSX
    -
    +
  • macOS
    +

    You do not have anything to do as it is installed by default.

  • -
  • Windows
    -
    +
  • Windows
    +

    You should install the Putty client. Once it is installed, look for the section on generating an SSH key. @@ -332,9 +335,9 @@ the section on generati

  • -
    -

    Setting up SSH on GitLab

    -
    +
    +

    Setting up SSH on GitLab

    +

    Here are all the official explanations on how to set up your SSH key on GitLab. Alternatively, you may also want to have a look at this @@ -345,15 +348,15 @@ video:

    -
    -

    Using Git through the command line to synchronize your local files with Gitlab

    -
    +
    +

    Using Git through the command line to synchronize your local files with Gitlab

    +

    This section describes a generic (through the command line) way to synchronize your local files with Gitlab. You will not need this if -you follow the Jupyter course. If you follow the Rstudio course, all -these operations can be done through Rstudio and you may want to read -the corresponding instructions. If you follow the Org-Mode course, all +you follow the Jupyter path. If you follow the RStudio path, all +these operations can be done through RStudio and you may want to read +the corresponding instructions. If you follow the Org-Mode path, all these operations can be done through Magit and you may want to read the corresponding instructions.

    diff --git a/module2/ressources/jupyter.html b/module2/ressources/jupyter.html index 7c2741c591555842e399ae9336c2b4ec9bce7844..e2b5200e14d62711563728fd717cef083306ee20 100644 --- a/module2/ressources/jupyter.html +++ b/module2/ressources/jupyter.html @@ -4,28 +4,26 @@

    Table of Contents

    -
    -

    1. Jupyter Tips and tricks

    -
    +
    +

    1. Jupyter tips and tricks

    +

    The following webpage lists several Jupyter tricks (in particular, it -illustrates many Ipython magic commands) that should improve your +illustrates many IPython magic commands) that should improve your efficiency (note that this blog post is about two years old so some of the tricks may have been integrated in the default behavior of Jupyter now).

    -
    -

    Running R and Python in the same notebook

    -
    +
    +

    Running R and Python in the same notebook

    +

    rpy2 package allows to use both languages in the same notebook by:

    @@ -84,32 +82,44 @@ python cell.

    -
    -

    Other languages

    -
    +
    +

    Other languages

    +

    -For any reason, you may be unsatisfied with the use of R or of -Python. Many other languages are available: +Jupyter is not limited to Pytyhon and R. Many other languages are available: https://github.com/jupyter/jupyter/wiki/Jupyter-kernels, including -non-free languages like SAS, Mathematica, Matlab… +non-free languages like SAS, Mathematica, Matlab… Note that the maturity of these kernels differs widely.

    None of these other languages have been deployed in the context of our -MOOC but you may want to read the next sections to know more about how -to set up your own Jupyter notebooks on your computer and benefit from these -extensions. +MOOC but you may want to read the next sections to learn how +to set up your own Jupyter on your computer and benefit from these extensions. +

    + +

    +Since the question was asked several times, if you really need to stay +with SAS, you should know that SAS can be used within Jupyter using +either the Python SASKernel or the Python SASPy package (step by step +explanations about this are given here). +

    + +

    +Since proprietary software such as SAS cannot easily be inspected, we discourage its use as it hinders reproducibility by +essence. But perfection does not exist anyway and using Jupyter +literate programming approach allied with systematic control version +and environment control will certainly help anyway.

    -
    -

    2. Installing and configuring Jupyter on your computer

    -
    + +
    +

    2. Installing and configuring Jupyter on your computer

    +

    -In this Section, we provide information on how to set up on your own -computer a Jupyter environment similar to the one deployed for this -MOOC. +In this section, we explain how to set up a Jupyter environment on +your own computer similar to the one deployed for this MOOC.

    @@ -117,15 +127,13 @@ Note that Jupyter notebooks are only a small part of the picture and that Jupyter is now part of a bigger project: JupyterLab, which allows you to mix various components (including notebooks) in your browser. In the context of this MOOC, our time frame was too short to -benefit from JupyterLab which was still under active development but -this is probably the best option now if you want to benefit from -cutting-edge Jupyter notebooks. +benefit from JupyterLab which was still under active development. You may, however, prefer JupyterLab when doing an installation on your own computer.

    -
    -

    2.1 Installing Jupyter

    -
    +
    +

    2.1 Installing Jupyter

    +

    Follow these instructions if you wish to have a Jupyter environment on your own computer similar to the one we set up for this MOOC. @@ -158,18 +166,18 @@ jupyter notebook

    -
    -

    2.2 Making sure Jupyter allows you to use R

    -
    +
    +

    2.2 Making sure Jupyter allows you to use R

    +

    -The previous environment should ship with R but if you proceeded -otherwise and only have python available in Jupyter, you may want to -read the following section. +The environment described in the last section should include R, but if +you proceeded otherwise and only have Python available in Jupyter, you +may want to read the following section.

    -
    -

    • Installing IRKernel (R package)

    -
    +
    +

    • Installing IRKernel (R package)

    +

    Do the following in R console:

    @@ -205,9 +213,9 @@ IRkernel::installspec() #
    -
    -

    • Installing rpy2 (Python package)

    -
    +
    +

    • Installing rpy2 (Python package)

    +

    On Linux, the rpy2 package is available in standard distributions

    @@ -216,8 +224,8 @@ On Linux, the rpy2 package is available in standard distributions

    -An other (not really recommended if the first one is available) -alternative consists in going through the python package manager with +An alternative (not really recommended if the first one is available) +consists in going through the python package manager with

    pip3 install rpy2
    @@ -229,7 +237,7 @@ alternative consists in going through the python package manager with
     

    -Download rpy2 binary file by choosing the right operating system. +Download the rpy2 binary file by choosing the right operating system.

    @@ -250,15 +258,15 @@ Install also tzlocal:

    -
    -

    2.3 Additional tips

    -
    +
    +

    2.3 Additional tips

    +
    -
    -

    • Exporting a notebook

    -
    +
    +

    • Exporting a notebook

    +

    -Here is what we had to install on our recent debian computer to make sure +Here is what we had to install on a recent Debian computer to make sure the notebook export via LaTeX works:

    @@ -294,19 +302,16 @@ taste.

    -Download and install MiKTeX from the MiKTeX webpage -by choosing the right operating system. You will be prompted to install some specific +Download and install MiKTeX from the MiKTeX webpage by choosing the +right operating system. You will be prompted to install some specific packages when exporting to pdf.

    -
    -

    • Side note about Jupyter, JupyterLab, JupyterHub…

    -
    -
    -

    • Improving notebook readability

    -
    +
    +

    • Improving notebook readability

    +

    Here are a few extensions that can ease your life:

    @@ -333,11 +338,11 @@ jupyter-serverextension enable --py hide_code
    -
    -

    • Interacting with GitLab and GitHub

    -
    +
    +

    • Interacting with GitLab and GitHub

    +

    -To ease your experience, we added some pull/push buttons that allow +To ease your experience, we added pull/push buttons that allow you to commit and sync with GitLab. This development was specific to the MOOC but inspired from a previous proof of concept. We have recently discovered that someone else developed about at the same time @@ -358,38 +363,7 @@ is a long an interesting discussion about various options on

    -Last but not least, remember that Jupyter notebooks are only a small -part of the picture and that Jupyter is now part of a bigger project: -JupyterLab, which allows you to mix various components (including -notebooks) in your browser. A specific JupyterLab git plugin has been -developed to offer a nice version control experience. -

    -
    -
    - -
    -

    • Using other languages (e.g., SAS, Matlab, Mathematica, etc.)

    -
    -

    -For any reason, you may be unsatisfied with the use of R or of -Python. Many other languages are available: -https://github.com/jupyter/jupyter/wiki/Jupyter-kernels, including -non-free languages like SAS, Mathematica, Matlab… -

    - -

    -Since the question was asked several times, if you really need to stay -with SAS, you should know that SAS can be used within Jupyter using -either the Python SASKernel or the Python SASPy package (step by step -explanations about this are given here). -

    - -

    -Since such software cannot easily be opened for inspection not widely -used, we discourage this approach as it hinders reproducibility by -essence. But perfection does not exist anyway and using Jupyter -literate programming approach allied with systematic control version -and environment control will certainly help anyway. +For those who use JupyterLab rather than the plain Jupyter, a specific JupyterLab git plugin has been developed to offer a nice version control experience.

    diff --git a/module2/ressources/rstudio.html b/module2/ressources/rstudio.html index 97f9b77146a2c9fa7e0304871c91fd3f259782fc..94066ebdb07ffeb8089663e9190a4611c27618d8 100644 --- a/module2/ressources/rstudio.html +++ b/module2/ressources/rstudio.html @@ -4,84 +4,44 @@

    Table of Contents

    -

    -This document provides information on how to install Rstudio. -

    - -

    -If you have never used git with Rstudio, we strongly advise that you -follow our tutorial on using git from Rstudio ("Rstudio et Gitlab" in French). Before -proceeding, make sure you also have followed the "git/GitLab -configuration" tutorial (in French). -

    - -

    -Alternatively, you may want to watch this video (in English). If you -do not like videos, you should have a look at the step-by-step -explanations from Software Carpentry. It comes with many screenshots -and is quite progressive. -

    - -

    -Last, you should know that the Rstudio team has created a lot of very -good material and tutorials. You should definitively look at the Cheat -sheets webpage. In particular you may want to have look at the -following ones: -

    - -

    -In case it helps, here are some (sometimes outdated) French versions -of these documents: -

    - -
    -

    Installing Rstudio

    -
    +
    +

    Installing RStudio

    +
    -
    -

    Linux (debian, ubuntu)

    -
    +
    +

    Linux (debian, ubuntu)

    +

    -We provide here only instructions for debian-based distributions. Feel +We provide here only instructions for Debian-based distributions. Feel free to contribute to this document to provide up-to-date information -for other distributions (e.g., redhat, fedora). +for other distributions (e.g., RedHat, Fedora).

    Today, the stable versions of the most common distributions provide -recent enough versions of emacs and org-mode: +recent enough versions of R:

    -
    -

    Installing R

    -
    +
    +

    Installing R

    +

    -Beforehand, you need to install the R language and convenient packages +First, you need to install the R language and convenient packages by running (as root):

    @@ -110,7 +70,7 @@ by running (as root): Alternatively, if the installation of r-cran-gplot2 or r-cran-knitr fails, you may want to install them locally (through the R packaging system) and manually by running the following commands in R (or -Rstudio): +RStudio):

    install.packages("knitr")
    @@ -129,14 +89,13 @@ to run (as root):
     
    -
    -

    Installing rstudio

    -
    +
    +

    Installing RStudio

    +

    -Rstudio is unfortunately not packaged within debian so the easiest is -to download the corresponding debian package on the Rstudio webpage -and then to install it manually (depending on when you do this, you -can obviously change the version number). Here is how to install it: +RStudio is unfortunately not packaged within Debian so the easiest is +to download the corresponding Debian package on the RStudio webpage +and then to install it manually (you may have to adjust the version number in the following example). Here is how to install it:

    @@ -149,9 +108,9 @@ sudo apt-get update ; sudo apt-get -f install -

    Mac OSX and Windows

    -
    +
    +

    Mac OSX and Windows

    +

    Some instructions on installing R and knitr must be missing. This @@ -160,9 +119,9 @@ should be tested and improved.

    • Download and install R from the CRAN webpage by choosing the right operating system.
    • -
    • Download and install Rstudio from the Rstudio webpage by choosing the right operating system.
    • +
    • Download and install RStudio from the RStudio webpage by choosing the right operating system.
    • Download and install MiKTeX from the MiKTeX webpage by choosing the right operating system. You will be prompted to install some specific packages when exporting to pdf.
    • -
    • Open Rstudio and type the following commands in the console to install knitr and ggplot2:
    • +
    • Open RStudio and type the following commands in the console to install knitr and ggplot2:
    install.packages("knitr", dep=TRUE)
    @@ -172,13 +131,56 @@ install.packages("ggplot2", dep=
     
    -
    -

    Using Git from Rstudio

    -
    +
    +

    RStudio documentation

    +
    +

    +The RStudio team has created a lot of very good material and +tutorials. You should definitively look at the Cheat sheets +webpage. In particular you may want to have look at the following +ones: +

    + +

    +In case it helps, here are some (sometimes outdated) French versions +of these documents: +

    +
    -
    -

    Cloning a repository

    -
    +
    + +
    +

    Using Git from RStudio

    +
    +

    +If you have never used git with RStudio, we strongly advise that you +follow our tutorial on using git from RStudio ("RStudio et Gitlab" +in French). Before proceeding, make sure you also have followed the +"git/GitLab configuration" tutorial (in French). +

    + +

    +Alternatively, you may want to watch this video (in English). If you +do not like videos, you should have a look at the step-by-step +explanations from Software Carpentry. It comes with many screenshots +and is quite progressive. +

    +
    + +
    +

    Cloning a repository

    +

    Open RStudio and do the following steps:

    @@ -198,7 +200,7 @@ Create a new version controled project: File / New Project / Version Contr
  • -Get the url from your gitlab repository: +Get the URL from your GitLab repository:

    @@ -235,9 +237,9 @@ controled repository.
  • -
    -

    Modifying a file

    -
    +
    +

    Modifying a file

    +
    -
    -

    Additional references

    -
    +
    +

    Additional references

    +
    -
    -

    "Thoughts" on language/software stability

    -
    +
    +

    "Thoughts" on language/software stability

    +

    As we explained, the programming language used in an analysis has a clear influence on the reproducibility of your analysis. It is not a @@ -56,12 +56,12 @@ committee (even though some compilers may not respect this norm).

    On the other end of the spectrum, Python had a much more organic -development based on a readability philosophy and has evolved with -time. Furthermore, python is commonly used as a wrapping language -(e.g., to easily use C or FORTRAN libraries) and has its own packaging -system to make everyone's life easier. All these design choices tend -to make reproducibility often a bit painful with python, even though -the community is slowly taking this into account. +development based on a readability philosophy and valuing continuous +improvement over backwards-compatibility. Furthermore, Python is +commonly used as a wrapping language (e.g., to easily use C or FORTRAN +libraries) and has its own packaging system. All these design choices +tend to make reproducibility often a bit painful with Python, even +though the community is slowly taking this into account. The transition from Python 2 to the not fully backwards compatible Python 3 has been a particularly painful process, not least because the two languages are so similar that is it not always easy to figure out if a given script or module is written in Python 2 or Python 3. It isn't even rare to see Python scripts that work under both Python 2 and Python 3, but produce different results due to the change in the behavior of integer division.

    @@ -72,32 +72,32 @@ solid/stable. R is obviously not immune to evolutions that break old versions and hinder reproducibility/backward compatibility. Here is a relatively recent true story about this and some colleagues who worked on the statistics introductory course with R on FUN reported us -several issues with functions from a few functions (plotmeans from -gplots, survfit from survival, or hclust) whose default -parameters had changed over the years. It is thus probably a good -practice to explicitly indicate in your code default values (, which -can be cumbersome) and to restrict your dependencies as much as -possible. +several issues with a few functions (plotmeans from gplots, +survfit from survival, or hclust) whose default parameters had +changed over the years. It is thus probably good practice to give +explicit values for all parameters (which can be cumbersome) instead +of relying on default values, and to restrict your dependencies as much +as possible.

    This being said, the R development community is generally quite -careful about stability. We (the authors of this MOOC) think open -source (, which allows to inspect how computation is done and to -identify both mistakes and sources of non reproducibility) is more -important than the rock solid stability of SAS, which is a proprietary +careful about stability. We (the authors of this MOOC) believe that open +source (which allows to inspect how computation is done and to +identify both mistakes and sources of non-reproducibility) is more +important than the rock solid stability of SAS, which is proprietary software. Yet, if you really need to stay with SAS (similar solutions probably exist for other languages as well), you should know that SAS can be used within Jupyter using either the Python SASKernel or the Python SASPy package (step by step explanations about this are given here). Using such literate programming approach allied with systematic -control version and environment control will help anyway. +version and environment control will always help.

    -
    -

    Controlling your software environment

    -
    +
    +

    Controlling your software environment

    +

    As we mentioned in the video sequences, there are several solutions to control your environment: @@ -108,8 +108,8 @@ control your environment: clean environment and install only what's strictly necessary (and document it):

    • The very well known Docker
    • -
    • Singularity or Spack, which are more targeted toward high -performance computing users that have specific needs
    • +
    • Singularity or Spack, which are more targeted toward the specific +needs of high performance computing users
    • Guix, Nix that are very clean (perfect?) solutions to this dependency hell and which we recommend
    @@ -133,9 +133,9 @@ Hinsen on Active Papers (http://www.a

    -
    -

    Preservation/Archiving

    -
    + -
    -

    Workflows

    -
    +
    +

    Workflows

    +

    -In the video sequences, we mentioned workflows (original domain in parenthesis): +In the video sequences, we mentioned workflow managers (original application domain in parenthesis):

    • Galaxy (genomics), Kepler (ecology), Taverna (bio-informatics), Pegasus @@ -172,12 +172,12 @@ Bio-informatics: Current Status, Solutions and Research Opportunities
    -
    -

    Numerical and statistical issues

    -
    +
    +

    Numerical and statistical issues

    +

    -These topics could only be mentioned in our MOOC but could by no way -be properly covered. We only suggest here a few interesting talks +We have mentioned these topics in our MOOC but we could by no way +cover them properly. We only suggest here a few interesting talks about this.

      @@ -190,16 +190,16 @@ reproducibility and present recent research work on this topic.
    -
    -

    Publication practices

    -
    +
    +

    Publication practices

    +

    You may want to have a look at the following two webinars:

    -
    -

    Experimentation

    -
    +
    +

    Experimentation

    +

    -Experimentation was not covered in this MOOC whereas it is an +Experimentation was not covered in this MOOC, although it is an essential part of science. The main reason is that practices and -constraints can vary so wildly from a domain to an other that it could +constraints can vary so wildly from one domain to another that it could not be properly covered in a first edition. We would be happy to gather references you consider as interesting in your domain so do not hesitate to provide us with such references by using the forum and we @@ -230,13 +230,13 @@ Science.

    -
    -

    Tracking environment information

    -
    +
    +

    Tracking environment information

    +
    -
    -

    Getting information about your Git repository

    -
    +
    +

    Getting information about your Git repository

    +

    When taking notes, it may be difficult to remember which version of the code or of a file was used. This is what version control is useful @@ -295,7 +295,7 @@ new directories it did not previously know about.

    -Then, I often include commands at the end of my notebook indicating +Then, we often include commands at the end of our notebook indicating how to commit the results (adding the new files, committing with a clear message and pushing). E.g.,

    @@ -327,13 +327,13 @@ is the price to pay for running git from within the notebook itself.

    -
    -

    Getting information about Python(3) libraries

    -
    +
    +

    Getting information about Python(3) libraries

    +
    -
    -

    Getting the list of installed packages and their version

    -
    +
    +

    Getting the list of installed packages and their version

    +

    This topic is discussed on StackOverflow. When using pip (the Python package installer) within a shell command, it is easy to query the @@ -343,7 +343,7 @@ versions of Python are available on your machine

    -Here for example how I get these information on my machine: +Here is for example how I get this information on my machine:

    pip3 freeze
    @@ -441,13 +441,13 @@ Requires: patsy, pandas
     
    -
    -

    How to list imported modules?

    -
    +
    +

    How to list imported modules?

    +

    Without resorting to pip (that will list all available packages), you may want to know which modules are loaded in a Python session as well -as their version. Inspiring from StackOverflow, here is a simple +as their version. Inspired by StackOverflow, here is a simple function that lists loaded package (that have a __version__ attribute, which is unfortunately not completely standard).

    @@ -455,9 +455,11 @@ which is unfortunately not completely standard).
    def print_imported_modules():
         import sys
    -    for name,val in sorted(sys.modules.items()):
    +    for name, val in sorted(sys.modules.items()):
             if(hasattr(val, '__version__')): 
                 print(val.__name__, val.__version__)
    +        else
    +            print(val.__name__, "(unknown version)")
     
     print("**** Package list in the beginning ****");
     print_imported_modules()
    @@ -504,9 +506,9 @@ zlib 1.0
     
    -
    -

    Setting up an environment with pip

    -
    +
    +

    Saving and restoring an environment with pip

    +

    The easiest way to go is as follows:

    @@ -523,9 +525,9 @@ dynamic libraries that are wrapped by Python though.

    -
    -

    Installing a new package or a specific version

    -
    +
    +

    Installing a new package or a specific version

    +

    The Jupyter environment we deployed on our servers for the MOOC is based on the version 4.5.4 of Miniconda and Python 3.6. In this @@ -592,13 +594,13 @@ It is even possible to install a specific (possibly much older) version, e.g.,:

    -
    -

    Getting information about R libraries

    -
    +
    +

    Getting information about R libraries

    +
    -
    -

    Getting the list imported modules and their version

    -
    +
    +

    Getting the list imported modules and their version

    +

    The best way seems to be to rely on the devtools package (if this package is not installed, you should install it first by running in R @@ -661,14 +663,14 @@ Packages ----------------------------------------------------------------------

    Some actually advocate that writing a reproducible research compendium -can be done by writing an R package. Those of you willing to have a +is best done by writing an R package. Those of you willing to have a clean R dependency management should thus have a look at Packrat.

    -
    -

    Getting the list of installed packages and their version

    -
    +
    +

    Getting the list of installed packages and their version

    +

    Finally, it is good to know that there is a built-in R command (installed.packages) allowing to retrieve and list the details of all @@ -923,19 +925,19 @@ packages installed.

    -
    -

    Installing a new package or a specific version

    -
    +
    +

    Installing a new package or a specific version

    +

    This section is mostly a cut and paste from the recent post by Ian -Pylvainen on this topic. It comprises a very clear explanation on how +Pylvainen on this topic. It comprises a very clear explanation of how to proceed.

      -
    • Installing a pre-compiled version
      -
      +
    • Installing a pre-compiled version
      +

      If you're on a Debian or a Ubuntu system, it may be difficult to access a specific version without breaking your system. So unless you @@ -959,8 +961,8 @@ install.packages(packageurl, repos=Using devtools
      -

      +
    • Using devtools
      +

      The simplest method to install the version you need is to use the install_version() function of the devtools package (obviously, you @@ -975,11 +977,12 @@ install_version("ggplot2", version =

    • -
    • Alternatively, you may want to install an older package from source
      -
      +
    • Installing from source code
      +

      -If you devtools fails or if you do not want to depend on it, you can -install it from source via install.packages() directed to the right +Alternatively, you may want to install an older package from source If +devtools fails or if you do not want to depend on it, you can install +it from source via install.packages() directed using the right URL. This URL can be obtained by browsing the CRAN Package Archive.

      @@ -1004,8 +1007,8 @@ R CMD INSTALL ggplot2_0.9.1.tar.gz
    -
  • Potential issues
    -
    +
  • Potential issues
    +

    There are a few potential issues that may arise with installing older versions of packages: