# -*- mode: org -*-
#+TITLE:     Jupyter
#+AUTHOR:    Arnaud Legrand, Benoit Rospars, Konrad Hinsen
#+DATE: June, 2018
#+STARTUP: overview indent
#+OPTIONS: num:nil toc:t
#+PROPERTY: header-args :eval never-export

* 1. Tips and tricks
The following [[https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/][webpage]] lists several Jupyter tricks (in particular, it
illustrates many =Ipython magic= commands) that should improve your
efficiency (note that this blog post is about two years old so some of
the tricks may have been integrated in the default behavior of Jupyter
now).
** Running R and Python in the same notebook
=rpy2= package allows to use both languages in the same notebook by:
1. Loading =rpy2=: 
   #+begin_src python :results output :exports both
   %load_ext rpy2.ipython
   #+end_src
2. Using the =%R= Ipython magic:
   #+begin_src python :results output :exports both
   %%R 
   summary(cars)
   #+end_src
   Python objects can then even be passed to R as follows (assuming =df=
   is a pandas dataframe):
   #+begin_src python :results output :exports both
   %%R -i df
   plot(df)
   #+end_src
Note that this =%%R= notation allows you to use R for the whole cell but
an other possibility is to use =%R= to have a single line of R within a
python cell.

* 2. Installing and configuring Jupyter on your machine
** 2.1 Existing install of Jupyter
*** • Installing [[https://github.com/IRkernel/IRkernel][IRKernel]] (R package)
Do the following in R console:

Install =devtools= package:

#+begin_src R :results output :session *R* :exports both
install.packages('devtools',dep=TRUE)
#+end_src

Define proxy if needed:

#+begin_src R :results output :session *R* :exports both
library(httr)
set_config(use_proxy(url="proxy", port=80, username="username", password="password")) 
#+end_src

Install =IRkernel= package:

#+begin_src R :results output :session *R* :exports both
devtools::install_github('IRkernel/IRkernel')
IRkernel::installspec()  # to register the kernel in the current R installation
#+end_src

*** • Installing rpy2 (Python package)
#+begin_src shell :results output :exports both
sudo apt-get install python3-rpy2 python3-tzlocal
#+end_src
An other (not really recommended if the first one is available)
alternative consists in going through the python package manager with
#+begin_src python :results output :exports both
pip3 install rpy2
#+end_src

*Windows*

Download =rpy2= [[https://www.lfd.uci.edu/~gohlke/pythonlibs/#rpy2][binary file]] by choosing the right operating system.

Open a DOS console and type the following command:
#+begin_src shell :results output :exports both
python -m pip install rpy2‑2.9.4‑cp37‑cp37m‑win_amd64.whl # adapt filename
#+end_src

Install also =tzlocal=:
#+begin_src shell :results output :exports both
python -m pip install tzlocal
#+end_src

** 2.2 New install of Jupyter
Follow these instructions if you wish to have a Jupyter environment on
your own machine similar to the one we set up for this MOOC.

First, download and install the [[https://conda.io/miniconda.html][latest version of Miniconda]]. We use
Miniconda version =4.5.4= and Python version =3.6= on our server.

Miniconda is a light version of Anaconda, which includes Python, the Jupyter Notebook, and other commonly used packages for scientific computing and data science.

Then download the [[https://gist.github.com/brospars/4671d9013f0d99e1c961482dab533c57][mooc_rr environment file]] and create the environment using conda:
#+begin_src shell :results output :exports both
conda env create -f environment.yml

# Windows activate the environment
activate mooc_rr

# Linux and MacOS activate the environment
source activate mooc_rr

jupyter notebook
#+end_src

** 2.3 Additional tips
*** • Exporting a notebook
Here is what we had to install on our recent debian machine to make sure
the notebook export via latex works:
#+begin_src shell :results output :exports both
sudo apt-get install texlive-xetex wkhtmltopdf
#+end_src

Obviously, you can convert to html or pdf using the using the =File > Download as > HTML= (or =PDF=) menu option.  This can also be done from
the command line with the following command:

#+begin_src sh :results output :exports both
ipython3 nbconvert --to pdf Untitled.ipynb
#+end_src

If you want to use a specific style, then the =nbconvert= exporter
should be customized. This is discussed and demoed [[http://markus-beuckelmann.de/blog/customizing-nbconvert-pdf.html][here]]. We encourage
you to simply read the [[https://nbconvert.readthedocs.io/en/latest/][doc of nbconvert]]. 

Instead of going directly through LaTeX and playing too much with the
=nbconvert= exporter, an other option consists in exporting to Markdown
and playing with [[https://pandoc.org/][pandoc]]. Both approaches work, it's rather a matter of
taste.

*Windows*

Download and install MiKTeX from the [[https://miktex.org/download][MiKTeX webpage]] 
by choosing the right operating system. You will be prompted to install some specific 
packages when exporting to pdf.

*** • Side note about Jupyter, JupyterLab, JupyterHub...
Note that Jupyter notebooks are only a small part of the picture and
that Jupyter is now part of a bigger project: [[https://blog.jupyter.org/jupyterlab-is-ready-for-users-5a6f039b8906][JupyterLab]], which allows
you to mix various components (including notebooks) in your
browser. In the context of this MOOC, our time frame was too short to
benefit from JupyterLab which was still under active development but
this is probably the best option now if you want to benefit from
cutting-edge Jupyter notebooks.

*** • Improving notebook readability
Here are a few extensions that can ease your life:
- [[https://stackoverflow.com/questions/33159518/collapse-cell-in-jupyter-notebook][Code folding]] to improve readability when browsing the notebook.
  #+begin_src shell :results output :exports both
  pip3 install jupyter_contrib_nbextensions
  # jupyter contrib nbextension install --user # not done yet
  #+end_src
- [[https://github.com/kirbs-/hide_code][Hiding code]] to improve readability when exporting.
  #+begin_src sh :results output :exports both
  sudo pip3 install hide_code
  sudo jupyter-nbextension  install --py hide_code
  jupyter-nbextension  enable --py hide_code
  jupyter-serverextension enable --py hide_code
  #+end_src
  
*** • Interacting with GitLab and GitHub
To ease your experience, we added some pull/push buttons that allow
you to commit and sync with GitLab. This development was specific to
the MOOC but inspired from a previous [[https://github.com/Lab41/sunny-side-up][proof of concept]]. We have
recently discovered that someone else developed about at the same time
a [[https://github.com/sat28/githubcommit][rather generic version of this Jupyter plugin]]. Otherwise, remember
that it is very easy to insert a shell cell in Jupyter in which you
can easily issue git commands. This is how we work most of the time.
  
This being said, you may have noticed that Jupyter keeps a perfect
track of the sequence in which cells have been run by updating the
"output index". This is a very good property from the reproducibility
point of view but depending on your usage, you may find it a bit
painful when committing. Some people have thus developed [[https://gist.github.com/pbugnion/ea2797393033b54674af][specific git
hooks]] to ignore these numbers when committing Jupyter notebooks. There
is a long an interesting discussion about various options on
[[https://stackoverflow.com/questions/18734739/using-ipython-notebooks-under-version-control][StackOverflow]].

Last but not least, remember that Jupyter notebooks are only a small
part of the picture and that Jupyter is now part of a bigger project:
[[https://blog.jupyter.org/jupyterlab-is-ready-for-users-5a6f039b8906][JupyterLab]], which allows you to mix various components (including
notebooks) in your browser. A specific [[https://github.com/jupyterlab/jupyterlab-git][JupyterLab git plugin]] has been
developed to offer a nice version control experience.

*** • Using other languages (e.g., SAS, Matlab, Mathematica, etc.)
For any reason, you may be unsatisfied with the use of R or of
Python. Many other languages are available:
[[https://github.com/jupyter/jupyter/wiki/Jupyter-kernels][https://github.com/jupyter/jupyter/wiki/Jupyter-kernels]], including
non-free languages like SAS, Mathematica, Matlab...

Since the question was asked several times, if you really need to stay
with SAS, you should know that SAS can be used within Jupyter using
either the [[https://sassoftware.github.io/sas_kernel/][Python SASKernel]] or the [[https://sassoftware.github.io/saspy/][Python SASPy]] package (step by step
explanations about this are given [[https://app-learninglab.inria.fr/gitlab/85bc36e0a8096c618fbd5993d1cca191/mooc-rr/blob/master/documents/tuto_jupyter_windows/tuto_jupyter_windows.md][here]]). 

Since such software cannot easily be opened for inspection not widely
used, we discourage this approach as it hinders reproducibility by
essence. But perfection does not exist anyway and using Jupyter
literate programming approach allied with systematic control version
and environment control will certainly help anyway.

