diff --git a/module3/ressources/analyse-syndrome-grippal-orgmode.org b/module3/ressources/analyse-syndrome-grippal-orgmode.org index 1720b70df3a93009c79592b68d1dcf57a53f6341..c782a54c994a5a973de860d7c6bae56988ef9e1b 100644 --- a/module3/ressources/analyse-syndrome-grippal-orgmode.org +++ b/module3/ressources/analyse-syndrome-grippal-orgmode.org @@ -27,11 +27,15 @@ if sys.version_info.major < 3 or sys.version_info.minor < 6: print("Veuillez utiliser Python 3.6 (ou plus) !") #+END_SRC +#+RESULTS: + #+BEGIN_SRC emacs-lisp :results output (unless (featurep 'ob-python) (print "Veuillez activer python dans org-babel (org-babel-do-languages) !")) #+END_SRC +#+RESULTS: + ** R 3.4 Nous n'utilisons que des fonctionnalités de base du langage R, une version antérieure devrait suffire. @@ -40,6 +44,8 @@ Nous n'utilisons que des fonctionnalités de base du langage R, une version ant (print "Veuillez activer R dans org-babel (org-babel-do-languages) !")) #+END_SRC +#+RESULTS: + * Préparation des données Les données de l'incidence du syndrome grippal sont disponibles du site Web du [[http://www.sentiweb.fr/][Réseau Sentinelles]]. Nous les récupérons sous forme d'un fichier en format CSV dont chaque ligne correspond à une semaine de la période d'observation. Nous téléchargeons toujours le jeu de données complet (rien d'autre n'est proposé), qui commence en 1984 et se termine avec une semaine récente. L'URL est: @@ -80,6 +86,13 @@ Regardons ce que nous avons obtenu: table[:5] #+END_SRC +#+RESULTS: +| week | indicator | inc | inc_low | inc_up | inc100 | inc100_low | inc100_up | geo_insee | geo_name | +| 201911 | 3 | 31787 | 26747 | 36827 | 48 | 40 | 56 | FR | France | +| 201910 | 3 | 49871 | 43378 | 56364 | 76 | 66 | 86 | FR | France | +| 201909 | 3 | 88354 | 79564 | 97144 | 134 | 121 | 147 | FR | France | +| 201908 | 3 | 172604 | 160024 | 185184 | 262 | 243 | 281 | FR | France | + ** Recherche de données manquantes Il y a malheureusement beaucoup de façon d'indiquer l'absence d'un point de données. Nous testons ici seulement pour la présence de champs vides. Il faudrait aussi rechercher des valeurs non-numériques dans les colonnes à priori numériques. Nous ne le faisons pas ici, mais une vérification ultérieure capterait des telles anomalies. @@ -94,6 +107,9 @@ for row in table: valid_table.append(row) #+END_SRC +#+RESULTS: +: ['198919', '3', '0', '', '', '0', '', '', 'FR', 'France'] + ** Extraction des colonnes utilisées Il y a deux colonnes qui nous intéressent: la première (~"week"~) et la troisième (~"inc"~). Nous vérifions leurs noms dans l'en-tête, que nous effaçons par la suite. Enfin, nous créons un tableau avec les deux colonnes pour le traitement suivant. #+BEGIN_SRC python :results silent @@ -111,6 +127,21 @@ Regardons les premières et les dernières lignes. Nous insérons ~None~ pour in [('week', 'inc'), None] + data[:5] + [None] + data[-5:] #+END_SRC +#+RESULTS: +| week | inc | +|--------+--------| +| 201911 | 31787 | +| 201910 | 49871 | +| 201909 | 88354 | +| 201908 | 172604 | +| 201907 | 307338 | +|--------+--------| +| 198448 | 78620 | +| 198447 | 72029 | +| 198446 | 87330 | +| 198445 | 135223 | +| 198444 | 68422 | + ** Vérification Il est toujours prudent de vérifier si les données semblent crédibles. Nous savons que les semaines sont données par six chiffres (quatre pour l'année et deux pour la semaine), et que les incidences sont des nombres entiers positifs. #+BEGIN_SRC python :results output @@ -121,6 +152,8 @@ for week, inc in data: print("Valeur suspecte dans la colonne 'inc': ", (week, inc)) #+END_SRC +#+RESULTS: + Pas de problème ! ** Conversions @@ -140,6 +173,21 @@ str_data = [(str(date), str(inc)) for date, inc in converted_data] [('date', 'inc'), None] + str_data[:5] + [None] + str_data[-5:] #+END_SRC +#+RESULTS: +| date | inc | +|------------+--------| +| 1984-10-29 | 68422 | +| 1984-11-05 | 135223 | +| 1984-11-12 | 87330 | +| 1984-11-19 | 72029 | +| 1984-11-26 | 78620 | +|------------+--------| +| 2019-02-11 | 307338 | +| 2019-02-18 | 172604 | +| 2019-02-25 | 88354 | +| 2019-03-04 | 49871 | +| 2019-03-11 | 31787 | + ** Vérification des dates Nous faisons encore une vérification: nos dates doivent être séparées d'exactement une semaine, sauf autour du point manquant. #+BEGIN_SRC python :results output @@ -149,6 +197,9 @@ for date1, date2 in zip(dates[:-1], dates[1:]): print(f"Il y a {date2-date1} entre {date1} et {date2}") #+END_SRC +#+RESULTS: +: Il y a 14 days, 0:00:00 entre 1989-05-01 et 1989-05-15 + ** Passage Python -> R Nous passons au langage R pour inspecter nos données, parce que l'analyse et la préparation de graphiques sont plus concises en R, sans nécessiter aucune bibliothèque supplémentaire. @@ -164,17 +215,32 @@ data$date <- as.Date(data$date) summary(data) #+END_SRC +#+RESULTS: +: date inc +: Min. :1984-10-29 Min. : 0 +: 1st Qu.:1993-06-07 1st Qu.: 5137 +: Median :2002-01-07 Median : 16182 +: Mean :2002-01-06 Mean : 62939 +: 3rd Qu.:2010-08-09 3rd Qu.: 51746 +: Max. :2019-03-11 Max. :1001824 + ** Inspection Regardons enfin à quoi ressemblent nos données ! #+BEGIN_SRC R :results output graphics :file inc-plot.png plot(data, type="l", xlab="Date", ylab="Incidence hebdomadaire") #+END_SRC +#+RESULTS: +[[file:inc-plot.png]] + Un zoom sur les dernières années montre mieux la situation des pics en hiver. Le creux des incidences se trouve en été. #+BEGIN_SRC R :results output graphics :file inc-plot-zoom.png plot(tail(data, 200), type="l", xlab="Date", ylab="Incidence hebdomadaire") #+END_SRC +#+RESULTS: +[[file:inc-plot-zoom.png]] + * Étude de l'incidence annuelle ** Calcul de l'incidence annuelle @@ -201,19 +267,42 @@ inc_annuelle = data.frame(annee = annees, head(inc_annuelle) #+END_SRC +#+RESULTS: +| 1986 | 5100540 | +| 1987 | 2861556 | +| 1988 | 2766142 | +| 1989 | 5460155 | +| 1990 | 5233987 | +| 1991 | 1660832 | + ** Inspection Voici les incidences annuelles en graphique. #+BEGIN_SRC R :results output graphics :file annual-inc-plot.png plot(inc_annuelle, type="p", xlab="Année", ylab="Incidence annuelle") #+END_SRC +#+RESULTS: +[[file:annual-inc-plot.png]] + ** Identification des épidémies les plus fortes Une liste triée par ordre décroissant d'incidence annuelle permet de plus facilement repérer les valeurs les plus élevées: #+BEGIN_SRC R :results output head(inc_annuelle[order(-inc_annuelle$incidence),]) #+END_SRC +#+RESULTS: +: annee incidence +: 4 1989 5460155 +: 5 1990 5233987 +: 1 1986 5100540 +: 28 2013 4182265 +: 25 2010 4085126 +: 14 1999 3897443 + Enfin, un histogramme montre bien que les épidémies fortes, qui touchent environ 10% de la population française, sont assez rares: il y en eu trois au cours des 35 dernières années. #+BEGIN_SRC R :results output graphics :file annual-inc-hist.png hist(inc_annuelle$incidence, breaks=10, xlab="Incidence annuelle", ylab="Nb d'observations", main="") #+END_SRC + +#+RESULTS: +[[file:annual-inc-hist.png]] diff --git a/module3/ressources/annual-inc-hist.png b/module3/ressources/annual-inc-hist.png new file mode 100644 index 0000000000000000000000000000000000000000..ddd5bd5f20a6e95a313ef4aea814b56d1a849126 Binary files /dev/null and b/module3/ressources/annual-inc-hist.png differ diff --git a/module3/ressources/annual-inc-plot.png b/module3/ressources/annual-inc-plot.png new file mode 100644 index 0000000000000000000000000000000000000000..60d9f9151e48bcb52eb16eec34703fe99b5a533d Binary files /dev/null and b/module3/ressources/annual-inc-plot.png differ diff --git a/module3/ressources/inc-plot-zoom.png b/module3/ressources/inc-plot-zoom.png new file mode 100644 index 0000000000000000000000000000000000000000..14853540697c0fb577dfe2a83d8ec1c8bfdaad3c Binary files /dev/null and b/module3/ressources/inc-plot-zoom.png differ diff --git a/module3/ressources/inc-plot.png b/module3/ressources/inc-plot.png new file mode 100644 index 0000000000000000000000000000000000000000..e45b6490192f2d54a2a2dd9cf7f8b1d3c6830f76 Binary files /dev/null and b/module3/ressources/inc-plot.png differ diff --git a/module3/ressources/influenza-like-illness-analysis-orgmode.org b/module3/ressources/influenza-like-illness-analysis-orgmode.org index 6c8b47ad2eefaa2efae0fcda6640ec8b078e7c32..9ecafaf46df9393d37f1160252eafa45932aa2e4 100644 --- a/module3/ressources/influenza-like-illness-analysis-orgmode.org +++ b/module3/ressources/influenza-like-illness-analysis-orgmode.org @@ -26,11 +26,15 @@ if sys.version_info.major < 3 or sys.version_info.minor < 6: print("Please use Python 3.6 (or higher)!") #+END_SRC +#+RESULTS: + #+BEGIN_SRC emacs-lisp :results output (unless (featurep 'ob-python) (print "Please activate python in org-babel (org-babel-do-languages)!")) #+END_SRC +#+RESULTS: + ** R 3.4 We use only basic R functionality, so a earlier version might be OK, but we did not test this. @@ -39,6 +43,8 @@ We use only basic R functionality, so a earlier version might be OK, but we did (print "Please activate R in org-babel (org-babel-do-languages)!")) #+END_SRC +#+RESULTS: + * Data preprocessing The data on the incidence of influenza-like illness are available from the Web site of the [[http://www.sentiweb.fr/][Réseau Sentinelles]]. We download them as a file in CSV format, in which each line corresponds to a week in the observation period. Only the complete dataset, starting in 1984 and ending with a recent week, is available for download. The URL is: @@ -79,6 +85,13 @@ Let's have a look at what we have so far: table[:5] #+END_SRC +#+RESULTS: +| week | indicator | inc | inc_low | inc_up | inc100 | inc100_low | inc100_up | geo_insee | geo_name | +| 201911 | 3 | 31787 | 26747 | 36827 | 48 | 40 | 56 | FR | France | +| 201910 | 3 | 49871 | 43378 | 56364 | 76 | 66 | 86 | FR | France | +| 201909 | 3 | 88354 | 79564 | 97144 | 134 | 121 | 147 | FR | France | +| 201908 | 3 | 172604 | 160024 | 185184 | 262 | 243 | 281 | FR | France | + ** Checking for missing data Unfortunately there are many ways to indicate the absence of a data value in a dataset. Here we check for a common one: empty fields. For completeness, we should also look for non-numerical data in numerical columns. We don't do this here, but checks in later processing steps would catch such anomalies. @@ -93,6 +106,9 @@ for row in table: valid_table.append(row) #+END_SRC +#+RESULTS: +: ['198919', '3', '0', '', '', '0', '', '', 'FR', 'France'] + ** Extraction of the required columns There are only two columns that we will need for our analysis: the first (~"week"~) and the third (~"inc"~). We check the names in the header to be sure we pick the right data. We make a new table containing just the two columns required, without the header. #+BEGIN_SRC python :results silent @@ -110,6 +126,21 @@ Let's look at the first and last lines. We insert ~None~ to indicate to org-mode [('week', 'inc'), None] + data[:5] + [None] + data[-5:] #+END_SRC +#+RESULTS: +| week | inc | +|--------+--------| +| 201911 | 31787 | +| 201910 | 49871 | +| 201909 | 88354 | +| 201908 | 172604 | +| 201907 | 307338 | +|--------+--------| +| 198448 | 78620 | +| 198447 | 72029 | +| 198446 | 87330 | +| 198445 | 135223 | +| 198444 | 68422 | + ** Verification It is always prudent to verify if the data looks credible. A simple fact we can check for is that weeks are given as six-digit integers (four for the year, two for the week), and that the incidence values are positive integers. #+BEGIN_SRC python :results output @@ -120,6 +151,8 @@ for week, inc in data: print("Suspicious value in column 'inc': ", (week, inc)) #+END_SRC +#+RESULTS: + No problem - fine! ** Date conversion @@ -139,6 +172,21 @@ str_data = [(str(date), str(inc)) for date, inc in converted_data] [('date', 'inc'), None] + str_data[:5] + [None] + str_data[-5:] #+END_SRC +#+RESULTS: +| date | inc | +|------------+--------| +| 1984-10-29 | 68422 | +| 1984-11-05 | 135223 | +| 1984-11-12 | 87330 | +| 1984-11-19 | 72029 | +| 1984-11-26 | 78620 | +|------------+--------| +| 2019-02-11 | 307338 | +| 2019-02-18 | 172604 | +| 2019-02-25 | 88354 | +| 2019-03-04 | 49871 | +| 2019-03-11 | 31787 | + ** Date verification We do one more verification: our dates must be separated by exactly one week, except around the missing data point. #+BEGIN_SRC python :results output @@ -148,6 +196,9 @@ for date1, date2 in zip(dates[:-1], dates[1:]): print(f"The difference between {date1} and {date2} is {date2-date1}") #+END_SRC +#+RESULTS: +: The difference between 1989-05-01 and 1989-05-15 is 14 days, 0:00:00 + ** Transfer Python -> R We switch to R for data inspection and analysis, because the code is more concise in R and requires no additional libraries. @@ -163,17 +214,32 @@ data$date <- as.Date(data$date) summary(data) #+END_SRC +#+RESULTS: +: date inc +: Min. :1984-10-29 Min. : 0 +: 1st Qu.:1993-06-07 1st Qu.: 5137 +: Median :2002-01-07 Median : 16182 +: Mean :2002-01-06 Mean : 62939 +: 3rd Qu.:2010-08-09 3rd Qu.: 51746 +: Max. :2019-03-11 Max. :1001824 + ** Inspection Finally we can look at a plot of our data! #+BEGIN_SRC R :results output graphics :file inc-plot.png plot(data, type="l", xlab="Date", ylab="Weekly incidence") #+END_SRC +#+RESULTS: +[[file:inc-plot.png]] + A zoom on the last few years makes the peaks in winter stand out more clearly. #+BEGIN_SRC R :results output graphics :file inc-plot-zoom.png plot(tail(data, 200), type="l", xlab="Date", ylab="Weekly incidence") #+END_SRC +#+RESULTS: +[[file:inc-plot-zoom.png]] + * Study of the annual incidence ** Computation of the annual incidence @@ -201,19 +267,42 @@ annnual_inc = data.frame(year = years, head(annnual_inc) #+END_SRC +#+RESULTS: +| 1986 | 5100540 | +| 1987 | 2861556 | +| 1988 | 2766142 | +| 1989 | 5460155 | +| 1990 | 5233987 | +| 1991 | 1660832 | + ** Inspection A plot of the annual incidence: #+BEGIN_SRC R :results output graphics :file annual-inc-plot.png plot(annnual_inc, type="p", xlab="Année", ylab="Annual incidence") #+END_SRC +#+RESULTS: +[[file:annual-inc-plot.png]] + ** Identification of the strongest epidemics A list sorted by decreasing annual incidence makes it easy to find the most important ones: #+BEGIN_SRC R :results output head(annnual_inc[order(-annnual_inc$incidence),]) #+END_SRC +#+RESULTS: +: year incidence +: 4 1989 5460155 +: 5 1990 5233987 +: 1 1986 5100540 +: 28 2013 4182265 +: 25 2010 4085126 +: 14 1999 3897443 + Finally, a histogram clearly shows the few very strong epidemics, which affect about 10% of the French population, but are rare: there were three of them in the course of 35 years. The typical epidemic affects only half as many people. #+BEGIN_SRC R :results output graphics :file annual-inc-hist.png hist(annnual_inc$incidence, breaks=10, xlab="Annual incidence", ylab="Number of observations", main="") #+END_SRC + +#+RESULTS: +[[file:annual-inc-hist.png]]