{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Incidence of influenza-like illness in France" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "import pandas as pd\n", "import isoweek" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The data on the incidence of influenza-like illness are available from the Web site of the [Réseau Sentinelles](http://www.sentiweb.fr/). We download them as a file in CSV format, in which each line corresponds to a week in the observation period. Only the complete dataset, starting in 1984 and ending with a recent week, is available for download." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "data_url = \"http://www.sentiweb.fr/datasets/incidence-PAY-3.csv\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is the documentation of the data from [the download site](https://ns.sentiweb.fr/incidence/csv-schema-v1.json):\n", "\n", "| Column name | Description |\n", "|--------------|---------------------------------------------------------------------------------------------------------------------------|\n", "| `week` | ISO8601 Yearweek number as numeric (year times 100 + week nubmer) |\n", "| `indicator` | Unique identifier of the indicator, see metadata document https://www.sentiweb.fr/meta.json |\n", "| `inc` | Estimated incidence value for the time step, in the geographic level |\n", "| `inc_low` | Lower bound of the estimated incidence 95% Confidence Interval |\n", "| `inc_up` | Upper bound of the estimated incidence 95% Confidence Interval |\n", "| `inc100` | Estimated rate incidence per 100,000 inhabitants |\n", "| `inc100_low` | Lower bound of the estimated incidence 95% Confidence Interval |\n", "| `inc100_up` | Upper bound of the estimated rate incidence 95% Confidence Interval |\n", "| `geo_insee` | Identifier of the geographic area, from INSEE https://www.insee.fr |\n", "| `geo_name` | Geographic label of the area, corresponding to INSEE code. This label is not an id and is only provided for human reading |\n", "\n", "The first line of the CSV file is a comment, which we ignore with `skip=1`." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
weekindicatorincinc_lowinc_upinc100inc100_lowinc100_upgeo_inseegeo_name
02024073163687148996.0178378.0245223.0267.0FRFrance
12024063191550179309.0203791.0287269.0305.0FRFrance
22024053216237203595.0228879.0324305.0343.0FRFrance
32024043213196200547.0225845.0320301.0339.0FRFrance
42024033163457152276.0174638.0245228.0262.0FRFrance
\n", "
" ], "text/plain": [ " week indicator inc inc_low inc_up inc100 inc100_low \\\n", "0 202407 3 163687 148996.0 178378.0 245 223.0 \n", "1 202406 3 191550 179309.0 203791.0 287 269.0 \n", "2 202405 3 216237 203595.0 228879.0 324 305.0 \n", "3 202404 3 213196 200547.0 225845.0 320 301.0 \n", "4 202403 3 163457 152276.0 174638.0 245 228.0 \n", "\n", " inc100_up geo_insee geo_name \n", "0 267.0 FR France \n", "1 305.0 FR France \n", "2 343.0 FR France \n", "3 339.0 FR France \n", "4 262.0 FR France " ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "raw_data = pd.read_csv(data_file, skiprows=1)\n", "raw_data.head(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Are there missing data points? Yes, week 19 of year 1989 does not have any observed values." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
weekindicatorincinc_lowinc_upinc100inc100_lowinc100_upgeo_inseegeo_name
18141989193-NaNNaN-NaNNaNFRFrance
\n", "
" ], "text/plain": [ " week indicator inc inc_low inc_up inc100 inc100_low inc100_up \\\n", "1814 198919 3 - NaN NaN - NaN NaN \n", "\n", " geo_insee geo_name \n", "1814 FR France " ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "raw_data[raw_data.isnull().any(axis=1)]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We delete this point, which does not have big consequence for our rather simple analysis." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In order to protect us in case the Réseau Sentinelles Web server disappears or is modified, we make a local copy of this dataset that we store together with our analysis. It is unnecessary and even risky to download the data at each execution, because in case of a malfunction we might be replacing our file by a corrupted version. Therefore we download the data only if no local copy exists." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "data_file = \"syndrome-grippal.csv\"\n", "\n", "import os\n", "import urllib.request\n", "if not os.path.exists(data_file):\n", " urllib.request.urlretrieve(data_url, data_file)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
weekindicatorincinc_lowinc_upinc100inc100_lowinc100_upgeo_inseegeo_name
02024073163687148996.0178378.0245223.0267.0FRFrance
12024063191550179309.0203791.0287269.0305.0FRFrance
22024053216237203595.0228879.0324305.0343.0FRFrance
32024043213196200547.0225845.0320301.0339.0FRFrance
42024033163457152276.0174638.0245228.0262.0FRFrance
52024023129436119453.0139419.0194179.0209.0FRFrance
62024013120769109452.0132086.0181164.0198.0FRFrance
72023523115446103738.0127154.0174156.0192.0FRFrance
82023513148755136546.0160964.0224206.0242.0FRFrance
92023503147971136787.0159155.0223206.0240.0FRFrance
102023493147552136422.0158682.0222205.0239.0FRFrance
112023483124204113479.0134929.0187171.0203.0FRFrance
122023473110910100658.0121162.0167152.0182.0FRFrance
1320234638385375096.092610.0126113.0139.0FRFrance
1420234537200363178.080828.010895.0121.0FRFrance
1520234434995242813.057091.07564.086.0FRFrance
1620234334498238170.051794.06858.078.0FRFrance
1720234235684249277.064407.08675.097.0FRFrance
1820234135835751032.065682.08877.099.0FRFrance
1920234036889460069.077719.010491.0117.0FRFrance
2020233937200363452.080554.010895.0121.0FRFrance
2120233836321855227.071209.09583.0107.0FRFrance
2220233734908542079.056091.07463.085.0FRFrance
2320233633824732237.044257.05849.067.0FRFrance
2420233533169526013.037377.04839.057.0FRFrance
2520233432666321057.032269.04032.048.0FRFrance
2620233331914413161.025127.02920.038.0FRFrance
2720233231464110285.018997.02215.029.0FRFrance
2820233131528610705.019867.02316.030.0FRFrance
292023303132058647.017763.02013.027.0FRFrance
.................................
202119852132609619621.032571.04735.059.0FRFrance
202219852032789620885.034907.05138.064.0FRFrance
202319851934315432821.053487.07859.097.0FRFrance
202419851834055529935.051175.07455.093.0FRFrance
202519851733405324366.043740.06244.080.0FRFrance
202619851635036236451.064273.09166.0116.0FRFrance
202719851536388145538.082224.011683.0149.0FRFrance
20281985143134545114400.0154690.0244207.0281.0FRFrance
20291985133197206176080.0218332.0357319.0395.0FRFrance
20301985123245240223304.0267176.0445405.0485.0FRFrance
20311985113276205252399.0300011.0501458.0544.0FRFrance
20321985103353231326279.0380183.0640591.0689.0FRFrance
20331985093369895341109.0398681.0670618.0722.0FRFrance
20341985083389886359529.0420243.0707652.0762.0FRFrance
20351985073471852432599.0511105.0855784.0926.0FRFrance
20361985063565825518011.0613639.01026939.01113.0FRFrance
20371985053637302592795.0681809.011551074.01236.0FRFrance
20381985043424937390794.0459080.0770708.0832.0FRFrance
20391985033213901174689.0253113.0388317.0459.0FRFrance
204019850239758680949.0114223.0177147.0207.0FRFrance
204119850138548965918.0105060.0155120.0190.0FRFrance
204219845238483060602.0109058.0154110.0198.0FRFrance
2043198451310172680242.0123210.0185146.0224.0FRFrance
20441984503123680101401.0145959.0225184.0266.0FRFrance
2045198449310107381684.0120462.0184149.0219.0FRFrance
204619844837862060634.096606.0143110.0176.0FRFrance
204719844737202954274.089784.013199.0163.0FRFrance
204819844638733067686.0106974.0159123.0195.0FRFrance
20491984453135223101414.0169032.0246184.0308.0FRFrance
205019844436842220056.0116788.012537.0213.0FRFrance
\n", "

2050 rows × 10 columns

\n", "
" ], "text/plain": [ " week indicator inc inc_low inc_up inc100 inc100_low \\\n", "0 202407 3 163687 148996.0 178378.0 245 223.0 \n", "1 202406 3 191550 179309.0 203791.0 287 269.0 \n", "2 202405 3 216237 203595.0 228879.0 324 305.0 \n", "3 202404 3 213196 200547.0 225845.0 320 301.0 \n", "4 202403 3 163457 152276.0 174638.0 245 228.0 \n", "5 202402 3 129436 119453.0 139419.0 194 179.0 \n", "6 202401 3 120769 109452.0 132086.0 181 164.0 \n", "7 202352 3 115446 103738.0 127154.0 174 156.0 \n", "8 202351 3 148755 136546.0 160964.0 224 206.0 \n", "9 202350 3 147971 136787.0 159155.0 223 206.0 \n", "10 202349 3 147552 136422.0 158682.0 222 205.0 \n", "11 202348 3 124204 113479.0 134929.0 187 171.0 \n", "12 202347 3 110910 100658.0 121162.0 167 152.0 \n", "13 202346 3 83853 75096.0 92610.0 126 113.0 \n", "14 202345 3 72003 63178.0 80828.0 108 95.0 \n", "15 202344 3 49952 42813.0 57091.0 75 64.0 \n", "16 202343 3 44982 38170.0 51794.0 68 58.0 \n", "17 202342 3 56842 49277.0 64407.0 86 75.0 \n", "18 202341 3 58357 51032.0 65682.0 88 77.0 \n", "19 202340 3 68894 60069.0 77719.0 104 91.0 \n", "20 202339 3 72003 63452.0 80554.0 108 95.0 \n", "21 202338 3 63218 55227.0 71209.0 95 83.0 \n", "22 202337 3 49085 42079.0 56091.0 74 63.0 \n", "23 202336 3 38247 32237.0 44257.0 58 49.0 \n", "24 202335 3 31695 26013.0 37377.0 48 39.0 \n", "25 202334 3 26663 21057.0 32269.0 40 32.0 \n", "26 202333 3 19144 13161.0 25127.0 29 20.0 \n", "27 202332 3 14641 10285.0 18997.0 22 15.0 \n", "28 202331 3 15286 10705.0 19867.0 23 16.0 \n", "29 202330 3 13205 8647.0 17763.0 20 13.0 \n", "... ... ... ... ... ... ... ... \n", "2021 198521 3 26096 19621.0 32571.0 47 35.0 \n", "2022 198520 3 27896 20885.0 34907.0 51 38.0 \n", "2023 198519 3 43154 32821.0 53487.0 78 59.0 \n", "2024 198518 3 40555 29935.0 51175.0 74 55.0 \n", "2025 198517 3 34053 24366.0 43740.0 62 44.0 \n", "2026 198516 3 50362 36451.0 64273.0 91 66.0 \n", "2027 198515 3 63881 45538.0 82224.0 116 83.0 \n", "2028 198514 3 134545 114400.0 154690.0 244 207.0 \n", "2029 198513 3 197206 176080.0 218332.0 357 319.0 \n", "2030 198512 3 245240 223304.0 267176.0 445 405.0 \n", "2031 198511 3 276205 252399.0 300011.0 501 458.0 \n", "2032 198510 3 353231 326279.0 380183.0 640 591.0 \n", "2033 198509 3 369895 341109.0 398681.0 670 618.0 \n", "2034 198508 3 389886 359529.0 420243.0 707 652.0 \n", "2035 198507 3 471852 432599.0 511105.0 855 784.0 \n", "2036 198506 3 565825 518011.0 613639.0 1026 939.0 \n", "2037 198505 3 637302 592795.0 681809.0 1155 1074.0 \n", "2038 198504 3 424937 390794.0 459080.0 770 708.0 \n", "2039 198503 3 213901 174689.0 253113.0 388 317.0 \n", "2040 198502 3 97586 80949.0 114223.0 177 147.0 \n", "2041 198501 3 85489 65918.0 105060.0 155 120.0 \n", "2042 198452 3 84830 60602.0 109058.0 154 110.0 \n", "2043 198451 3 101726 80242.0 123210.0 185 146.0 \n", "2044 198450 3 123680 101401.0 145959.0 225 184.0 \n", "2045 198449 3 101073 81684.0 120462.0 184 149.0 \n", "2046 198448 3 78620 60634.0 96606.0 143 110.0 \n", "2047 198447 3 72029 54274.0 89784.0 131 99.0 \n", "2048 198446 3 87330 67686.0 106974.0 159 123.0 \n", "2049 198445 3 135223 101414.0 169032.0 246 184.0 \n", "2050 198444 3 68422 20056.0 116788.0 125 37.0 \n", "\n", " inc100_up geo_insee geo_name \n", "0 267.0 FR France \n", "1 305.0 FR France \n", "2 343.0 FR France \n", "3 339.0 FR France \n", "4 262.0 FR France \n", "5 209.0 FR France \n", "6 198.0 FR France \n", "7 192.0 FR France \n", "8 242.0 FR France \n", "9 240.0 FR France \n", "10 239.0 FR France \n", "11 203.0 FR France \n", "12 182.0 FR France \n", "13 139.0 FR France \n", "14 121.0 FR France \n", "15 86.0 FR France \n", "16 78.0 FR France \n", "17 97.0 FR France \n", "18 99.0 FR France \n", "19 117.0 FR France \n", "20 121.0 FR France \n", "21 107.0 FR France \n", "22 85.0 FR France \n", "23 67.0 FR France \n", "24 57.0 FR France \n", "25 48.0 FR France \n", "26 38.0 FR France \n", "27 29.0 FR France \n", "28 30.0 FR France \n", "29 27.0 FR France \n", "... ... ... ... \n", "2021 59.0 FR France \n", "2022 64.0 FR France \n", "2023 97.0 FR France \n", "2024 93.0 FR France \n", "2025 80.0 FR France \n", "2026 116.0 FR France \n", "2027 149.0 FR France \n", "2028 281.0 FR France \n", "2029 395.0 FR France \n", "2030 485.0 FR France \n", "2031 544.0 FR France \n", "2032 689.0 FR France \n", "2033 722.0 FR France \n", "2034 762.0 FR France \n", "2035 926.0 FR France \n", "2036 1113.0 FR France \n", "2037 1236.0 FR France \n", "2038 832.0 FR France \n", "2039 459.0 FR France \n", "2040 207.0 FR France \n", "2041 190.0 FR France \n", "2042 198.0 FR France \n", "2043 224.0 FR France \n", "2044 266.0 FR France \n", "2045 219.0 FR France \n", "2046 176.0 FR France \n", "2047 163.0 FR France \n", "2048 195.0 FR France \n", "2049 308.0 FR France \n", "2050 213.0 FR France \n", "\n", "[2050 rows x 10 columns]" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = raw_data.dropna().copy()\n", "data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our dataset uses an uncommon encoding; the week number is attached\n", "to the year number, leaving the impression of a six-digit integer.\n", "That is how Pandas interprets it.\n", "\n", "A second problem is that Pandas does not know about week numbers.\n", "It needs to be given the dates of the beginning and end of the week.\n", "We use the library `isoweek` for that.\n", "\n", "Since the conversion is a bit lengthy, we write a small Python \n", "function for doing it. Then we apply it to all points in our dataset. \n", "The results go into a new column 'period'." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def convert_week(year_and_week_int):\n", " year_and_week_str = str(year_and_week_int)\n", " year = int(year_and_week_str[:4])\n", " week = int(year_and_week_str[4:])\n", " w = isoweek.Week(year, week)\n", " return pd.Period(w.day(0), 'W')\n", "\n", "data['period'] = [convert_week(yw) for yw in data['week']]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are two more small changes to make.\n", "\n", "First, we define the observation periods as the new index of\n", "our dataset. That turns it into a time series, which will be\n", "convenient later on.\n", "\n", "Second, we sort the points chronologically." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "sorted_data = data.set_index('period').sort_index()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We check the consistency of the data. Between the end of a period and\n", "the beginning of the next one, the difference should be zero, or very small.\n", "We tolerate an error of one second.\n", "\n", "This is OK except for one pair of consecutive periods between which\n", "a whole week is missing.\n", "\n", "We recognize the dates: it's the week without observations that we\n", "have deleted earlier!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "periods = sorted_data.index\n", "for p1, p2 in zip(periods[:-1], periods[1:]):\n", " delta = p2.to_timestamp() - p1.end_time\n", " if delta > pd.Timedelta('1s'):\n", " print(p1, p2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A first look at the data!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sorted_data['inc'].plot()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A zoom on the last few years shows more clearly that the peaks are situated in winter." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sorted_data['inc'][-200:].plot()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Study of the annual incidence" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since the peaks of the epidemic happen in winter, near the transition\n", "between calendar years, we define the reference period for the annual\n", "incidence from August 1st of year $N$ to August 1st of year $N+1$. We\n", "label this period as year $N+1$ because the peak is always located in\n", "year $N+1$. The very low incidence in summer ensures that the arbitrariness\n", "of the choice of reference period has no impact on our conclusions.\n", "\n", "Our task is a bit complicated by the fact that a year does not have an\n", "integer number of weeks. Therefore we modify our reference period a bit:\n", "instead of August 1st, we use the first day of the week containing August 1st.\n", "\n", "A final detail: the dataset starts in October 1984, the first peak is thus\n", "incomplete, We start the analysis with the first full peak." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "first_august_week = [pd.Period(pd.Timestamp(y, 8, 1), 'W')\n", " for y in range(1985,\n", " sorted_data.index[-1].year)]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Starting from this list of weeks that contain August 1st, we obtain intervals of approximately one year as the periods between two adjacent weeks in this list. We compute the sums of weekly incidences for all these periods.\n", "\n", "We also check that our periods contain between 51 and 52 weeks, as a safeguard against potential mistakes in our code." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "year = []\n", "yearly_incidence = []\n", "for week1, week2 in zip(first_august_week[:-1],\n", " first_august_week[1:]):\n", " one_year = sorted_data['inc'][week1:week2-1]\n", " assert abs(len(one_year)-52) < 2\n", " yearly_incidence.append(one_year.sum())\n", " year.append(week2.year)\n", "yearly_incidence = pd.Series(data=yearly_incidence, index=year)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And here are the annual incidences." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "yearly_incidence.plot(style='*')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A sorted list makes it easier to find the highest values (at the end)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "yearly_incidence.sort_values()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, a histogram clearly shows the few very strong epidemics, which affect about 10% of the French population,\n", "but are rare: there were three of them in the course of 35 years. The typical epidemic affects only half as many people." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "yearly_incidence.hist(xrot=20)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.4" } }, "nbformat": 4, "nbformat_minor": 1 }