{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Incidence of influenza-like illness in France"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import isoweek"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data on the incidence of influenza-like illness are available from the Web site of the [Réseau Sentinelles](http://www.sentiweb.fr/). We download them as a file in CSV format, in which each line corresponds to a week in the observation period. Only the complete dataset, starting in 1984 and ending with a recent week, is available for download."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_url = \"http://www.sentiweb.fr/datasets/incidence-PAY-3.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the documentation of the data from [the download site](https://ns.sentiweb.fr/incidence/csv-schema-v1.json):\n",
    "\n",
    "| Column name  | Description                                                                                                               |\n",
    "|--------------|---------------------------------------------------------------------------------------------------------------------------|\n",
    "| `week`       | ISO8601 Yearweek number as numeric (year times 100 + week nubmer)                                                               |\n",
    "| `indicator`  | Unique identifier of the indicator, see metadata document https://www.sentiweb.fr/meta.json                               |\n",
    "| `inc`        | Estimated incidence value for the time step, in the geographic level                                                      |\n",
    "| `inc_low`    | Lower bound of the estimated incidence 95% Confidence Interval                                                            |\n",
    "| `inc_up`     | Upper bound of the estimated incidence 95% Confidence Interval                                                            |\n",
    "| `inc100`     | Estimated rate incidence per 100,000 inhabitants                                                                          |\n",
    "| `inc100_low` | Lower bound of the estimated incidence 95% Confidence Interval                                                            |\n",
    "| `inc100_up`  | Upper bound of the estimated rate incidence 95% Confidence Interval                                                       |\n",
    "| `geo_insee`  | Identifier of the geographic area, from INSEE https://www.insee.fr                                                        |\n",
    "| `geo_name`   | Geographic label of the area, corresponding to INSEE code. This label is not an id and is only provided for human reading |\n",
    "\n",
    "The first line of the CSV file is a comment, which we ignore with `skip=1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_data = pd.read_csv(data_url, skiprows=1)\n",
    "raw_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Are there missing data points? Yes, week 19 of year 1989 does not have any observed values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_data[raw_data.isnull().any(axis=1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We delete this point, which does not have big consequence for our rather simple analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = raw_data.dropna().copy()\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our dataset uses an uncommon encoding; the week number is attached\n",
    "to the year number, leaving the impression of a six-digit integer.\n",
    "That is how Pandas interprets it.\n",
    "\n",
    "A second problem is that Pandas does not know about week numbers.\n",
    "It needs to be given the dates of the beginning and end of the week.\n",
    "We use the library `isoweek` for that.\n",
    "\n",
    "Since the conversion is a bit lengthy, we write a small Python \n",
    "function for doing it. Then we apply it to all points in our dataset. \n",
    "The results go into a new column 'period'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def convert_week(year_and_week_int):\n",
    "    year_and_week_str = str(year_and_week_int)\n",
    "    year = int(year_and_week_str[:4])\n",
    "    week = int(year_and_week_str[4:])\n",
    "    w = isoweek.Week(year, week)\n",
    "    return pd.Period(w.day(0), 'W')\n",
    "\n",
    "data['period'] = [convert_week(yw) for yw in data['week']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two more small changes to make.\n",
    "\n",
    "First, we define the observation periods as the new index of\n",
    "our dataset. That turns it into a time series, which will be\n",
    "convenient later on.\n",
    "\n",
    "Second, we sort the points chronologically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sorted_data = data.set_index('period').sort_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We check the consistency of the data. Between the end of a period and\n",
    "the beginning of the next one, the difference should be zero, or very small.\n",
    "We tolerate an error of one second.\n",
    "\n",
    "This is OK except for one pair of consecutive periods between which\n",
    "a whole week is missing.\n",
    "\n",
    "We recognize the dates: it's the week without observations that we\n",
    "have deleted earlier!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "periods = sorted_data.index\n",
    "for p1, p2 in zip(periods[:-1], periods[1:]):\n",
    "    delta = p2.to_timestamp() - p1.end_time\n",
    "    if delta > pd.Timedelta('1s'):\n",
    "        print(p1, p2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A first look at the data!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_data['inc'].plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A zoom on the last few years shows more clearly that the peaks are situated in winter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_data['inc'][-200:].plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Study of the annual incidence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the peaks of the epidemic happen in winter, near the transition\n",
    "between calendar years, we define the reference period for the annual\n",
    "incidence from August 1st of year $N$ to August 1st of year $N+1$. We\n",
    "label this period as year $N+1$ because the peak is always located in\n",
    "year $N+1$. The very low incidence in summer ensures that the arbitrariness\n",
    "of the choice of reference period has no impact on our conclusions.\n",
    "\n",
    "Our task is a bit complicated by the fact that a year does not have an\n",
    "integer number of weeks. Therefore we modify our reference period a bit:\n",
    "instead of August 1st, we use the first day of the week containing August 1st.\n",
    "\n",
    "A final detail: the dataset starts in October 1984, the first peak is thus\n",
    "incomplete, We start the analysis with the first full peak."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "first_august_week = [pd.Period(pd.Timestamp(y, 8, 1), 'W')\n",
    "                     for y in range(1985,\n",
    "                                    sorted_data.index[-1].year)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Starting from this list of weeks that contain August 1st, we obtain intervals of approximately one year as the periods between two adjacent weeks in this list. We compute the sums of weekly incidences for all these periods.\n",
    "\n",
    "We also check that our periods contain between 51 and 52 weeks, as a safeguard against potential mistakes in our code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "year = []\n",
    "yearly_incidence = []\n",
    "for week1, week2 in zip(first_august_week[:-1],\n",
    "                        first_august_week[1:]):\n",
    "    one_year = sorted_data['inc'][week1:week2-1]\n",
    "    assert abs(len(one_year)-52) < 2\n",
    "    yearly_incidence.append(one_year.sum())\n",
    "    year.append(week2.year)\n",
    "yearly_incidence = pd.Series(data=yearly_incidence, index=year)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here are the annual incidences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yearly_incidence.plot(style='*')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A sorted list makes it easier to find the highest values (at the end)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yearly_incidence.sort_values()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, a histogram clearly shows the few very strong epidemics, which affect about 10% of the French population,\n",
    "but are rare: there were three of them in the course of 35 years. The typical epidemic affects only half as many people."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yearly_incidence.hist(xrot=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}