{ "cells": [ { "cell_type": "markdown", "metadata": { "hideCode": true, "hidePrompt": true }, "source": [ "# Etude sur le paradoxe de Simpson" ] }, { "cell_type": "markdown", "metadata": { "hideCode": true, "hidePrompt": true }, "source": [ "Paradoxe de Simpson, c'est quoi ca encore ? " ] }, { "cell_type": "markdown", "metadata": { "hideCode": true, "hidePrompt": true }, "source": [ "Un paradoxe plutot intéressant qui se cache dans les jeux de données. Il est important de le connaitre pour éviter des erruers monumentales surtout si on travaille en médecine pour analyser correctement des données de santé. \n", "\n", "Il se produit dans au moins deux cas: \n", "1) Un facteur de confusion qui ne saute pas aux yeux de prime abord mais qui se cache et qui va avoir un impact sur le résultat final. \n", "2) les données ne sont pas réparties de manière homogène = ne suivent pas une distribution normalion (loi normale).\n", "\n", "Utile pour aiguiser son esprit critique quant à ce qui nous est présenté dans la litterature, à la tv pour ne pas se méprendre. Une facon simple de s'en débarasser est de vérifier si notre jeu de données suit bien une distibution selon la loi normale. " ] }, { "cell_type": "markdown", "metadata": { "hideCode": true, "hidePrompt": true }, "source": [ "Pour le comprendre nous allons utiliser un jeu de données historique comparant le taux de mortalité de femmes fumeuses ou non sur 20 ans d'étude -> Appleton, David R., Joyce M. French, and Mark PJ Vanderpump. « Ignoring a covariate: An example of Simpson’s paradox. » The American Statistician 50.4 (1996): 340-341." ] }, { "cell_type": "markdown", "metadata": { "hideCode": true, "hidePrompt": true }, "source": [ "Avant de commencer importons les modules nécessaires à notre analyse: " ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "hideCode": true, "hidePrompt": true }, "outputs": [], "source": [ "%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "import pandas as pd\n" ] }, { "cell_type": "markdown", "metadata": { "hideCode": true, "hidePrompt": true }, "source": [ "Chargeons les données depuis le lien donné par notre navigateur:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "hideCode": true, "hideOutput": true, "hidePrompt": true }, "outputs": [], "source": [ "data_url = \"https://gitlab.inria.fr/learninglab/mooc-rr/mooc-rr-ressources/-/raw/master/module3/Practical_session/Subject6_smoking.csv?inline=false\"" ] }, { "cell_type": "markdown", "metadata": { "hideCode": true, "hidePrompt": true }, "source": [ "Transformons ce jeu de données en DataFrame pandas pour pouvoir l'analyser comme il se doit. " ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "hideCode": true, "hideOutput": true, "hidePrompt": true }, "outputs": [], "source": [ "raw_data = pd.read_csv(data_url)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "hideCode": true, "hidePrompt": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
SmokerStatusAge
0YesAlive21.0
1YesAlive19.3
2NoDead57.5
3NoAlive47.1
4YesAlive81.4
5NoAlive36.8
6NoAlive23.8
7YesDead57.5
8YesAlive24.8
9YesAlive49.5
10YesAlive30.0
11NoDead66.0
12YesAlive49.2
13NoAlive58.4
14NoDead60.6
15NoAlive25.1
16NoAlive43.5
17NoAlive27.1
18NoAlive58.3
19YesAlive65.7
20NoDead73.2
21YesAlive38.3
22NoAlive33.4
23YesDead62.3
24NoAlive18.0
25NoAlive56.2
26YesAlive59.2
27NoAlive25.8
28NoDead36.9
29NoAlive20.2
............
1284YesDead36.0
1285YesAlive48.3
1286NoAlive63.1
1287NoAlive60.8
1288YesDead39.3
1289NoAlive36.7
1290NoAlive63.8
1291NoDead71.3
1292NoAlive57.7
1293NoAlive63.2
1294NoAlive46.6
1295YesDead82.4
1296YesAlive38.3
1297YesAlive32.7
1298NoAlive39.7
1299YesDead60.0
1300NoDead71.0
1301NoAlive20.5
1302NoAlive44.4
1303YesAlive31.2
1304YesAlive47.8
1305YesAlive60.9
1306NoDead61.4
1307YesAlive43.0
1308NoAlive42.1
1309YesAlive35.9
1310NoAlive22.3
1311YesDead62.1
1312NoDead88.6
1313NoAlive39.1
\n", "

1314 rows × 3 columns

\n", "
" ], "text/plain": [ " Smoker Status Age\n", "0 Yes Alive 21.0\n", "1 Yes Alive 19.3\n", "2 No Dead 57.5\n", "3 No Alive 47.1\n", "4 Yes Alive 81.4\n", "5 No Alive 36.8\n", "6 No Alive 23.8\n", "7 Yes Dead 57.5\n", "8 Yes Alive 24.8\n", "9 Yes Alive 49.5\n", "10 Yes Alive 30.0\n", "11 No Dead 66.0\n", "12 Yes Alive 49.2\n", "13 No Alive 58.4\n", "14 No Dead 60.6\n", "15 No Alive 25.1\n", "16 No Alive 43.5\n", "17 No Alive 27.1\n", "18 No Alive 58.3\n", "19 Yes Alive 65.7\n", "20 No Dead 73.2\n", "21 Yes Alive 38.3\n", "22 No Alive 33.4\n", "23 Yes Dead 62.3\n", "24 No Alive 18.0\n", "25 No Alive 56.2\n", "26 Yes Alive 59.2\n", "27 No Alive 25.8\n", "28 No Dead 36.9\n", "29 No Alive 20.2\n", "... ... ... ...\n", "1284 Yes Dead 36.0\n", "1285 Yes Alive 48.3\n", "1286 No Alive 63.1\n", "1287 No Alive 60.8\n", "1288 Yes Dead 39.3\n", "1289 No Alive 36.7\n", "1290 No Alive 63.8\n", "1291 No Dead 71.3\n", "1292 No Alive 57.7\n", "1293 No Alive 63.2\n", "1294 No Alive 46.6\n", "1295 Yes Dead 82.4\n", "1296 Yes Alive 38.3\n", "1297 Yes Alive 32.7\n", "1298 No Alive 39.7\n", "1299 Yes Dead 60.0\n", "1300 No Dead 71.0\n", "1301 No Alive 20.5\n", "1302 No Alive 44.4\n", "1303 Yes Alive 31.2\n", "1304 Yes Alive 47.8\n", "1305 Yes Alive 60.9\n", "1306 No Dead 61.4\n", "1307 Yes Alive 43.0\n", "1308 No Alive 42.1\n", "1309 Yes Alive 35.9\n", "1310 No Alive 22.3\n", "1311 Yes Dead 62.1\n", "1312 No Dead 88.6\n", "1313 No Alive 39.1\n", "\n", "[1314 rows x 3 columns]" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "raw_data" ] }, { "cell_type": "markdown", "metadata": { "hideCode": true, "hidePrompt": true }, "source": [ "Nous avons donc dans cette étude 1314 femmes dont nous savons leur age et statut tabagique lors de la première étude ainsi que leur survie 20 ans plus tard. \n", "=> Etudions donc le taux de mortalité chez ces femmes fumeuses et non fumeuses en dividant le nombre de femmes décédées par le nombre de femmes totales dans chacun des sous-groupes. " ] }, { "cell_type": "markdown", "metadata": { "hideCode": true, "hidePrompt": true }, "source": [ "Pour cela créons 2 catégories de femmes: les fumeuses (smokers) et les non fumeuses (non_smokers)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "hideCode": true, "hidePrompt": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
SmokerStatusAge
0YesAlive21.0
1YesAlive19.3
4YesAlive81.4
7YesDead57.5
8YesAlive24.8
9YesAlive49.5
10YesAlive30.0
12YesAlive49.2
19YesAlive65.7
21YesAlive38.3
23YesDead62.3
26YesAlive59.2
30YesAlive34.6
31YesAlive51.9
32YesAlive49.9
35YesAlive46.7
36YesAlive44.4
37YesAlive29.5
38YesDead33.0
39YesAlive35.6
40YesAlive39.1
42YesAlive35.7
46YesDead44.3
48YesAlive37.5
49YesAlive22.1
53YesAlive39.0
56YesAlive40.1
60YesAlive58.1
61YesAlive37.3
63YesDead36.3
............
1240YesAlive29.7
1243YesAlive40.1
1251YesAlive27.8
1252YesAlive52.4
1253YesAlive27.8
1254YesAlive41.0
1259YesAlive40.8
1260YesAlive20.4
1263YesAlive20.9
1264YesAlive45.5
1269YesAlive38.8
1270YesAlive55.5
1271YesAlive24.9
1273YesAlive55.7
1276YesAlive58.5
1278YesAlive43.7
1282YesAlive51.2
1284YesDead36.0
1285YesAlive48.3
1288YesDead39.3
1295YesDead82.4
1296YesAlive38.3
1297YesAlive32.7
1299YesDead60.0
1303YesAlive31.2
1304YesAlive47.8
1305YesAlive60.9
1307YesAlive43.0
1309YesAlive35.9
1311YesDead62.1
\n", "

582 rows × 3 columns

\n", "
" ], "text/plain": [ " Smoker Status Age\n", "0 Yes Alive 21.0\n", "1 Yes Alive 19.3\n", "4 Yes Alive 81.4\n", "7 Yes Dead 57.5\n", "8 Yes Alive 24.8\n", "9 Yes Alive 49.5\n", "10 Yes Alive 30.0\n", "12 Yes Alive 49.2\n", "19 Yes Alive 65.7\n", "21 Yes Alive 38.3\n", "23 Yes Dead 62.3\n", "26 Yes Alive 59.2\n", "30 Yes Alive 34.6\n", "31 Yes Alive 51.9\n", "32 Yes Alive 49.9\n", "35 Yes Alive 46.7\n", "36 Yes Alive 44.4\n", "37 Yes Alive 29.5\n", "38 Yes Dead 33.0\n", "39 Yes Alive 35.6\n", "40 Yes Alive 39.1\n", "42 Yes Alive 35.7\n", "46 Yes Dead 44.3\n", "48 Yes Alive 37.5\n", "49 Yes Alive 22.1\n", "53 Yes Alive 39.0\n", "56 Yes Alive 40.1\n", "60 Yes Alive 58.1\n", "61 Yes Alive 37.3\n", "63 Yes Dead 36.3\n", "... ... ... ...\n", "1240 Yes Alive 29.7\n", "1243 Yes Alive 40.1\n", "1251 Yes Alive 27.8\n", "1252 Yes Alive 52.4\n", "1253 Yes Alive 27.8\n", "1254 Yes Alive 41.0\n", "1259 Yes Alive 40.8\n", "1260 Yes Alive 20.4\n", "1263 Yes Alive 20.9\n", "1264 Yes Alive 45.5\n", "1269 Yes Alive 38.8\n", "1270 Yes Alive 55.5\n", "1271 Yes Alive 24.9\n", "1273 Yes Alive 55.7\n", "1276 Yes Alive 58.5\n", "1278 Yes Alive 43.7\n", "1282 Yes Alive 51.2\n", "1284 Yes Dead 36.0\n", "1285 Yes Alive 48.3\n", "1288 Yes Dead 39.3\n", "1295 Yes Dead 82.4\n", "1296 Yes Alive 38.3\n", "1297 Yes Alive 32.7\n", "1299 Yes Dead 60.0\n", "1303 Yes Alive 31.2\n", "1304 Yes Alive 47.8\n", "1305 Yes Alive 60.9\n", "1307 Yes Alive 43.0\n", "1309 Yes Alive 35.9\n", "1311 Yes Dead 62.1\n", "\n", "[582 rows x 3 columns]" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "smokers = raw_data[raw_data['Smoker'] == 'Yes']\n", "non_smokers = raw_data[raw_data['Smoker'] == 'No']" ] }, { "cell_type": "markdown", "metadata": { "hideCode": true, "hidePrompt": true }, "source": [ "Il nous faudrait maintenant, dans ces 2 catégories de femmes, des sous groupes dans lesquels nous pourrions distinguer si la personne est en vie ou nous après 20 ans." ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "hideCode": true, "hidePrompt": true }, "outputs": [], "source": [ "Alive_smokers = smokers[smokers['Status'] == 'Alive']\n", "Dead_smokers = smokers[smokers['Status'] == 'Dead']\n", "Alive_non_smokers = non_smokers[non_smokers['Status'] == 'Alive']\n", "Dead_non_smokers = non_smokers[non_smokers['Status'] == 'Dead']" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true, "hideCode": true, "hidePrompt": true }, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "hideCode": true, "hidePrompt": true }, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": { "hideCode": true, "hidePrompt": true }, "source": [ "\n", "\n", "len(smokers) + len(non_smokers)" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "hideCode": true, "hidePrompt": true }, "outputs": [ { "data": { "text/plain": [ "(230, 502)" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(Dead_non_smokers), len(Alive_non_smokers)" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "hideCode": true, "hidePrompt": true }, "outputs": [ { "data": { "text/plain": [ "(139, 443)" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(Dead_smokers), len(Alive_smokers)" ] }, { "cell_type": "markdown", "metadata": { "hideCode": true, "hidePrompt": true }, "source": [ "Dans le but de présenter les données sous forme d'un tableau, il nous faut créer une liste de liste, autrement un dictionnaire. " ] }, { "cell_type": "markdown", "metadata": { "hideCode": true, "hidePrompt": true }, "source": [ "Représentez dans un tableau le nombre total de femmes vivantes et décédées sur la période en fonction de leur habitude de tabagisme. Calculez dans chaque groupe (fumeuses / non fumeuses) le taux de mortalité (le rapport entre le nombre de femmes décédées dans un groupe et le nombre total de femmes dans ce groupe). Vous pourrez proposer une représentation graphique de ces données et calculer des intervalles de confiance si vous le souhaitez. En quoi ce résultat est-il surprenant ?\n", "\n", "Reprenez la question 1 (effectifs et taux de mortalité) en rajoutant une nouvelle catégorie liée à la classe d'âge. On considérera par exemple les classes suivantes : 18-34 ans, 34-54 ans, 55-64 ans, plus de 65 ans. En quoi ce résultat est-il surprenant ? Arrivez-vous à expliquer ce paradoxe ? De même, vous pourrez proposer une représentation graphique de ces données pour étayer vos explications.\n", "\n", "Afin d'éviter un biais induit par des regroupements en tranches d'âges arbitraires et non régulières, il est envisageable d'essayer de réaliser une régression logistique. Si on introduit une variable Death valant 1 ou 0 pour indiquer si l'individu est décédé durant la période de 20 ans, on peut étudier le modèle Death ~ Age pour étudier la probabilité de décès en fonction de l'âge selon que l'on considère le groupe des fumeuses ou des non fumeuses. Ces régressions vous permettent-elles de conclure sur la nocivité du tabagisme ? Vous pourrez proposer une représentation graphique de ces régressions (en n'omettant pas les régions de confiance).\n", "\n", "Déposez votre étude dans FUN" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hideCode": true, "hidePrompt": true }, "outputs": [], "source": [] } ], "metadata": { "hide_code_all_hidden": true, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.4" } }, "nbformat": 4, "nbformat_minor": 2 }