diff --git a/module2/ressources/video_examples/README.org b/module2/ressources/video_examples/README.org
new file mode 100755
index 0000000000000000000000000000000000000000..e82c575468ac8eeb867b16293c2626f0cac19b2a
--- /dev/null
+++ b/module2/ressources/video_examples/README.org
@@ -0,0 +1,62 @@
+# -*- coding: utf-8 -*-
+# -*- mode: org -*-
+
+#+TITLE: Org document examples
+#+AUTHOR: Arnaud Legrand
+#+STARTUP: overview indent inlineimages logdrawer
+#+LANGUAGE: en
+
+In the MOOC video, I quickly demo how org-mode can be used in various
+contexts. Here are the (sometimes trimmed) corresponding
+org-files. These documents depend on many other external data files
+and are not meant to lead to reproducible documents but it will give
+you an idea of how it can be organized:
+
+1. [[file:journal.org][journal.org]]: an excerpt (I've only left a few code samples and links
+ to some resources on R, Stats, ...) from my own journal. This is a
+ personal document where everything (meeting notes, hacking, random
+ thoughts, ...) goes by default. Entries are created with the =C-c c=
+ shortcut.
+2. [[file:labbook_single.org][labbook_single.org]]: this is an excerpt from the laboratory notebook
+ [[https://cornebize.net/][Tom Cornebize]] wrote during his Master thesis internship under my
+ supervision. This a personal labbook. I consider this notebook to be
+ excellent and was the ideal level of details for us to communicate
+ without any ambiguity and for him to move forward with confidence.
+3. [[file:paper.org][paper.org]]: this is an ongoing paper based on the previous labbook of
+ Tom Cornebize. As such it is not reproducible as there are hardcoded
+ paths and uncleaned dependencies but writing it from the labbook was
+ super easy as we just had to cut and paste the parts we
+ needed. What may be interesting is the organization and the org
+ tricks to export to the right LaTeX style. As you may notice, in
+ the end of the document, there is a commented section with emacs
+ commands that are automatically executed when opening the file. It
+ is an effective way to depend less on the =.emacs/init.el= which is
+ generally customized by everyone.
+4. [[file:labbook_several.org][labbook_several.org]]: this is a labbook for a specific project shared
+ by several persons. As a consequence it starts with information
+ about installation, common scripts, has section with notes about all
+ our meetings, a section with information about experiments and an
+ other one about analysis. Entries could have been labeled by who
+ wrote them but there were only a few of us and this information was
+ available in git so we did not bother. In such labbook, it is common
+ to find annotations indicating that such experiment was =:FLAWED:= as
+ it had some issues.
+5. [[file:technical_report.org][technical_report.org]]: this is a short technical document I wrote
+ after a colleague sent me a PDF describing an experiment he was
+ conducting and asked me about how reproducible I felt it was. It
+ turned out I had to cut and paste the C code from the PDF, then
+ remove all the line numbers and fix syntax, etc. Obviously I got
+ quite different performance results but writing everything in
+ org-mode made it very easy to generate both HTML and PDF and to
+ explicitly explain how the measurements were done.
+
+Here are a few links to other kind of examples:
+- Slides: all my slides for a series of lectures is available here:
+ https://github.com/alegrand/SMPE. Here is a [[https://raw.githubusercontent.com/alegrand/SMPE/master/lectures/lecture_central_limit_theorem.org][typical source]] and the
+ [[https://raw.githubusercontent.com/alegrand/SMPE/master/lectures/lecture_central_limit_theorem.pdf][resulting PDF]]
+- Lucas Schnorr, a colleague, maintains:
+ - a set of templates for various computer science
+ journals/conferences: [[https://github.com/schnorr/ieeeorg][IEEE]], [[https://github.com/schnorr/wileyorg][Wiley]], [[https://github.com/schnorr/acmorg][ACM]], [[https://github.com/schnorr/llncsorg][LNCS]]
+ - his lecture on programming languages for undergrads:
+ https://github.com/schnorr/mlp/tree/master/conteudo
+
diff --git a/module2/ressources/video_examples/journal.org b/module2/ressources/video_examples/journal.org
new file mode 100755
index 0000000000000000000000000000000000000000..70223877aafd5db908324763d3ec1f063bb266fa
--- /dev/null
+++ b/module2/ressources/video_examples/journal.org
@@ -0,0 +1,2136 @@
+# -*- coding: utf-8 -*-
+#+TITLE: Blog
+#+AUTHOR: Arnaud Legrand
+#+HTML_HEAD:
+#+STARTUP: overview indent inlineimages logdrawer
+#+LANGUAGE: en
+#+TAGS: Seminar(s)
+#+TAGS: SG(s) WP1(1) WP2(2) WP3(3) WP4(4) WP5(5) WP6(6) WP7(7) WP8(8) WP0(0) Argonne(A)
+#+TAGS: POLARIS(P) LIG(L) INRIA (I) HOME(H) Europe(E)
+#+TAGS: twitter(t)
+#+TAGS: Workload(w) BOINC(b) Blog noexport(n) Stats(S)
+#+TAGS: BULL(B)
+#+TAGS: autotuning(a)
+#+TAGS: Epistemology(E) Vulgarization(V) Teaching(T)
+#+TAGS: R(R) Python(p) OrgMode(O) HACSPECIS(h)
+#+PROPERTY: header-args :eval never-export
+#+EXPORT_SELECT_TAGS: Blog
+#+OPTIONS: H:3 num:t toc:t \n:nil @:t ::t |:t ^:t -:t f:t *:t <:t
+#+OPTIONS: TeX:t LaTeX:nil skip:nil d:nil todo:t pri:nil tags:not-in-toc
+#+LATEX_HEADER: %\usepackage{palatino,a4wide,eurosym,graphicx}\usepackage[francais]{babel}
+#+INFOJS_OPT: view:nil toc:nil ltoc:t mouse:underline buttons:0 path:http://orgmode.org/org-info.js
+#+EXPORT_SELECT_TAGS: export
+#+EXPORT_EXCLUDE_TAGS: noexport
+#+EPRESENT_FRAME_LEVEL: 2
+#+COLUMNS: %25ITEM %TODO %3PRIORITY %TAGS
+#+SEQ_TODO: TODO(t!) STARTED(s!) WAITING(w!) APPT(a!) | DONE(d!) CANCELLED(c!) DEFERRED(f!) DELEGATED(D!)
+
+
+* 2011
+** 2011-02 février
+*** 2011-02-08 mardi :R:
+**** Pour apprendre:
+ - Pour les débutants:
+ http://wiki.stdout.org/rcookbook/
+ http://www.r-bloggers.com/
+ http://rstudio.org/ but emacs is just great too once ess is installed
+ - Indispensables:
+ + http://had.co.nz/ggplot2/
+ + http://plyr.had.co.nz/ et la démonstration par l'exemple
+ http://plyr.had.co.nz/09-user/
+ - Une intro pas trop mal faite:
+ - http://bioconnector.github.io/workshops/lessons/intro-r-lifesci/01-intro-r/
+ - Encore plus interactif: http://swirlstats.com/
+ - Plus avancé:
+ http://cran.r-project.org/doc/contrib/Paradis-rdebuts_fr.pdf
+ - Pour ceux qui veulent aller plus loin et coder
+ http://zoonek2.free.fr/UNIX/48_R/all.html
+ - Bien plus avancé pour les fans de sémantique et de ruses de
+ fou par Hadlay Wickam:
+ http://adv-r.had.co.nz/Computing-on-the-language.html
+ - Un [[http://ww2.coastal.edu/kingw/statistics/R-tutorials/dataframes.html][excellent tutorial on data frames]] (=attach=, =with=, =rownames=,
+ =dimnames=, notions of scope...)
+**** R 101 :Blog:
+[[file:public_html/blog/2012/09/12/R101.org][Moved to the blog]]
+[[file:~/Work/SimGrid/infra-songs/slides/140422-compas-R101/R101.org][Compas tutorial]]
+**** R tricks
+***** Reshaping
+http://www.statmethods.net/management/reshape.html
+#+begin_src R :results output :session :exports both
+# example of melt function
+library(reshape)
+mdata <- melt(mydata, id=c("id","time"))
+#+end_src
+***** somme d'éléments avec fenêtre glissante
+#+begin_src R
+filter(x, rep(1,4))
+#+end_src
+***** sorting a data frame
+#+BEGIN_SRC R
+ dd[with(dd, order(-z, b)), ]
+#+END_SRC
+***** Capture output
+#+begin_src R
+sink("myfile.txt", append=TRUE, split=TRUE)
+#+end_src
+When redirecting output, use the cat() function to annotate the
+output.
+***** Batch processing
+#+begin_src sh
+R CMD BATCH [options] my_script.R [outfile]
+#+end_src
+***** Convenient commands
+ - describe
+ - structure
+ - ddply
+ - cbind/rbind
+***** Labels and Factors
+ http://stackoverflow.com/questions/12075037/ggplot-legends-change-labels-order-and-title
+#+begin_src R
+dtt$model <- factor(dtt$model, levels=c("mb", "ma", "mc"), labels=c("MBB", "MAA", "MCC"))
+#+end_src
+ here is another way of reordering factors:
+#+begin_src R
+dtt$model <- relevel(dtt$model, ref="MBB").
+#+end_src
+This puts the factor given by ref at the beginning.
+***** "parallel" Prefix
+#+BEGIN_SRC
+cumsum
+#+END_SRC
+***** knitr preembule
+check out "Tools for making a paper" in R-bloggers:
+#+BEGIN_SRC
+<>=
+opts_knit$set(stop_on_error=2L)
+@
+<>=
+suppressMessages(require(memisc))
+@
+#+END_SRC
+***** Annotate in facet_wrap/facet_grid
+http://www.ansci.wisc.edu/morota/R/ggplot2/ggplot2.html
+***** Interactive plotting
+ http://rstudio.org/docs/advanced/manipulate
+ [[file:~/Work/SimGrid/infra-songs/WP4/R/Sweep3D_analysis/analyze.Rnw]]
+#+begin_src R
+GC <- function(df,start,end) {
+ ggplot(
+ df[(df$Start>=start & df$Start<=end)|(df$End>=start &
+ df$End<=end)|
+ (df$Start<=start & df$End>=end) ,],
+ aes(xmin=Start,xmax=End, ymin=ResourceId, ymax=ResourceId+1,
+ fill=Value))+
+ theme_bw()+geom_rect()+coord_cartesian(xlim = c(start, end))
+}
+GC(df_tau,1.1,1.2)
+animate(GC(df_tau, start, end),start=slider...)
+#+end_src
+***** scoping issue with ggplot: mixing external variables with column names
+ There is a magical function designed for this: here()
+#+BEGIN_EXAMPLE
+ddply(df_native, c("ResourceId"), here(transform),
+ Chunk = compute_chunk(Start,End,Duration,min_time_pure))
+#+END_EXAMPLE
+ Here, min_time_pure is an external variable, not a column name.
+***** speeding things up with parallel plyR
+#+BEGIN_SRC R
+ library(doMP)
+ library(plyr)
+ perf_win <- ddply(df_win,c("host_id"), summarize,
+ astro_avg=sum(et_avg*astro_win),
+ astro_var=sum(et_var*astro_win),
+ seti_avg=sum(et_avg*seti_win),
+ seti_var=sum(et_var*seti_win),
+ .parallel=TRUE, .progress = "text")
+#+END_SRC
+
+***** dessin de graphe avec courbes de bezier
+ https://gist.github.com/dsparks/4331058
+***** Side effect in local functions
+http://my.safaribooksonline.com/book/programming/r/9781449377502/9dot-functions/id3440389
+***** Non-standard evaluation
+http://adv-r.had.co.nz/Computing-on-the-language.html
+***** Arrays of functions in for loops
+http://stackoverflow.com/questions/26064649/enclosing-variables-within-for-loop
+**** R weblinks/statistiques r-cran :WP8:
+ http://en.wikibooks.org/wiki/R_Programming/Graphics
+
+ http://freecode.com/articles/creating-charts-and-graphs-with-gnu-r
+
+ http://www.statmethods.net/graphs/density.html
+
+ http://addictedtor.free.fr/graphiques/RGraphGallery.php?graph=78
+ (scatterplot + histogram)
+
+ http://cran.r-project.org/doc/contrib/Ricci-distributions-en.pdf
+ (comparaison et fitting de distributions)
+
+ http://www.sr.bham.ac.uk/~ajrs/R/r-gallery.html
+ http://www.som.yale.edu/faculty/pks4/files/teaching/handouts/r2_tstat_explained.pdf
+ about t-values
+
+ http://www.statmethods.net/stats/anova.html
+
+ http://www.stat.wisc.edu/courses/st850-lindstro/handouts/blocking.pdf
+ (blocking dans un anova en R)
+
+ http://www-rocq.inria.fr/axis/modulad/archives/numero-34/Goupy-34/goupy-34.pdf
+ (tutorial on DOE in French)
+
+ file:/home/alegrand/Work/Documents/Enseignements/M2R_Mesures_Analyse_Eval_Perf_06/Intro_Statistics/doesimp2excerpt--chap3.pdf
+ file:/home/alegrand/Work/Documents/Enseignements/M2R_Mesures_Analyse_Eval_Perf_06/Intro_Statistics/doeprimer.pdf
+ DOE
+
+ http://cran.r-project.org/doc/contrib/Faraway-PRA.pdf
+ (gros bouquin de R sur l'ANOVA)
+ http://pages.cs.wisc.edu/~cyffka/R_regression-and-anova.pdf
+
+ https://marvelig.liglab.fr/doku.php/thematiques/methodologie/accueil
+ Documents de Nadine Mandran + pointeurs vers cours de stat
+
+ http://pbil.univ-lyon1.fr/R/pdf/bsa.pdf
+ http://grasland.script.univ-paris-diderot.fr/go303/ch5/doc_ch5.htm
+ Document sur l'analyse de données spatialisées
+
+ http://nsaunders.wordpress.com/2010/08/20/a-brief-introduction-to-apply-in-r/
+ Utilisation de Apply
+
+ http://zoonek2.free.fr/UNIX/48_R/all.html
+ Un utilisateur de R qui a noté tout un tas de choses utiles et
+ d'exemples, nottamment de programmation.
+
+ http://sharpstatistics.co.uk/r/ggplot/
+ http://rug.mnhn.fr/semin-r/PDF/INED-SFdS-MNHN_Sueur_280411.pdf
+ Tutorial ggplot2
+
+ https://catalyst.uw.edu/workspace/tbranch/24589/155528
+ Cours sur la visu en R à la Tufte
+
+***** Linear regression and heteroscedasticity :ATTACH:
+:PROPERTIES:
+:Attachments: ModeleLineaireRegrDegerine.pdf Regression101R.pdf GLSHeteroskedasticity.pdf week2_ht.pdf
+:ID: b3ced951-cda8-40ce-b281-cc71b55f1da9
+:END:
+- http://ljk.imag.fr/membres/Anatoli.Iouditski/cours/MLDESS.pdf (see
+ attachment) cours en français sur la régression linéaire, vision
+ proba.
+- http://smat.epfl.ch/courses/Regression/Slides/week2_ht.pdf slides
+ sur la régression linéaire et le lien avec maximum likelihood
+- http://www.r-tutor.com/elementary-statistics/simple-linear-regression/confidence-interval-linear-regression
+ #+begin_src R :results output :session :exports both
+ predict(eruption.lm, newdata, interval="confidence")
+ #+end_src
+- http://www.princeton.edu/~otorres/Regression101R.pdf (std error et
+ confidence interval on parameters estimates + heteroscedasticity)
+- http://www.econ.uiuc.edu/~wsosa/econ471/GLSHeteroskedasticity.pdf
+ Comment gérer l'hétéroscedastisité.
+***** Time series :ATTACH:
+:PROPERTIES:
+:Attachments: SCBio.pdf
+:ID: 40d5498d-e8b3-4c73-8722-7d0056667c15
+:END:
+http://ljk.imag.fr/membres/Serge.Degerine/Enseignement/SCBio.pdf
+
+**** Quantile Regression and Bootstrap :Stats:ATTACH:
+:PROPERTIES:
+:Attachments: mcgill-r.pdf st-m-app-bootstrap.pdf stnews70.pdf
+:ID: 8e3038dc-fa3e-4a7d-a4b1-216513e4359f
+:END:
+http://freakonometrics.hypotheses.org/date/2012/04 (open data and
+ecological falacies (Simpson's paradox)).
+
+http://freakonometrics.hypotheses.org/2396
+(Talk-on-quantiles-at-the-R-Montreal-group)
+
+http://www.cscu.cornell.edu/news/statnews/stnews70.pdf
+**** Reproducible research :WP8:
+Andrew Davison tutorial, which is full of interesting references:
+http://rrcns.readthedocs.org/en/latest/index.html
+***** org-mode
+Une autre approche, uniquement en org.
+http://orgmode.org/worg/org-contrib/babel/how-to-use-Org-Babel-for-R.html
+***** R/Sweave/knitr
+http://users.stat.umn.edu/~geyer//Sweave/
+Sweave, exemple minimaux, emacs.
+
+http://www.bepress.com/cgi/viewcontent.cgi?article=1001&context=bioconductor
+Un article sur reproducible research et sweave
+
+http://cran.r-project.org/web/packages/pgfSweave/vignettes/pgfSweave.pdf
+Pgfsweave, un paquet latex qui améliore le look et la vitesse de
+sweave. Le paquet est mort ceci dit et mes premiers essais n'étaient
+pas concluants car tout convertir en pgf, c'est un peu bourrin.
+
+http://yihui.name/knitr/
+knitr, le dernier, bien à la mode, stable et très prometteur
+
+
+http://www.stat.uiowa.edu/~rlenth/StatWeave/OLD/SRC-talk.pdf
+StatWeave. Permet aussi de mettre du Maple.
+***** Ipython notebook
+https://osf.io/h9gsd/
+sympa, facile à mettre en place
+***** ActivePapers
+- http://www.activepapers.org/
+- https://bitbucket.org/khinsen/active_papers_py/wiki/Tutorial
+***** Elsevier approach
+http://www.elsevier.com/physical-sciences/computer-science/executable-papers
+https://collage.elsevier.com/manual/
+http://is.ieis.tue.nl/staff/pvgorp/research/?page=SCP11
+***** Research Gate
+https://www.researchgate.net/publicliterature.OpenReviewInfo.html
+***** Conference or general discussions
+http://reproducibleresearch.net/index.php/Main_Page
+http://wiki.stodden.net/Main_Page
+
+
+- [[http://www.eecg.toronto.edu/~enright/wddd/][Workshop on Duplicating, Deconstructing and Debunking (WDDD)]] ([[http://cag.engr.uconn.edu/isca2014/workshop_tutorial.html][2014
+ edition]])
+- http://evaluate2010.inf.usi.ch
+- [[http://www.stodden.net/AMP2011/][Reproducible Research: Tools and Strategies for Scientific Computing]]
+- [[http://wssspe.researchcomputing.org.uk/][Working towards Sustainable Software for Science: Practice and
+ Experiences]]
+- [[http://hunoldscience.net/conf/reppar14/pc.html][REPPAR'14: 1st International Workshop on Reproducibility in Parallel
+ Computing]]
+- [[https://www.xsede.org/web/reproducibility][Reproducibility@XSEDE: An XSEDE14 Workshop]]
+- [[http://www.occamportal.org/reproduce][Reproduce/HPCA 2014]]
+- [[http://www.ctuning.org/cm/wiki/index.php?title%3DEvents:TRUST2014][TRUST 2014]]
+- http://vee2014.cs.technion.ac.il/docs/VEE14-present602.pdf
+
+ http://www-958.ibm.com/software/data/cognos/manyeyes/visualizations
+ http://www.myexperiment.org/
+ http://wiki.galaxyproject.org/
+ http://www.runmycode.org/CompanionSite/
+
+ http://evaluate.inf.usi.ch/
+
+ github ?
+ workflow ?
+ vistrails ?
+ sumatra
+ vcr
+***** Politics
+http://michaelnielsen.org/blog/how-you-can-help-the-federal-research-public-access-act-frpaa-become-law/7
+
+http://en.wikipedia.org/wiki/Federal_Research_Public_Access_Act
+
+http://michaelnielsen.org/blog/on-elsevier/
+
+**** General discussions about scientific practice :WP8:
+http://michaelnielsen.org/blog/three-myths-about-scientific-peer-review*/
+http://michaelnielsen.org/blog/some-garbage-in-gold-out/
+http://www.ploscompbiol.org/article/info%3Adoi%2F10.1371%2Fjournal.pcbi.1003285
+**** Coursera
+- https://www.coursera.org/course/compdata
+- https://class.coursera.org/exdata-002/lecture
+- https://class.coursera.org/repdata-002
+**** ggplot2 cool examples
+http://felixfan.github.io/rstudy/2014/02/28/ggplot2-cheatsheet/
+http://blog.revolutionanalytics.com/graphics/
+http://grrrraphics.blogspot.com.br/2012/05/ever-wanted-to-see-at-glance.html
+http://www.ancienteco.com/2012/03/basic-introduction-to-ggplot2.html
+http://sape.inf.usi.ch/quick-reference/ggplot2
+http://www.r-bloggers.com/overplotting-solution-for-black-and-white-graphics/
+http://stats.stackexchange.com/questions/12029/is-it-possible-to-create-parallel-sets-plot-using-r
+http://novyden.blogspot.fr/2013/09/how-to-expand-color-palette-with-ggplot.html
+https://gastonsanchez.wordpress.com/2012/08/27/scatterplot-matrices-with-ggplot/
+**** Visualisations
+http://www.visual-literacy.org/periodic_table/periodic_table.html
+**** Design of Experiments (DoE)
+- Montgommery book
+- http://www.cs.wayne.edu/~hzhang/courses/7290/Lectures/4%20-%20Introduction%20to%20Experimental%20Design.pdf
+- http://www.obgyn.cam.ac.uk/cam-only/statsbook/stexdes.html#3g
+- http://mescal.imag.fr/membres/arnaud.legrand/teaching/2011/EP_czitrom.pdf
+- http://www.basic.northwestern.edu/statguidefiles/oneway_anova_ass_viol.html
+- http://techdigest.jhuapl.edu/TD/td2703/telford.pdf
+*** 2011-02-15 mardi
+**** Réunion CIGRI
+***** Présents
+ - Olivier Richard, Mcf UJF/MESCAL, gestions de ressources,
+ initiateur de OAR et Cigri, G5K
+ - Bruno Bzeznik, Ingénieur CIMENT (admin, gestion clusters) et
+ MESCAL (dev OAR, outils pour CIMENT).
+ - Chislain Charrier, Ingénieur INRIA G5K à Rennes depuis quelques
+ mois. Mission: s'occuper des campagnes d'expérimentations.
+ - Philippe Leprouster, Ingénieur CDD UJF MESCAL pour bosser sur
+ l'optimisation d'OAR
+ - Bernard Boutherin, responsable info au LPSC, noeud Tier3 de la
+ grille EGI (600 coeurs de calcul, 700 To de stoquage,
+ précurseur autour du free-cooling, installation à moins de 60
+ kW depuis 2008).
+ - Catherine Biscarat, IR CNRS qui va s'occuper de la liaison
+ CIGRI/LPSC.
+ - Pierre Neyron, IR CNRS MESCAL/MOAIS, responsable de digitalis.
+***** Point de Bruno sur l'état actuel de Cigri
+ Site web: https://ciment-grid.ujf-grenoble.fr
+ Logiciel principalement déployé dans CIMENT. Exploite
+ actuellement 3000 cores sur une vingtaine de machines.
+ R2D2 et fostino sont les plus grosses et gérées par un seul
+ serveur OAR.
+
+ Resources très faiblement utilisées (en général un ou deux
+ utilisateurs à un instant donnée). Besoin d'accompagner les
+ utilisateurs qui n'ont pas forcément conscience que CIGRI est
+ adapté à leurs besoins. Les utilisateur qui utilisent CIGRI
+ actuellement sont de gros consommateurs de ressources.
+
+ Collaboration CIGRI/LPSC initiée par un projet autour du
+ stoquage. Bruno a du coup équipé CIGRI de noeuds de stoquage et
+ a déployé Irods.
+***** Plus d'infos sur : http://wiki-oar.imag.fr/index.php/CiGri-ng
+
+Entered on [2011-02-15 mar. 09:41]
+
+ [[file:~/Liste.org]]
+* 2012
+** 2013-02 février
+**** 2013-02-11 lundi
+***** Reproducible research links :WP8:R:
+ http://wiki.stodden.net/ICERM_Reproducibility_in_Computational_and_Experimental_Mathematics:_Readings_and_References
+ http://www.rpubs.com/
+ An interesting article with a dissenting opinion on reproducible research:
+ http://cogprints.org/8675/
+
+ Entered on [2013-02-11 lun. 09:52]
+***** Audio StarPU :WP4:
+ Lionel, Samuel, Paul, Luka, Brice.
+****** Séquentialisation des comms
+ - Idées: faire des mesures automatiques
+ - Deux implems' (Sam & Paul), pas équivalentes, l'une modélisant
+ plus les communications synchrone et l'autre les
+ asynchrones. À creuser.
+****** Petites macros pour mesurer/injecter le temps
+ - Temps injecté dans la version initiale de Sam = temps moyen
+ observé par StarPU .
+ - Une fois les problème de communication réglés (virer le
+ slow-start, séquentialiser ce qui doit l'être), les dernières
+ différences viennent de la variabilité vraie vie / simu
+ (surtout sur CPU).
+ - Objectif: insérer variabilité. C'est le même problème que pour
+ SMPI. Dans la version actuelle, on regarde le temps pris lors
+ de la simulation et on le réinjecte, d'où une très mauvaise portabili
+ - Idée: identifier les blocs, capturer les temps et utiliser en
+ simu un tirage à partir du profil capturé. C'est assez
+ "nouveau" car FSuter capturait une trace de niveau MPI donc
+ sans info sur quel bloc de code => pas d'information sur la
+ source de la variabilité.
+ - On commence par une approche basique: à la compilation, on
+ identifie un bloc par FILE,LINE, avec éventuellement une
+ extension via une annotation manuelle (c'est le cas pour
+ StarPU qui lance les calculs toujours au même endroit).
+ - Niveau workflow, première exécution pour avoir les timings,
+ puis R, puis réinsertion ds SG.
+ - La capture est pas compliquée et comme il y a le même besoin
+ pour SMPI, on factorise pour éviter les divergences. Ce code
+ est donc dans SG. Paul et Luka ont fait ça la semaine dernière
+ et Paul l'a utilisé dans *PU, reste à tester pour confirmer
+ - Luka essaie maintenant de mettre ça ds SMPI, c'est plus
+ difficile de savoir où mettre les benchmarks. L'ideal serait
+ de regarder dans la pile, c'est un peu compliqué donc on reste
+ sur notre approche simple pour l'instant et on raffinera plus
+ tard si c'est vraiment nécessaire. L'avantage escompté, c'est
+ sur les plates-formes Mt Blanc par exemples, on peut exécuter
+ une fois et utiliser ensuite les timings pour faire des tests
+ de scalabilité sur une vraie machine de brute qui va vite.
+****** Objectifs des uns et des autres
+ - Lionel & Paul à Bdx: objectif = proposer des modèles, support
+ - Sam: objectif = bricoler DES ordonnanceurs, lancer vite sur
+ différentes et évaluer l'impact de tailles de blocs ou de la
+ taille d'une fenêtre glissante. C'est donc clairement un outil
+ de développement pour tester des choses et il faut donc que
+ l'outil soit un minimum stable. Rien de grave mais il faut
+ bien en être conscient en terme de développement. Il sera
+ important de propager les informations du genre "attention on
+ a corrigé un truc, ça risque d'invalider les expériences
+ précédentes".
+ - Arnaud rappelle que d'un point de vue développement, c'est
+ comme pour SMPI, il faut être conscient qu'il y a trois types
+ de tâches toutes aussi importantes les unes que les autres
+ (i.e., quand on en néglige une on s'en mord toujours les
+ doigts à un moment ou à un autre):
+ + Exploration: le plus fun, de petites expériences pour voir
+ si ça marche. Pour moi, la démarche de Sam et ses
+ expériences faites rentrent dans cette catégorie.
+ + Ingénierie: écriture de code, petites fonctions
+ techniques. Dans le contexte de starPU, typiquement, il
+ s'agit du travail initial de Sam mais aussi du codage de la
+ séquentialisation des communications ou bien des macros de
+ capture de traces.
+ + Consolidation: moins drôle, mais il faut le faire pour
+ vérifier que tout le monde peut fairer ses mesures,
+ réutiliser, et qu'on puisse avancer en toute
+ confiance. Souvent, sur une nouvelle machine, de nouveaux
+ phénomènes apparaissent et ce n'est qu'avec des outils
+ d'exploration systématique et automatiques qu'on s'en sort.
+
+ Il faut donc mettre en place dès le début des outils de capture
+ d'information et d'analyse.
+****** Roadmap
+ - Séquentialisation / Parallélisation: Paul à Bordeaux s'en
+ occupe. Il met en place le code qui crache une matrice
+ d'interférences et met en place dans StarPU/SG le code qui
+ l'exploite.
+ - Infrastructure de mesure / collecte de traces: Luka et Arnaud à
+ Grenoble s'en occupent. Réflexion sur un workflow qui va bien
+ pour garder de bonnes traces et pouvoir facilement tester de
+ nouvelles machines.
+ - Paul vient le 14 Mars à Grenoble et on en profitera pour faire
+ le point.
+****** Divers
+******* Campagne d'expériences pour valider le modèle / invalider les précédents. Comment faire ?
+ Il est difficile (impossible ?) de dire qu'un modèle est
+ valide. Il est plus raisonnable de montrer à quel point on a
+ essayer de l'invalider, ce qui permet à chacun d'évaluer à quel
+ point il fait confiance aux capacités d'extrapolation et
+ d'explication du modèle.
+
+ On peut donc montrer l'impact des améliorations successives du
+ modèle, soit sur le temps final soit sur des choses plus fines
+ de la trace. Il faut tester sur des cas de plus en plus
+ complexes, d'où la nécessité d'avoir une méthode un peu
+ automatique pour comparer des résultats. À titre d'illustration
+ voici le genre de choses que Martin a raconté à l'éval d'Héméra.
+
+ http://mescal.imag.fr/membres/arnaud.legrand/uss_simgrid/130211-HEMERA-eval.pdf
+
+ + Mesure 0: makespan. C'est ce qui nous intéresse mais c'est
+ généralement très pauvre et on peut arriver par hasard à de
+ bons résultats ou avoir de mauvais résultats juste parce
+ qu'un paramètre a été mal mesuré. Même si c'est uniquement,
+ cette mesure là qui nous intéresse au final, il est
+ indispensable de comparer pour des mesures plus fines car
+ c'est ce qui permet de mettre une certaine confiance dans les
+ capacités d'extrapolation de l'outil.
+ + Mesure 1: comparaison visuelle de gantt chartt, peut être
+ joli, facile et instructif avec R, mais difficilement
+ quantifiable.
+ + Mesure 2: Regarder les distributions de temps passés dans
+ différents états. On peut faire ça en partie pour StarPU mais
+ uniquement pour les temps de calcul, pas pour les temps de
+ communication. En effet, on a peu de maîtrises sur les temps
+ de comms, on ne sait pas vraiment à quel moment la
+ communication s'est terminée ni quand elle a commencé.
+ + Mesure 3: Comparer les schedules... C'est difficile car déjà
+ quand c'est stable, une métrique n'est pas évidente à définir
+ mais quand en plus c'est variable d'une fois sur
+ l'autre... Idéalement, il faudrait comparer plutôt la
+ distribution des schedules plutôt que les schedules
+ individuels... C'est difficile mais passionnant. Ce qui est
+ super, c'est qu'on a l'outil qui permet de générer les
+ traces.
+******* Problème de main() entre *PU et SG
+ Passer systématiquement par le XML a résolu le problème de
+ l'initialisation et du lancement de simgrid. Ça reste gênant
+ parce qu'il faut recompiler l'appli Sam. aimerait pouvoir
+ changer l'appli en changeant le LD_LIBRARY_PATH, avec 2
+ versions de libstarpu.so => Mais alors comment passer les
+ arguments à SG ?
+
+ C'est l'appli qui donne argc,argv dans l'appel à *pu_init
+ L'appli risque de pas aimer les options de SG.
+
+ *pu passe par des variables d'environnements pour éviter ces soucis
+
+ Vision d'arnaud : --platform=toto.xml mangé par *pu, reste des
+ args mangés par SG Comment faire pour mettre les stats de temps
+ d'exécution adaptés au fichier XML Ptetre qu'on peut identifier
+ les GPUs par modèle plutôt que par numéro de GPU Possibilité de
+ vérifier les hostname (STARPU_HOSTNAME)
+******* Comment faire pour que ça marche pour exécuter en local.
+ On peut pondre le .xml, tout classer par hostname. Tout est dans
+ .starpu/sampling, pour les différents hostnames et codelets, les
+ traces de perfs.
+
+ Du coup ce serait *pu qui génèrerait automatiquement le .xml ?
+ ça coûte pas cher. Mais les valeurs qu'on met dedans, comment
+ on les obtient ?
+ + BP/latence du bus sont mesuréesde toutes façons au départ
+ + Pour la matrice d'interférence de Paul, il faudrait aussi la
+ mettre dans le XML.
+ Du coup, il serait peut-être plus naturel que ce soit le script
+ de Paul qui ponde le .XML qui s'occupe de l'étalonnage
+
+ Entered on [2013-02-11 lun. 09:53]
+** 2013-05 mai
+*** 2013-05-21 mardi
+**** BIS Workshop
+
+#+BEGIN_SRC sh :results output raw :exports both
+for i in gnome gnome-desktop-environment ifupdown iproute iproute-dev isc-dhcp-client libatm1 network-manager network-manager-gnome ; do
+ dir=`apt-cache showsrc $i | grep Directory | sed 's/.*: //'`
+ version=`apt-cache showsrc $i | grep ^Version | sed 's/.*: *//g'`
+ echo "wget http://http.us.debian.org/debian/$dir/$i""_$version"_amd64.deb
+done
+#+END_SRC
+
+Entered on [2013-05-21 mar. 08:46]
+**** Discussions avec Anne-Cécile à propos des timeouts TCP MPI :WP4:
+ I have some related news. I had the chance to chat with
+ Anne-Cecile and talked her about our timeout problem. After
+ digging a little, she was able to point me to related work:
+ - Understanding TCP Incast Throughput Collapse in Datacenter
+ Networks
+ - Safe and Effective Fine-grained TCP Retransmissions for
+ Datacenter Communication
+ - On the properties of an adaptive TCP minimum rto
+ - http://www.hjp.at/doc/rfc/rfc2988.txt
+
+ So the problem has a name (incast) and is linked to the following
+ TCP parameter:
+ | OS | TCP RTOmin |
+ |---------+-------------|
+ | Linux | 200ms |
+ | BSD | 200ms |
+ | Solaris | 400ms |
+
+ I haven't read the articles so I don't know the details. All I
+ can say so far is I don't know how to change this parameter
+ without recompiling the kernel...
+
+ #+BEGIN_SRC sh :results output raw :exports both
+ cd /usr/src/linux-headers-3.2.0-4-common
+ cg 'define *TCP_RTO_MIN' '*'
+ cg 'define *HZ' '*'
+ #+END_SRC
+
+ Basically, this value is good enough for wide area where RTT is
+ large but in our SAN setting, it's rather bad. Looking further,
+ http://comments.gmane.org/gmane.linux.network/162986, I learnt
+ that although this parameter cannot be modified through sysctl,
+ it could be overriden per route with iproute.
+
+
+ #+BEGIN_SRC sh :results output text :exports both
+ for i in ip ip-address ipcontroller iplogger ip-netns iproxy iptables-save ipython ip6tables ip-addrlabel ipcrm ip-maddress ip-ntable ip-rule iptables-xml ipython2.6 ip6tables-apply ipc ipcs ip-monitor ipptool iptables ip-tunnel ipython2.7 ip6tables-restore ipcluster ipengine ip-mroute ipptoolfile iptables-apply ipv6 ip6tables-save ipcmk ip-link ip-neighbour ip-route iptables-restore ip-xfrm ; do
+ man -T --troff-device=ascii $i | grep -i rto
+ done
+ #+END_SRC
+
+ #+RESULTS:
+
+Entered on [2013-05-21 mar. 12:37]
+**** Play with R xkcd :R:
+ #+begin_src R :results output :session :exports both
+ # install.packages('xkcd') # did not work so I made it manually
+ library(extrafont)
+ download.file("http://simonsoftware.se/other/xkcd.ttf", dest="xkcd.ttf")
+ system("mkdir ~/.fonts")
+ system("cp xkcd.tff -t ~/.fonts")
+ font_import()
+ loadfonts()
+ #+END_SRC
+
+ #+BEGIN_SRC R :results graphics :file /tmp/plot.png :exports results :width 600 :height 200 :session
+ library(xkcd)
+ theme_xkcd <- theme(
+ panel.background = element_rect(fill="white"),
+ axis.ticks = element_line(colour=NA),
+ panel.grid = element_line(colour="white"),
+ axis.text.y = element_text(colour=NA),
+ axis.text.x = element_text(colour="black"),
+ text = element_text(size=16, family="xkcd")
+ )
+ ggplot(data.frame(x=c(0, 10)), aes(x)) + theme_xkcd +
+ stat_function(fun=sin,position="jitter", color="red", size=2) +
+ stat_function(fun=cos,position="jitter", color="white", size=3) +
+ stat_function(fun=cos,position="jitter", color="blue", size=2) +
+ geom_text(family="xkcd", x=4, y=0.7, label="A SIN AND COS CURVE")+
+ xkcdaxis(c(0, 10),c(-1,1))
+ #+END_SRC
+
+ #+RESULTS:
+ [[file:/tmp/plot.png]]
+
+Entered on [2013-05-21 mar. 15:49]
+* 2015
+** 2015-01 janvier
+*** 2015-01-07 mercredi
+**** Helping Martin with R :Teaching:R:
+
+#+tblname: daily
+| Date | exos_java | traces_java | exos_python | traces_python | exos_scala | traces_scala |
+|------------+-----------+-------------+-------------+---------------+------------+--------------|
+| 2014.9.2 | 6 | 1 | 0 | 0 | 0 | 0 |
+| 2014.9.3 | 5 | 1 | 0 | 0 | 0 | 0 |
+| 2014.9.4 | 8 | 2 | 0 | 0 | 0 | 0 |
+| 2014.9.8 | 7 | 4 | 0 | 0 | 1290 | 86 |
+| 2014.9.9 | 0 | 0 | 3 | 1 | 1615 | 86 |
+| 2014.9.10 | 0 | 0 | 1 | 1 | 163 | 16 |
+| 2014.9.11 | 3 | 2 | 0 | 0 | 999 | 63 |
+| 2014.9.12 | 67 | 4 | 2 | 2 | 1149 | 67 |
+| 2014.9.13 | 20 | 3 | 1 | 1 | 132 | 14 |
+| 2014.9.14 | 7 | 1 | 0 | 0 | 170 | 12 |
+| 2014.9.15 | 9 | 2 | 0 | 0 | 1112 | 73 |
+| 2014.9.16 | 16 | 2 | 0 | 0 | 768 | 60 |
+| 2014.9.17 | 36 | 3 | 0 | 0 | 274 | 40 |
+| 2014.9.18 | 1 | 1 | 22 | 2 | 20 | 2 |
+| 2014.9.19 | 1 | 1 | 18 | 2 | 10 | 2 |
+| 2014.9.20 | 0 | 0 | 12 | 1 | 61 | 6 |
+| 2014.9.21 | 0 | 0 | 6 | 2 | 36 | 6 |
+| 2014.9.22 | 3 | 2 | 11 | 2 | 420 | 50 |
+| 2014.9.23 | 1 | 1 | 0 | 0 | 218 | 31 |
+| 2014.9.24 | 0 | 0 | 12 | 2 | 39 | 4 |
+| 2014.9.25 | 0 | 0 | 1 | 1 | 220 | 30 |
+| 2014.9.26 | 0 | 0 | 19 | 2 | 28 | 5 |
+| 2014.9.27 | 0 | 0 | 10 | 1 | 17 | 4 |
+| 2014.9.28 | 0 | 0 | 12 | 2 | 37 | 6 |
+| 2014.9.29 | 26 | 3 | 8 | 1 | 509 | 81 |
+| 2014.9.30 | 9 | 2 | 16 | 2 | 243 | 36 |
+| 2014.10.1 | 1 | 1 | 26 | 14 | 99 | 16 |
+| 2014.10.2 | 1 | 1 | 1 | 1 | 325 | 38 |
+| 2014.10.3 | 26 | 15 | 52 | 16 | 22 | 4 |
+| 2014.10.4 | 25 | 1 | 4 | 3 | 36 | 9 |
+| 2014.10.5 | 10 | 1 | 2 | 1 | 49 | 9 |
+| 2014.10.6 | 5 | 2 | 39 | 22 | 192 | 37 |
+| 2014.10.7 | 24 | 4 | 17 | 7 | 143 | 25 |
+| 2014.10.8 | 50 | 3 | 0 | 0 | 77 | 14 |
+| 2014.10.9 | 24 | 2 | 11 | 3 | 48 | 9 |
+| 2014.10.10 | 35 | 4 | 7 | 2 | 0 | 0 |
+| 2014.10.11 | 0 | 0 | 9 | 3 | 3 | 1 |
+| 2014.10.12 | 20 | 6 | 7 | 3 | 10 | 1 |
+| 2014.10.13 | 32 | 4 | 18 | 4 | 0 | 0 |
+| 2014.10.14 | 44 | 1 | 41 | 3 | 8 | 1 |
+| 2014.10.15 | 5 | 3 | 64 | 10 | 6 | 2 |
+| 2014.10.16 | 27 | 2 | 24 | 5 | 1 | 1 |
+| 2014.10.17 | 43 | 3 | 14 | 4 | 0 | 0 |
+| 2014.10.18 | 84 | 2 | 57 | 8 | 0 | 0 |
+| 2014.10.19 | 10 | 2 | 86 | 11 | 0 | 0 |
+| 2014.10.20 | 0 | 0 | 94 | 11 | 0 | 0 |
+| 2014.10.21 | 15 | 1 | 67 | 8 | 10 | 2 |
+| 2014.10.22 | 20 | 5 | 76 | 15 | 1 | 1 |
+| 2014.10.23 | 33 | 3 | 12 | 5 | 0 | 0 |
+| 2014.10.24 | 29 | 2 | 58 | 11 | 1 | 1 |
+| 2014.10.25 | 33 | 8 | 38 | 8 | 1 | 1 |
+| 2014.10.26 | 13 | 6 | 39 | 8 | 34 | 3 |
+| 2014.10.27 | 13 | 4 | 49 | 12 | 15 | 1 |
+| 2014.10.28 | 4 | 2 | 44 | 8 | 3 | 1 |
+| 2014.10.29 | 0 | 0 | 28 | 9 | 13 | 2 |
+| 2014.10.30 | 4 | 3 | 49 | 8 | 0 | 0 |
+| 2014.10.31 | 3 | 2 | 58 | 14 | 7 | 1 |
+| 2014.11.1 | 0 | 0 | 71 | 9 | 7 | 2 |
+| 2014.11.2 | 23 | 2 | 57 | 6 | 0 | 0 |
+| 2014.11.3 | 10 | 1 | 18 | 5 | 0 | 0 |
+| 2014.11.4 | 19 | 1 | 49 | 10 | 3 | 1 |
+| 2014.11.5 | 29 | 2 | 28 | 9 | 0 | 0 |
+| 2014.11.6 | 86 | 3 | 142 | 19 | 0 | 0 |
+| 2014.11.7 | 38 | 2 | 4 | 2 | 0 | 0 |
+| 2014.11.8 | 0 | 0 | 18 | 4 | 6 | 1 |
+| 2014.11.9 | 25 | 2 | 39 | 10 | 0 | 0 |
+| 2014.11.10 | 16 | 1 | 17 | 3 | 0 | 0 |
+| 2014.11.11 | 0 | 0 | 70 | 16 | 1 | 1 |
+| 2014.11.12 | 0 | 0 | 4 | 3 | 0 | 0 |
+| 2014.11.13 | 0 | 0 | 168 | 20 | 1 | 1 |
+| 2014.11.14 | 0 | 0 | 18 | 2 | 0 | 0 |
+| 2014.11.15 | 0 | 0 | 5 | 2 | 8 | 1 |
+| 2014.11.16 | 16 | 2 | 16 | 4 | 4 | 1 |
+| 2014.11.17 | 0 | 0 | 8 | 3 | 0 | 0 |
+| 2014.11.18 | 4 | 1 | 7 | 3 | 0 | 0 |
+| 2014.11.19 | 17 | 2 | 4 | 1 | 0 | 0 |
+| 2014.11.20 | 0 | 0 | 102 | 13 | 0 | 0 |
+| 2014.11.21 | 7 | 1 | 31 | 3 | 1 | 1 |
+| 2014.11.22 | 1 | 1 | 17 | 4 | 0 | 0 |
+| 2014.11.23 | 4 | 1 | 25 | 6 | 0 | 0 |
+| 2014.11.24 | 0 | 0 | 2 | 1 | 3 | 1 |
+| 2014.11.25 | 4 | 1 | 0 | 0 | 7 | 2 |
+| 2014.11.26 | 0 | 0 | 4 | 1 | 0 | 0 |
+| 2014.11.27 | 0 | 0 | 1 | 1 | 6 | 1 |
+| 2014.11.28 | 0 | 0 | 6 | 3 | 1 | 1 |
+| 2014.11.29 | 1 | 1 | 29 | 4 | 13 | 3 |
+| 2014.11.30 | 3 | 1 | 57 | 10 | 15 | 2 |
+| 2014.12.1 | 8 | 1 | 15 | 4 | 7 | 3 |
+| 2014.12.2 | 8 | 3 | 17 | 5 | 0 | 0 |
+| 2014.12.3 | 3 | 1 | 6 | 2 | 0 | 0 |
+| 2014.12.4 | 4 | 3 | 1 | 1 | 1 | 1 |
+| 2014.12.5 | 0 | 0 | 17 | 2 | 5 | 2 |
+| 2014.12.6 | 0 | 0 | 6 | 2 | 3 | 1 |
+| 2014.12.7 | 0 | 0 | 7 | 3 | 0 | 0 |
+| 2014.12.8 | 11 | 3 | 0 | 0 | 0 | 0 |
+| 2014.12.9 | 7 | 1 | 0 | 0 | 0 | 0 |
+| 2014.12.10 | 27 | 2 | 0 | 0 | 0 | 0 |
+| 2014.12.11 | 0 | 0 | 0 | 0 | 1 | 1 |
+| 2014.12.13 | 17 | 3 | 0 | 0 | 0 | 0 |
+| 2014.12.14 | 3 | 1 | 10 | 1 | 0 | 0 |
+| 2014.12.15 | 25 | 3 | 1 | 1 | 9 | 2 |
+| 2014.12.16 | 34 | 3 | 10 | 4 | 0 | 0 |
+| 2014.12.17 | 11 | 2 | 3 | 2 | 1 | 1 |
+| 2014.12.18 | 3 | 1 | 8 | 1 | 0 | 0 |
+| 2014.12.19 | 7 | 1 | 1 | 1 | 9 | 1 |
+| 2014.12.20 | 96 | 3 | 11 | 4 | 0 | 0 |
+| 2014.12.21 | 1 | 1 | 17 | 4 | 12 | 3 |
+| 2014.12.23 | 0 | 0 | 21 | 5 | 1 | 1 |
+| 2014.12.24 | 5 | 1 | 11 | 4 | 0 | 0 |
+| 2014.12.25 | 14 | 2 | 8 | 2 | 0 | 0 |
+| 2014.12.26 | 0 | 0 | 13 | 4 | 0 | 0 |
+| 2014.12.27 | 0 | 0 | 9 | 3 | 0 | 0 |
+| 2014.12.28 | 0 | 0 | 24 | 4 | 0 | 0 |
+| 2014.12.29 | 0 | 0 | 21 | 7 | 0 | 0 |
+| 2014.12.30 | 0 | 0 | 34 | 6 | 0 | 0 |
+| 2014.12.31 | 1 | 1 | 47 | 5 | 0 | 0 |
+| 2015.1.1 | 0 | 0 | 33 | 4 | 0 | 0 |
+| 2015.1.2 | 0 | 0 | 29 | 7 | 0 | 0 |
+| 2015.1.3 | 0 | 0 | 25 | 4 | 12 | 1 |
+| 2015.1.4 | 0 | 0 | 14 | 5 | 0 | 0 |
+| 2015.1.5 | 12 | 2 | 0 | 0 | 0 | 0 |
+
+
+#+tblname: idle_periods_mt
+| Start | End |
+|------------+------------|
+| 2014.9.2 | 2014.9.20 |
+| 2014.12.18 | 2014.12.31 |
+
+
+#+begin_src R :exports both :results output graphics :var daily=daily :var idle=idle_periods_mt :file /tmp/daily.png :width 600 :height 600
+ library(reshape2)
+ library(ggplot2)
+ require(gridExtra)
+
+ daily$Date <- as.Date(daily$Date, "%Y.%m.%d")
+
+ data_long <- melt(daily, id.vars=c("Date"))
+
+ idle$Start <- as.Date(idle$Start, "%Y.%m.%d")
+ idle$End <- as.Date(idle$End, "%Y.%m.%d")
+
+ ymax1=200
+
+ p1 <- ggplot() +
+ geom_area(data=data_long[data_long$variable %in% c("exos_scala","exos_python","exos_java"),],
+ aes(x=Date, y=value, color=variable,fill=variable)) +
+ ggtitle("Daily activity (exercises)") +
+ geom_rect(data=idle,aes(xmin=Start, xmax=End, ymin=0, ymax=ymax1),alpha=.1,fill="red",color="blue") +
+ theme(legend.justification=c(1,0), legend.position=c(1,.6)) +
+ coord_cartesian(ylim = c(0,ymax1)) +
+ ylab("Exercises (#)")
+
+ ymax2 = 40
+
+ p2 <- ggplot() +
+ geom_area(data=data_long[data_long$variable %in% c("traces_scala","traces_python","traces_java"),],
+ aes(x=Date, y=value, color=variable,fill=variable)) +
+ ggtitle("Daily activity (users)") +
+ geom_rect(data=idle,aes(xmin=Start, xmax=End, ymin=0, ymax=ymax2),alpha=.1,fill="red",color="blue") +
+ theme(legend.justification=c(1,0), legend.position=c(1,.6)) +
+ coord_cartesian(ylim = c(0,ymax2)) + ### zoom with ggplot
+ ylab("Active Traces (#)")
+ grid.arrange(p1, p2)
+#+end_src
+
+#+RESULTS:
+[[file:/tmp/daily.png]]
+
+Entered on [2015-01-07 mer. 16:27]
+
+ [[file:/tmp/plm-iticse.org::*Data%20Analysis][Data Analysis]]
+** 2015-07 juillet
+*** 2015-07-31 vendredi
+**** LOESS :WP8:Stats:R:
+An involved lecture:
+ http://web.as.uky.edu/statistics/users/pbreheny/621/F10/notes/11-4.pdf
+
+A few R examples illustrating the influence of bandwidth:
+- http://research.stowers-institute.org/efg/R/Statistics/loess.htm
+- http://www.duclert.org/Aide-memoire-R/Statistiques/Local-polynomial-fitting.php
+
+Entered on [2015-07-31 ven. 09:35]
+**** Harald Servat's Phd defense
+Comments:
+- I really enjoyed the very *clear presentation* of the document, of the
+ related work, etc.
+- I particularly enjoyed the fact that *hypothesis are clearly stated*,
+ which I think is the sign of a true *scientific approach* in term of
+ methodology. I also think that moving to *continous approximations* as
+ you tried by using Kriging or segmented linear regressions is an
+ excellent idea.
+- Last, *thanks for giving me the opportunity* to think more carefully
+ about the mathematical foundations of such tools and how they could
+ make sense or not. It actually raised a lots of questions.
+
+Questions:
+- You did not hesitate to use of *elaborate statistical tools*. Such
+ tools rely on probabilistic model and on of the interesting feature
+ is that they allow two things:
+ - Hypothesis testing
+ - Confidence interval calculation
+ Do you think it would be worth building on such features
+- You demonstrated that your clustering methodology could be applied
+ to many use cases. Can tell me if you can think about situations
+ where it would not apply.
+- 11: segmented linear regression seems more meaningful than krigging
+ here. Is it really the case ? Kriging does an interpolation and
+ maybe the nuggeting is unable to smooth things enough. But maybe the
+ different phases detected by segmented regression are not that
+ meaningful either ?
+- Reuse of previous analysis to capture better traces with a lower
+ overhead ?
+
+
+- 15: multi-dimensionnal segmented linear regression ?
+- 28: you are annoyed because time is a random variable too. There is
+ uncertainty on it, which is why the classical technique (krigging or
+ segmented regression) do not apply
+- Machine learning for pointing out to situations where correlation
+ make sense or not.
+- 39: shouldn't the compiler have been able to do such kind of
+ optimization ?
+- 49: L1-cache based sampling allowed to detect when MPI was receiving
+ a message
+
+
+Entered on [2015-07-31 ven. 10:58]
+** 2015-12 décembre
+*** 2015-12-22 mardi
+**** Programmation avec Clément: hexagone magique :Teaching:Python:
+***** Génération de permutations toute simple
+#+begin_src python :results output :exports both
+N = 5
+A = range(1,N)
+
+def generate(tab,i):
+ if i>=len(tab):
+ print(tab)
+ else:
+ for j in range(i,len(tab)):
+ tab[i],tab[j] = tab[j],tab[i]
+ generate(tab,i+1)
+ tab[i],tab[j] = tab[j],tab[i]
+
+generate(A,0)
+#+end_src
+
+#+RESULTS:
+#+begin_example
+[1, 2, 3, 4]
+[1, 2, 4, 3]
+[1, 3, 2, 4]
+[1, 3, 4, 2]
+[1, 4, 3, 2]
+[1, 4, 2, 3]
+[2, 1, 3, 4]
+[2, 1, 4, 3]
+[2, 3, 1, 4]
+[2, 3, 4, 1]
+[2, 4, 3, 1]
+[2, 4, 1, 3]
+[3, 2, 1, 4]
+[3, 2, 4, 1]
+[3, 1, 2, 4]
+[3, 1, 4, 2]
+[3, 4, 1, 2]
+[3, 4, 2, 1]
+[4, 2, 3, 1]
+[4, 2, 1, 3]
+[4, 3, 2, 1]
+[4, 3, 1, 2]
+[4, 1, 3, 2]
+[4, 1, 2, 3]
+#+end_example
+
+***** Exploration comme un bourrin
+On représente l'hexagone par un tableau numéroté comme ceci:
+#+BEGIN_EXAMPLE
+ 0 1 2
+ 3 4 5 6
+7 8 9 10 11
+ 12 13 14 15
+ 17 18 19
+#+END_EXAMPLE
+
+#+begin_src python :results output :exports both :tangle /tmp/test_bourrin.py
+def check(tab):
+ start = 0
+ for r in [3,4,5,4]:
+ if sum(tab[start:(start+r)])!=38:
+ return False
+ start = start + r
+ for t in [[2,6,11],[1,5,10,15],[0,4,9,14,18],[3,8,13,17]]:
+ if sum([tab[i] for i in t])!=38:
+ return False
+ for t in [[7,3,0],[1,4,8,12],[2,5,9,13,16],[6,10,14,17]]:
+ if sum([tab[i] for i in t])!=38:
+ return False
+ return True
+
+def generate(tab,i):
+ if i>=len(tab):
+ if check(tab):
+ print(tab)
+ else:
+ for j in range(i,len(tab)):
+ tab[i],tab[j] = tab[j],tab[i]
+ generate(tab,i+1)
+ tab[i],tab[j] = tab[j],tab[i]
+
+generate([3, 17, 18, 19, 7, 1, 11, 16, 2, 5, 6, 9, 12, 4, 8, 14, 10, 13, 15],0)
+#+end_src
+
+Bon, sauf que ça va être monstrueusement long en fait. Sur ma machine:
+#+begin_src sh :results output :exports both
+cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq
+#+end_src
+
+#+RESULTS:
+: 3300000
+
+Donc, dans l'hypothèse, ultra optimiste où je serais capable de
+vérifier une permutation par cycle d'horloge, il me faudrait:
+#+begin_src R :results output :session :exports both
+factorial(19)/3300000/24/3600/365
+#+end_src
+
+#+RESULTS:
+: [1] 1168.891
+
+Plus de 1000 ans. Bon, la loi de Moore finira par nous aider mais pas
+des masses. :)
+
+***** Génération de permutation avec coupe au plus tôt
+De façon à éliminer les branches au plus tôt, on représente l'hexagone
+par un tableau numéroté comme ceci:
+#+BEGIN_EXAMPLE
+ 0 1 2
+ 11 12 13 3
+10 17 19 14 4
+ 9 16 15 5
+ 8 7 6
+#+END_EXAMPLE
+
+Et on remarque qu'on n'a de choix de branchement que pour 0, 1, 3, 5,
+7, 9, et 12. Tous les autres sont induits par les précédents.
+
+#+begin_src python :results output :exports both :tangle /tmp/test_rapide.py
+def assign(tab, i, x):
+ if x in tab[i:len(tab)]:
+ for j in range(i,len(tab)):
+ if(tab[j]==x):
+ tab[i],tab[j] = tab[j],tab[i]
+ generate(tab,i+1)
+ tab[i],tab[j] = tab[j],tab[i]
+ return
+
+def generate(tab,i):
+ # print(i)
+ if i>=len(tab):
+ print(tab)
+ else:
+ if i in [0,1,3,5,7,9,12]:
+ for j in range(i,len(tab)):
+ tab[i],tab[j] = tab[j],tab[i]
+ generate(tab,i+1)
+ tab[i],tab[j] = tab[j],tab[i]
+ elif i in [2,4,6,8,10]:
+ x = 38 - (tab[i-1]+tab[i-2])
+ assign(tab,i,x)
+ elif i==11:
+ x = 38 - (tab[i-1]+tab[0])
+ assign(tab,i,x)
+ elif i==13:
+ x = 38 - (tab[11]+tab[12]+tab[3])
+ assign(tab,i,x)
+ elif i==14:
+ x = 38 - (tab[1]+tab[13]+tab[5])
+ assign(tab,i,x)
+ elif i==15:
+ x = 38 - (tab[3]+tab[14]+tab[7])
+ assign(tab,i,x)
+ elif i==16:
+ x = 38 - (tab[5]+tab[15]+tab[9])
+ assign(tab,i,x)
+ elif i==17:
+ x = 38 - (tab[7]+tab[16]+tab[11])
+ if x+tab[9]+tab[12]+tab[1]!=38:
+ return
+ assign(tab,i,x)
+ elif i==18:
+ if tab[10]+tab[17]+tab[18]+tab[14]+tab[4]==38 and \
+ tab[0]+tab[12]+tab[18]+tab[15]+tab[6]==38 and \
+ tab[2]+tab[13]+tab[18]+tab[16]+tab[8]==38:
+ generate(tab,i+1)
+
+generate(range(1,20),0)
+#+end_src
+
+
+Et maintenant, combien de temps pour avoir la solution ?
+#+begin_src sh :results output :exports both
+cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
+cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq
+time python /tmp/test_rapide.py 2>&1
+#+end_src
+
+#+RESULTS:
+#+begin_example
+performance
+3300000
+[3, 17, 18, 11, 9, 14, 15, 13, 10, 12, 16, 19, 7, 1, 6, 8, 4, 2, 5]
+[3, 19, 16, 12, 10, 13, 15, 14, 9, 11, 18, 17, 7, 2, 4, 8, 6, 1, 5]
+[9, 11, 18, 17, 3, 19, 16, 12, 10, 13, 15, 14, 6, 1, 7, 2, 4, 8, 5]
+[9, 14, 15, 13, 10, 12, 16, 19, 3, 17, 18, 11, 6, 8, 4, 2, 7, 1, 5]
+[10, 12, 16, 19, 3, 17, 18, 11, 9, 14, 15, 13, 4, 2, 7, 1, 6, 8, 5]
+[10, 13, 15, 14, 9, 11, 18, 17, 3, 19, 16, 12, 4, 8, 6, 1, 7, 2, 5]
+[15, 13, 10, 12, 16, 19, 3, 17, 18, 11, 9, 14, 8, 4, 2, 7, 1, 6, 5]
+[15, 14, 9, 11, 18, 17, 3, 19, 16, 12, 10, 13, 8, 6, 1, 7, 2, 4, 5]
+[16, 12, 10, 13, 15, 14, 9, 11, 18, 17, 3, 19, 2, 4, 8, 6, 1, 7, 5]
+[16, 19, 3, 17, 18, 11, 9, 14, 15, 13, 10, 12, 2, 7, 1, 6, 8, 4, 5]
+[18, 11, 9, 14, 15, 13, 10, 12, 16, 19, 3, 17, 1, 6, 8, 4, 2, 7, 5]
+[18, 17, 3, 19, 16, 12, 10, 13, 15, 14, 9, 11, 1, 7, 2, 4, 8, 6, 5]
+1.25user 0.00system 0:01.26elapsed 99%CPU (0avgtext+0avgdata 6668maxresident)k
+0inputs+0outputs (0major+873minor)pagefaults 0swaps
+#+end_example
+
+***** Solution par d'autres personnes
+Finalement, un peu de google nous donne ça:
+
+http://codegolf.stackexchange.com/questions/6304/code-solution-for-the-magic-hexagon
+
+La solution en C++ est fondamentalement la même mais sans utilisation
+de récursion, i.e. en inlinant les 9 boucles et avec une macro pour
+aléger l'écriture. Elle trouve les mêmes solutions que nous mais 60
+fois plus vite:
+
+#+begin_src cpp :results output :exports both :tangle /tmp/test_cpp.cpp
+#include
+#define LOOP(V) for(int V=1;V<20;V++){if(m&1<&1
+#+end_src
+
+#+RESULTS:
+#+begin_example
+3 17 18 19 7 1 11 16 2 5 6 9 12 4 8 14 10 13 15
+3 19 16 17 7 2 12 18 1 5 4 10 11 6 8 13 9 14 15
+9 11 18 14 6 1 17 15 8 5 7 3 13 4 2 19 10 12 16
+9 14 15 11 6 8 13 18 1 5 4 10 17 7 2 12 3 19 16
+10 12 16 13 4 2 19 15 8 5 7 3 14 6 1 17 9 11 18
+10 13 15 12 4 8 14 16 2 5 6 9 19 7 1 11 3 17 18
+15 13 10 14 8 4 12 9 6 5 2 16 11 1 7 19 18 17 3
+15 14 9 13 8 6 11 10 4 5 1 18 12 2 7 17 16 19 3
+16 12 10 19 2 4 13 3 7 5 8 15 17 1 6 14 18 11 9
+16 19 3 12 2 7 17 10 4 5 1 18 13 8 6 11 15 14 9
+18 11 9 17 1 6 14 3 7 5 8 15 19 2 4 13 16 12 10
+18 17 3 11 1 7 19 9 6 5 2 16 14 8 4 12 15 13 10
+0.02user 0.00system 0:00.02elapsed 100%CPU (0avgtext+0avgdata 1284maxresident)k
+0inputs+0outputs (0major+64minor)pagefaults 0swaps
+#+end_example
+
+Entered on [2015-12-22 mar. 17:59]
+*** 2015-12-23 mercredi
+**** Parrot :HOME:
+Appel le 23/12/15 vers 17:35. No de dossier: 516 440
+* 2016
+** 2016-02 février
+*** 2016-02-17 mercredi
+**** Programmation pendu avec Rémi :Teaching:Python:ATTACH:
+:PROPERTIES:
+:Attachments: lst.txt
+:ID: 0032c718-137f-464c-ab82-5cd3b378a222
+:END:
+***** Code
+#+begin_src python :results output :exports both :tangle /tmp/pendu.py
+from random import *
+from sys import stdin
+
+def valide(mot):
+ for l in mot:
+ ok = 0
+ for lp in ["a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z","-"]:
+ if(l==lp): ok=1
+ if(ok==0): return 0
+ return 1
+
+def lit_dictionnaire(nom_fichier):
+ f = open(nom_fichier, 'r')
+ L = []
+ for line in f:
+ mot = line.rstrip()
+ if(valide(mot)):
+ L.append(mot)
+
+ # print "Il y a " + str(len(L)) + " mots dans mon dictionnaire."
+ # print "Le premier est '" + L[0] + "'."
+ # print "Le dixieme est '" + L[9] + "'."
+ return L
+
+def choisit_mot(dico):
+ return dico[int(len(dico)*random())]
+
+def trouve(lettre,lettres_autorisees):
+ for l in lettres_autorisees:
+ if(lettre==l):
+ return 1
+ return 0
+
+def enleve(l,L):
+ for i in range(0,len(L)):
+ if(l==L[i]):
+ return(L[:i]+L[(i+1):])
+
+def lit_lettre(lettres_autorisees):
+ print(lettres_autorisees)
+ input = stdin.readline().rstrip()
+ while (len(input)!=1 or (trouve(input,lettres_autorisees)!=1) ):
+ print trouve(input,lettres_autorisees)
+ print "Imbecile! Donne moi UNE lettre et qui soit autorisee!"
+ input = stdin.readline().rstrip()
+ return input
+
+def remplace(mot_joueur, l, mot):
+ # print ">>> ("+mot_joueur+","+l+","+mot+")"
+ a_trouve = 0
+ for i in range(0,len(mot)):
+ if(mot[i] == l):
+ # print "Youpi!!! j'ai trouve: "+mot[i]
+ # print " "+mot_joueur
+ mot_joueur = mot_joueur[:i] + l + mot_joueur[(i+1):]
+ a_trouve = 1
+ # print " "+mot_joueur
+ return (mot_joueur,a_trouve)
+
+
+def motif_ok(mot,motif):
+ if(len(mot)!=len(motif)):
+ return 0
+ for i in range(0,len(motif)):
+ if(motif[i]!="#"):
+ if(mot[i]!=motif[i]):
+ return 0;
+ return 1
+
+def lettres_exclues_ok(mot,lettres_exclues):
+ for l in mot:
+ for le in lettres_exclues:
+ if l==le:
+ return 0
+ return 1
+
+def filtre(dictionnaire,motif,lettres_exclues):
+ nouveau_dico = []
+ for mot in dictionnaire:
+ if(motif_ok(mot,motif) and lettres_exclues_ok(mot, lettres_exclues)):
+ nouveau_dico.append(mot)
+ return nouveau_dico
+
+def conseille_stupide(mots_possibles, lettres_possibles):
+ return lettres_possibles[0]
+
+def lettre_dans_mot(l,mot):
+ for lm in mot:
+ if(lm==l):
+ return 1
+ return 0
+
+def conseille(mots_possibles, lettres_possibles):
+ nombre_mots = len(mots_possibles)
+
+ if nombre_mots==1:
+ for l in lettres_possibles:
+ if(lettre_dans_mot(l,mots_possibles[0])):
+ return l
+ def compte(l,mots_possibles):
+ num = 0
+ for mot in mots_possibles:
+ if lettre_dans_mot(l,mot)==1:
+ num=num+1
+ return num
+
+ nombre_mots_avec_la_bonne_lettre = []
+ score = []
+ for l in lettres_possibles:
+ num = compte(l,mots_possibles)
+ nombre_mots_avec_la_bonne_lettre.append(num)
+ score.append(abs(num-nombre_mots/2.0))
+
+ score_min = score[0]+.1
+ i_min = 0
+ for i in range(0,len(lettres_possibles)):
+ if(score[i]freq_max):
+ freq_max = frequence_lettre[i]
+ i_max = i
+ # print lettres_possibles
+ # print nombre_mots
+ # print nombre_mots_avec_la_bonne_lettre
+ # print score
+ # print i_min
+ return lettres_possibles[i_max]
+
+def jeu(dictionnaire,mot,mode):
+ mot_joueur = "#" * len(mot)
+ for i in range(0,len(mot)):
+ if mot[i]=="-": mot_joueur = mot_joueur[:i] + "-" + mot_joueur[(i+1):]
+
+ lettres_autorisees = ["a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z"]
+ lettres_exclues = []
+ max_erreur=18
+ erreur=0
+ mots_possibles=dictionnaire
+
+ # print mot
+
+ while(mot_joueur != mot):
+ if mode=="interactif": print mot_joueur + " | Nombre d'erreurs autorisees restant : " + str(max_erreur-erreur)
+ mots_possibles = filtre(mots_possibles,mot_joueur,lettres_exclues)
+ if mode=="interactif": print "Il reste " + str(len(mots_possibles)) + " mot(s)"
+ if mode=="interactif": lettre_conseillee = conseille(mots_possibles,lettres_autorisees)
+ if mode=="frequence": lettre_conseillee = conseille_freq(mots_possibles,lettres_autorisees)
+ if mode=="dichotomie": lettre_conseillee = conseille(mots_possibles,lettres_autorisees)
+
+ if mode=="interactif": print " Conseil: " + lettre_conseillee
+ if mode=="interactif": lettre = lit_lettre(lettres_autorisees)
+ else: lettre=lettre_conseillee
+ (mot_joueur,a_trouve) = remplace(mot_joueur, lettre, mot)
+ lettres_autorisees = enleve(lettre,lettres_autorisees)
+ if(a_trouve==0):
+ erreur += 1
+ lettres_exclues.append(lettre)
+ if(erreur==max_erreur):
+ if mode=="interactif": print "Tu as perdu!!!!"
+ if mode=="interactif": print "C'etait : " + mot
+ return erreur
+ if mode=="interactif": print mot_joueur + " | Nombre d'erreurs autorisees restant : " + str(max_erreur-erreur)
+ if mode=="interactif": print "Bravo!!!!"
+ return erreur
+
+
+def main():
+ mon_dico = lit_dictionnaire("/home/alegrand/Hacking/boggle/Words.txt");
+ while(1):
+ mot = choisit_mot(mon_dico);
+ jeu(mon_dico,mot,"interactif")
+
+def main2():
+ mon_dico = lit_dictionnaire("/home/alegrand/Hacking/boggle/Words.txt");
+ while(1):
+ mot = choisit_mot(mon_dico);
+ freq = jeu(mon_dico,mot,"frequence")
+ dicho = jeu(mon_dico,mot,"dichotomie")
+ print mot + " , " + str(freq) + " , " + str(dicho)
+
+main2()
+### Quelques constructions equivalentes
+
+# i=0
+# while(i<10):
+# print i
+# i=i+1
+#
+# for i in range(0,10):
+# print i
+
+# for i in range(0,len(liste)):
+# print liste[i]
+#
+# for mot in liste:
+# print mot
+
+#+end_src
+
+J'exécute et j'arrête au boût de 3 minutes
+#+begin_src sh :results output :exports both
+python pendu.py > lst.txt
+#+end_src
+
+#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 400 :height 400 :session
+library(ggplot2)
+df=read.csv("data/00/32c718-137f-464c-ab82-5cd3b378a222/lst.txt",strip.white=T,header=F)
+names(df)=c("mot","freq","dicho")
+ggplot(data=df,aes(x=freq,y=dicho)) + geom_point(alpha=.3)
+#+end_src
+
+#+RESULTS:
+[[file:/tmp/babel-9398SP8/figure9398caH.png]]
+
+
+#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 400 :height 400 :session
+ggplot(data=df,aes(x=freq-dicho)) + geom_histogram()
+#+end_src
+
+#+RESULTS:
+[[file:/tmp/babel-9398SP8/figure9398QDg.png]]
+
+
+#+begin_src R :results output :session :exports both
+summary(df)
+#+end_src
+
+#+RESULTS:
+: mot freq dicho
+: bionique : 2 Min. :0.000 Min. : 0.000
+: crevassait: 2 1st Qu.:0.000 1st Qu.: 1.000
+: pin : 2 Median :1.000 Median : 3.000
+: primevere : 2 Mean :1.296 Mean : 3.087
+: terrien : 2 3rd Qu.:2.000 3rd Qu.: 4.000
+: abattement: 1 Max. :8.000 Max. :11.000
+: (Other) :836
+
+#+begin_src R :results output :session :exports both
+X=df$freq-df$dicho
+summary(X)
+mean(X)
+err = sd(X)/sqrt(length(X))
+mean(X) - 2*err
+mean(X) + 2*err
+#+end_src
+
+#+RESULTS:
+: Min. 1st Qu. Median Mean 3rd Qu. Max.
+: -10.000 -3.000 -1.000 -1.791 0.000 7.000
+: [1] 2.304793
+: [1] 0.07919362
+***** Links
+http://web.stanford.edu/class/cs106l/handouts/assignment-2-evil-hangman.pdf
+http://www.sharkfeeder.com/hangman/
+http://blog.wolfram.com/2010/08/13/25-best-hangman-words/
+Entered on [2016-02-17 mer. 10:14]
+*** 2016-02-25 jeudi
+**** Hacking screenkey :Python:
+#+begin_src sh :session foo :results output :exports both
+diff -u /usr/share/pyshared/Screenkey/listenkdb.py_old /usr/share/pyshared/Screenkey/listenkdb.py
+#+end_src
+
+#+RESULTS:
+#+begin_example
+--- /usr/share/pyshared/Screenkey/listenkdb.py_old 2016-03-07 09:40:13.271193249 +0100
++++ /usr/share/pyshared/Screenkey/listenkdb.py 2016-03-07 09:42:41.216862924 +0100
+@@ -230,7 +230,14 @@
+ mod = mod + "Alt+"
+ if self.cmd_keys['super']:
+ mod = mod + "Super+"
+-
++
++ if self.cmd_keys['shift']:
++ if (len(key_shift)==1) and not(ord(key_normal) in range(97,123)) and not(ord(key_shift) in range(33,126)):
++ mod = mod + "Shift+"
+65000):
++ mod = mod + "Shift+"
+print "---------"
+print key, key_shift, keysym
+ if self.cmd_keys['shift']:
+ key = key_shift
+ if self.cmd_keys['capslock'] \
+#+end_example
+
+Entered on [2016-02-25 jeu. 11:19]
+
+** 2016-07 juillet
+*** 2016-07-19 mardi
+**** [[http://rmarkdown.rstudio.com/flexdashboard/][flexdashboard: Easy interactive dashboards for R]] :WP7:WP8:R:twitter:
+
+Entered on [2016-07-19 mar. 09:04]
+**** Steps toward reproducible research (Karl Broman) :WP8:twitter:R:
+- https://github.com/kbroman/Talk_ReproRes (slides)
+ - Why
+ - I'm sorry but I think you haven't used the right data.
+ - The results in Table 1 don’t seem to correspond to those in
+ Figure 2.
+ - In what order do I run these scripts?
+ - Where did we get this data file?
+ - Why did I omit those samples?
+ - How did I make that figure?
+ - Important points
+ - Organize your data & code
+ #+BEGIN_QUOTE
+ Your closest collaborator is you six months ago,
+ but you don’t reply to emails.
+ (paraphrasing Mark Holder)
+ #+END_QUOTE
+ - Everything with a script
+ If you do something once, you’ll do it 1000 times.
+ - Automate the process as much as you can
+ In addition to automating a complex process, it also documents
+ the process, including the dependencies among data files and
+ scripts.
+ - Turn scripts into reproducible reports
+ - Use version control (git/GitHub)
+ #+BEGIN_QUOTE
+ The most important tool is the mindset,
+ when starting, that the end product
+ will be reproducible.
+ – Keith Baggerly
+ #+END_QUOTE
+
+- [[http://kbroman.org/steps2rr/][initial steps toward reproducible research]] (lecture)
+- https://github.com/kbroman/Tools4RR (Materials for a one-credit
+ course on reproducible research)
+
+Entered on [2016-07-19 mar. 09:06]
+**** [[http://michaellevy.name/blog/teaching-r-to-200-students-in-a-week/][Teaching R to 200 students in a week • Michael Levy]] :Teaching:R:
+Link from Michael Blum
+- Motivation precedes detail: “Here’s what you’re going to learn to do this week”
+- Live coding: shows that I make mistakes, builds in flexibility, forces to slow down
+- Live code piped to their browsers (dropbox? pad? floobits?)
+- In-class exercises instead of lectures
+- Stickies and good assistants
+- Daily feedback: Each day, I asked the students to fill out a quick
+ survey: How well do you understand what was taught today, what’s
+ working for you, and what could use a change?
+- Advanced exercises for experts.
+
+Entered on [2016-07-19 mar. 09:36]
+ CEDEX 9
+* 2017
+** 2017-01 janvier
+*** 2017-01-09 lundi
+**** Ridge regression :Stats:R:
+http://web.as.uky.edu/statistics/users/pbreheny/764-F11/notes/9-1.pdf
+
+Ridge regression penalizes the size of the regression coefficients,
+which is convenient in the presence of multicollinearity
+
+- http://www.few.vu.nl/~wvanwie/Courses/HighdimensionalDataAnalysis/WNvanWieringen_HDDA_Lecture4_RidgeRegression_20162017.pdf
+- https://arxiv.org/pdf/1509.09169.pdf
+
+Entered on[2017-01-09 lun. 22:10]
+
+*** 2017-01-19 jeudi
+**** Groupe de lecture: [[file:public_html/readings/Jackson_Networks.pdf][Social and Economic Networks]] (session 1) :POLARIS:
+***** Chapter 1: Introduction
+- Example from the Medici graph
+- degree can be seen as a measure of the influence of the node but a
+ probably more interesting notion is the betweenness which indicates
+ the probability the path a pair of peole have to go through you to
+ communicate:
+
+ b(k) = 1/((n-1)(n-2)/2) \sum_{i\neq j} (number of shortest paths from i to j going through
+ k)/(number ofr shortest paths from i to j)
+***** Chapter 2: Basic notions
+- notion of adjacency matrix with convenient properties.
+- deg(i) = \sum_j g_{i,j}
+- #triangles = tr(g^3)/6
+- Clustering: "how close are you from a clique"
+ - local notion of connectivity
+ - Cl_i(G) = (#of pairs j,k connected to i s.t. j and k are also
+ connected )/(d(i).(d(i)-1)/2)
+ = \sum_{j,k s.t. i\ne j\ne k} g_{i,j} g_{j,k} g_{k,i} / \sum_{j,k s.t. i\ne j\ne k} g_{i,j} g_{i,k}
+ \approx (sum ith diag element of g^3) / (sum ith diag element of g^2)
+ - Cl(G) = 1/N \sum_i Cl_i(G) is then the /average clustering/ of the graph
+ if you average over nodes. If one consider a "star" of small
+ "cliques", all the nodes but the one in the center will have a
+ local clique of 1 (hence the average clustering tends to 1)
+ whereas it is far of being a clique itself.
+ - You could average over triples directly and get the /overall
+ clustering/ by considering directly:
+ \sum_{i\ne j\ne k} g_{i,j} g_{j,k} g_{k,i} / \sum_{i\ne j\ne k} g_{i,j} g_{i,k}
+ This may be a better notion as the overall clustering would go to
+ 0... But we could totally have the reverse...
+- Centrality:
+ - degree centrality is the average degree
+ - betweeness centrality is what we saw earlier
+ - Eigen centrality. Relates to a notion of prestige P:
+
+ P_i (g) = \sum_{j\ne i} g_{i,j} P_j(g)/\delta_g(j)
+
+ Hence P = \bar{g}.P with \bar{g} being the normalized graph (that
+ could be different if the graph is undirected or if one does not
+ consider \delta to be the degree d). Note that the Degree satisfies
+ such equation, hence you recove the degree centrality notion.
+ - In general, one may consider the larger eigenvalue of \bar{g} and
+ the Perron Frobenius tells that all components of the
+ corresponding eigenvector are positive. One may thus normalize by
+ the sum of the elements of the eigenvector P.
+- Another notion is distance or decay centrality:
+ - \sum_j 1/d(i,j
+- We don't really know whether there is a link between
+ Eigen-centrality and Betweenness centrality.
+**** MOOC, Jean-Marc Hasenfratz :Vulgarization:Teaching:WP8:
+Discussions sur l'opportunité de faire un MOOC Recherche reproductible
+sur FUN. On fait le tour des choses existantes:
+- Il y a des choses en stats, sur R
+ https://www.fun-mooc.fr/cours/#filter/subject/mathematiques-et-statistiques?page=1&rpp=50
+ mais rien sur les aspects recherche reproductible ou la
+ programmation littérale en tant que tel.
+- Sur coursera, il y a
+ https://www.coursera.org/specializations/jhu-data-science. Il me
+ semble que les 5 premiers (intro/techno, R basis, data cleaning,
+ EDA, litterate programming) valent le coup mais c'est trop long.
+- Lorena Barba's tutorial on RR ?
+ https://barbagroup.github.io/essential_skills_RRC/
+- Si on reste sur l'idée que le "cahier de laboratoire" c'est la base,
+ il faut décider de quelle techno on pousse: knitr, jupyter ou
+ org-mode. D'où une discussion prévue avec Konrad et Christophe, ce
+ qui permettrait en plus de donnée une dimension pluri-disciplinaire
+ intéressante.
+
+Entered on [2017-01-19 jeu. 14:29]
+
+** 2017-12 décembre
+*** 2017-12-07 jeudi
+**** Refs Recherche reproductible intéressantes :WP8:
+[[https://www.jove.com/blog/2017/10/27/reproducibility-librarian-yes-that-should-be-your-next-job/][Reproducible research librarian]]
+
+- [[http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0038234][The Effects of FreeSurfer Version, Workstation Type, and Macintosh
+ Operating System Version on Anatomical Volume and Cortical Thickness
+ Measurements]]:
+ - No differences were detected between repeated single runs nor
+ between single runs and parallel runs on the same workstation and
+ for the same FreeSurfer and OS version. For the same OS version,
+ all Mac workstations produced identical results. However,
+ differences were revealed between:
+ - Mac and HP workstations
+ - FreeSurfer versions v4.3.1, v4.5.0, and v5.0.0
+ - OSX 10.5.8 and OSX 10.6.4/5
+ - Focus sur l'importance des erreurs mais pas sur leur origine car
+ tout est très closed-source.
+- [[https://genomebiology.biomedcentral.com/articles/10.1186/s13059-016-1044-7][Gene name errors are widespread in the scientific literature]]
+- [[https://genomebiology.biomedcentral.com/articles/10.1186/s13059-015-0850-7][Five selfish reasons to work reproducibly]]: style sympa
+- Jupyter extension with Reprozip: https://www.youtube.com/watch?v=Y8YmGVYHhS8
+- [[https://github.com/pantsbuild/pex][Pex]]: un environnement permettant de transformer un script python en
+ un exécutable autonome (packing interne des libs).
+- [[http://o2r.info/][O2R]]: hyper préliminaire, aucun intérêt.
+
+Entered on [2017-12-07 jeu. 11:09]
+
+ [[file:~/org/journal.org::*Autotuning%20context:][Autotuning context:]]
+**** [[file:~/Archives/Cours/maths/R/Verzani-SimpleR.pdf][simpleR – Using R for Introductory StatisticsR]] :Stats:R:
+
+Un document que je n'avais pas pris le temps de lire mais qui est pas
+mal fichu du tout. Points intéressants sur R:
+
+- Trouver des outliers en interagissant avec les plots:
+ =identify(BUSH,BUCHANAN,n=2)=.
+- Using rlm or lqs for resistant regression. rlm minimise la somme
+ d'une fraction des résidus et pas de l'intégralité.
+- pas mal d'exemples/exos sur le test (paired, avec même variance ou
+ pas, Cox-Wilkinson, chi2, etc.)
+
+Entered on [2017-12-07 jeu. 12:28]
+
+ [[file:~/org/journal.org::*Refs%20Recherche%20reproductible%20int%C3%A9ressantes][Refs Recherche reproductible intéressantes]]
+* 2018
+** 2018-10 octobre
+*** 2018-10-02 mardi
+**** Learning emacs lisp
+https://www.gnu.org/software/emacs/manual/html_mono/elisp.html
+
+file:///home/alegrand/tmp/Programming%20in%20Emacs%20Lisp%20-%20https:_www.gnu.org_software_emacs_manual_html_mono_eintr.html
+
+Also eval some emacs lisp using M-:
+***** Evaluation
+#+begin_src emacs-lisp
+(+ 2 2)
+#+end_src
+
+#+RESULTS:
+: 4
+
+#+begin_src emacs-lisp
+'(this is a quoted list) ;; a list
+#+end_src
+
+#+begin_src emacs-lisp :
+(this is a quoted list) ;; won't work as it will call "this" with args "is" "a" "quoted" "list"
+#+end_src
+***** Setting variables
+#+begin_src emacs-lisp
+(setq toto 2)
+toto
+#+end_src
+
+#+RESULTS:
+: 2
+
+
+#+begin_src emacs-lisp
+(setq toto 2)
+(set 'toto' 2)
+toto
+#+end_src
+
+#+RESULTS:
+: 2
+
+#+begin_src emacs-lisp
+(setq counter 0) ; Let's call this the initializer.
+(setq counter (+ counter 1)) ; This is the incrementer.
+counter ; This is the counter.
+#+end_src
+
+#+RESULTS:
+: 1
+
+#+begin_src emacs-lisp
+(let ((var1 2)
+ (var2 3))
+ (+ var1 var2))
+#+end_src
+
+#+RESULTS:
+: 5
+
+#+begin_src emacs-lisp
+ (let ((zebra "stripes")
+ (tiger "fierce"))
+ (message "One kind of animal has %s and another is %s."
+ zebra tiger))
+#+end_src
+
+#+RESULTS:
+: One kind of animal has stripes and another is fierce.
+
+***** Defining and calling function
+#+begin_src emacs-lisp
+(defun tutu() 2)
+(defun tutu() '(2))
+(tutu)
+#+end_src
+
+#+RESULTS:
+| 2 |
+
+#+begin_src emacs-lisp
+(functionp 'tutu)
+#+end_src
+
+#+RESULTS:
+: t
+
+***** Testing
+#+begin_src emacs-lisp
+(if (functionp 'tutu) (message "this is a function"))
+#+end_src
+
+#+RESULTS:
+: this is a function
+
+#+begin_src emacs-lisp
+(defun type-of-animal (characteristic)
+ "Print message in echo area depending on CHARACTERISTIC.
+ If the CHARACTERISTIC is the string \"fierce\",
+ then warn of a tiger."
+ (if (equal characteristic "fierce")
+ (message "It is a tiger!")))
+(type-of-animal "fierce")
+;; (type-of-animal "striped")
+#+end_src
+
+#+RESULTS:
+: It is a tiger!
+
+#+begin_src emacs-lisp
+(if (> 4 5) ; if-part
+ (message "4 falsely greater than 5!") ; then-part
+ (message "4 is not greater than 5!")) ; else-part
+#+end_src
+
+#+RESULTS:
+: 4 is not greater than 5!
+
+***** Useful functions
+#+begin_src emacs-lisp
+;; describe-function
+;; describe-key
+;; list-matching-lines
+;; delete-window
+;; point-to-register
+;; eval-expression
+;; car, cdr, cons
+#+end_src
+***** Playing with babel templates (1)
+#+begin_src emacs-lisp
+;; (add-to-list 'org-structure-template-alist
+;; '("Y" "#+begin_src R\n?\n#+end_src"))
+;; (add-to-list 'org-structure-template-alist
+;; '("Y" '(tutu)))
+(setq tata "2")
+(add-to-list 'org-structure-template-alist
+ '("Y" tata))
+#+end_src
+
+#+RESULTS:
+| Y | tata |
+| Y | toto |
+| Y | (quote (tutu)) |
+| Y | (tutu) |
+| Y | tutu |
+| Y | #+begin_src R |
+
+#+begin_src emacs-lisp
+(setq a (assoc "Y" org-structure-template-alist))
+a
+#+end_src
+
+#+RESULTS:
+| Y | tata |
+
+Unfortunately, when expending, the code checks whether the right value
+is a string (through th stringp function).
+
+[[file:~/Work/org-mode/lisp/org.el::(defun%20org-try-structure-completion%20()][This is where org-structure-template-alist is used in org-mode 9.0.5's code]].
+
+#+begin_src emacs-lisp
+(defun org-try-structure-completion ()
+ "Try to complete a structure template before point.
+This looks for strings like \" @file Alya.f90
+ !! @author Guillaume Houzeaux
+ !! @brief Ayla main
+@@ -20,7 +21,13 @@
+ use def_master, only : kfl_goblk
+ use def_master, only : kfl_gocou
+ use def_coupli, only : kfl_gozon
++ use mod_parall, only : PAR_MY_WORLD_RANK
+ implicit none
++ INTEGER :: iter,ierror
++ character*100 striter
++ character*100 iterfile
++ real :: tnow
++ real, dimension(2) :: tarray
+ !
+ ! DLB should be disabled as we only wabnt to activate it for particular loops
+ ! Master does not disble to lend its resources automatically
+@@ -39,6 +46,10 @@
+
+ call Parall(22270_ip)
+
++ write(iterfile,'(a,i4.4,a)') 'iterations-', PAR_MY_WORLD_RANK, '.csv'
++ open(unit=2, file=iterfile)
++
++ iter = 1
+ optimization: do while ( kfl_goopt == 1 )
+
+ call Iniunk()
+@@ -49,6 +60,11 @@
+ time: do while ( kfl_gotim == 1 )
+
+ call Timste()
++ call ETIME(tarray, tnow)
++ write(2,*) tnow, tarray(1), tarray(2), PAR_MY_WORLD_RANK, iter
++
++ write(striter, '(a,i3.3)') 'iter',iter
++ SCOREP_USER_REGION_BY_NAME_BEGIN(striter, SCOREP_USER_REGION_TYPE_PHASE)
+
+ reset: do
+ call Begste()
+@@ -77,6 +93,13 @@
+
+ call Endste()
+
++ SCOREP_USER_REGION_BY_NAME_END(striter)
++ call ETIME(tarray, tnow)
++ write(2,*) tnow, tarray(1), tarray(2), PAR_MY_WORLD_RANK, iter
++ iter = iter + 1
++
++
++
+ call Filter(ITASK_ENDTIM)
+ call Output(ITASK_ENDTIM)
+
+@@ -91,6 +114,9 @@
+
+ end do optimization
+
++ close(2)
++
+ call Turnof()
+
++
+ end program Alya
+Index: Sources/services/parall/par_prepro.f90
+===================================================================
+--- Sources/services/parall/par_prepro.f90 (revision 9154)
++++ Sources/services/parall/par_prepro.f90 (working copy)
+@@ -23,6 +23,7 @@
+ use mod_redistribute, only : redistribute
+ use mod_redistribute, only : gather_to_master
+ use mod_parall, only : PAR_WORLD_SIZE
++ use mod_parall, only : PAR_MY_WORLD_RANK
+ use mod_parall, only : PAR_METIS4
+ use mod_parall, only : PAR_SFC
+ use mod_parall, only : PAR_ORIENTED_BIN
+@@ -39,6 +40,13 @@
+ integer(ip) :: npoin_tmp, nelem_tmp, nboun_tmp,ndims
+ real(rp) :: time1,time2,time3,time4,time5
+ character(100) :: messa_integ
++ ! LUCAS
++ character*100 dfile
++ integer(ip) ii
++ ! END LUCAS
++
++
++
+ !
+ ! Intermediate variables for partitioning
+ !
+@@ -384,4 +392,19 @@
+ end if
+
+
++ ! LUCAS
++ write(dfile,'(a,i4.4,a)') 'domain-', PAR_MY_WORLD_RANK, '.csv'
++ open(unit=12345, file=dfile)
++
++ write(12345,*) "T1", PAR_MY_WORLD_RANK, nelem, npoin, nboun, npoi3-npoi2
++ do ii = 1, nelem
++ write(12345,*) "T2", PAR_MY_WORLD_RANK, ii, ltype(ii)
++ end do
++ do ii = 1, nboun
++ write(12345,*) "T3", PAR_MY_WORLD_RANK, ii, ltypb(ii)
++ end do
++
++ close(12345)
++ ! END LUCAS
++
+ end subroutine par_prepro
+#+end_example
+
+**** Compilation
+
+In the =./ThirdParties/metis-4.0/=, compile Metis:
+
+#+begin_src shell :results output
+cd ./ThirdParties/metis-4.0/
+make
+#+end_src
+
+In the =./Executables/unix/= Alya directory, do.
+
+Copy =./configure.in/config_gfortran.in= to config.in.
+
+Make it look like this (in the begginning):
+
+#+BEGIN_EXAMPLE
+SCOREP = ~/install/scorep-4.0/bin/scorep
+MPIF90 = ~//spack-ALYA/opt/spack/linux-debian9-x86_64/gcc-6.3.0/openmpi-3.0.0-a7g33v4ulwtb4g2verliyelvtifybrq3/bin/mpif90
+MPICC = ~//spack-ALYA/opt/spack/linux-debian9-x86_64/gcc-6.3.0/openmpi-3.0.0-a7g33v4ulwtb4g2verliyelvtifybrq3/bin/mpicc
+F77 = $(SCOREP) --user --nocompiler --nopomp --noopenmp $(MPIF90) -cpp
+F90 = $(SCOREP) --user --nocompiler --nopomp --noopenmp $(MPIF90) -cpp
+FCOCC = $(SCOREP) --user --nocompiler --nopomp --noopenmp $(MPICC) -c
+#+END_EXAMPLE
+
+Then, configure the compilation
+
+#+begin_src shell :results output
+./configure nastin parall
+make -j 48
+#+end_src
+
+The compilation fails with
+
+#+BEGIN_EXAMPLE
+/gpfs/home/bsc21/bsc21835/alya-bsc-sfc/Sources/kernel/coupli/mod_commdom_alya.f90:489:2:
+
+ CPLNG%sendrecv(1,6) = (current_task==ITASK_BEGSTE).and.( current_when==ITASK_BEFORE).and.& ! \
+ 1
+Error: Unclassifiable statement at (1)
+#+END_EXAMPLE
+
+The solution is to comment those lines (and =make= again).
+
+The compilation fails with =par_prepro.f90=, just comment (or remove)
+the line that has =use mod_par_partit_sfc,=.
+
+*** R
+
+Installed in the image.
+
+** Jequi
+* Scripts
+** Get info from the platform
+*** Get machine information before doing the experiment
+#+begin_src sh :results output :tangle scripts/get_info.sh :tangle-mode (identity #o755)
+#!/bin/bash
+# Script for to get machine information before doing the experiment
+
+set +e # Don't fail fast since some information is maybe not available
+
+title="Experiment results"
+starpu_build=""
+inputfile=""
+host="$(hostname | sed 's/[0-9]*//g' | cut -d'.' -f1)"
+help_script()
+{
+ cat << EOF
+Usage: $0 [options] outputfile.org
+
+Script for to get machine information before doing the experiment
+
+OPTIONS:
+ -h Show this message
+ -t Title of the output file
+ -s Path to the StarPU installation
+ -i Input file name if doing SimGrid simulation based on input
+EOF
+}
+# Parsing options
+while getopts "t:s:i:h" opt; do
+ case $opt in
+ t)
+ title="$OPTARG"
+ ;;
+ s)
+ starpu_build="$OPTARG"
+ ;;
+ i)
+ inputfile="$OPTARG"
+ ;;
+ h)
+ help_script
+ exit 4
+ ;;
+ \?)
+ echo "Invalid option: -$OPTARG"
+ help_script
+ exit 3
+ ;;
+ esac
+done
+
+shift $((OPTIND - 1))
+filedat=$1
+if [[ $# != 1 ]]; then
+ echo 'ERROR!'
+ help_script
+ exit 2
+fi
+
+##################################################
+# Preambule of the output file
+echo "#+TITLE: $title" >> $filedat
+echo "#+DATE: $(eval date)" >> $filedat
+echo "#+AUTHOR: $(eval whoami)" >> $filedat
+echo "#+MACHINE: $(eval hostname)" >> $filedat
+echo "#+FILE: $(eval basename $filedat)" >> $filedat
+if [[ -n "$inputfile" ]];
+then
+ echo "#+INPUTFILE: $inputfile" >> $filedat
+fi
+echo " " >> $filedat
+
+##################################################
+# Collecting metadata
+echo "* MACHINE INFO:" >> $filedat
+
+echo "** PEOPLE LOGGED WHEN EXPERIMENT STARTED:" >> $filedat
+who >> $filedat
+echo "############################################" >> $filedat
+
+echo "** ENVIRONMENT VARIABLES:" >> $filedat
+env >> $filedat
+echo "############################################" >> $filedat
+
+echo "** HOSTNAME:" >> $filedat
+hostname >> $filedat
+echo "############################################" >> $filedat
+
+if [[ -n $(command -v lstopo) ]];
+then
+ echo "** MEMORY HIERARCHY:" >> $filedat
+ lstopo --of console >> $filedat
+ echo "############################################" >> $filedat
+fi
+
+if [[ -n "$starpu_build" ]];
+then
+ echo "** STARPU MACHINE DISPLAY:" >> $filedat
+ $starpu_build/bin/starpu_machine_display 1> tmp 2> /dev/null
+ cat tmp >> $filedat
+ rm -f tmp
+ echo "############################################" >> $filedat
+fi
+
+if [ -f /proc/cpuinfo ];
+then
+ echo "** CPU INFO:" >> $filedat
+ cat /proc/cpuinfo >> $filedat
+ echo "############################################" >> $filedat
+fi
+
+if [ -f /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor ];
+then
+ echo "** CPU GOVERNOR:" >> $filedat
+ ONLINECPUS=$(for CPU in $(find /sys/devices/system/cpu/ | grep cpu[0-9]*$); do [[ $(cat $CPU/online) -eq 1 ]] && echo $CPU; done | grep cpu[0-9]*$ | sed 's/.*cpu//')
+ for PU in ${ONLINECPUS}; do
+ echo -n "CPU frequency for cpu${PU}: " >> $filedat
+ cat /sys/devices/system/cpu/cpu${PU}/cpufreq/scaling_governor >> $filedat
+ done
+ echo "############################################" >> $filedat
+fi
+
+if [ -f /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq ];
+then
+ echo "** CPU FREQUENCY:" >> $filedat
+ ONLINECPUS=$(for CPU in $(find /sys/devices/system/cpu/ | grep cpu[0-9]*$); do [[ $(cat $CPU/online) -eq 1 ]] && echo $CPU; done | grep cpu[0-9]*$ | sed 's/.*cpu//')
+ for PU in ${ONLINECPUS}; do
+ echo -n "CPU frequency for cpu${PU}: " >> $filedat
+ cat /sys/devices/system/cpu/cpu${PU}/cpufreq/scaling_cur_freq >> $filedat
+ done
+ echo "############################################" >> $filedat
+fi
+
+if [ -f /usr/bin/cpufreq-info ];
+then
+ echo "** CPUFREQ_INFO" >> $filedat
+ cpufreq-info >> $filedat
+ echo "############################################" >> $filedat
+fi
+
+if [ -f /usr/bin/lspci ];
+then
+ echo "** LSPCI" >> $filedat
+ lspci >> $filedat
+ echo "############################################" >> $filedat
+fi
+
+if [ -f /usr/bin/ompi_info ];
+then
+ echo "** OMPI_INFO" >> $filedat
+ ompi_info --all >> $filedat
+ echo "############################################" >> $filedat
+fi
+
+if [ -f /sbin/ifconfig ];
+then
+ echo "** IFCONFIG" >> $filedat
+ /sbin/ifconfig >> $filedat
+ echo "############################################" >> $filedat
+fi
+
+if [[ -n $(command -v nvidia-smi) ]];
+then
+ echo "** GPU INFO FROM NVIDIA-SMI:" >> $filedat
+ nvidia-smi -q >> $filedat
+ echo "############################################" >> $filedat
+fi
+
+if [ -f /proc/version ];
+then
+ echo "** LINUX AND GCC VERSIONS:" >> $filedat
+ cat /proc/version >> $filedat
+ echo "############################################" >> $filedat
+fi
+
+if [[ -n $(command -v module) ]];
+then
+ echo "** MODULES:" >> $filedat
+ module list 2>> $filedat
+ echo "############################################" >> $filedat
+fi
+#+end_src
+
+** Common Scripts
+*** Colors + error report
+#+begin_src sh :results output :tangle scripts/colors_error_report.sh :tangle-mode (identity #o755)
+#!/bin/bash
+set -euo pipefail
+
+# Error function
+function error_script {
+ ERRORMSG=${1:-}
+ echo "Error: $ERRORMSG"
+ exit 1
+}
+
+# Check functions
+function check_var {
+ VAR=${1:-}
+ MSG=${2:-}
+ if [[ -z "$VAR" ]]; then
+ error_script $MSG
+ fi
+}
+
+function check_bin {
+ BIN=${1:-}
+ MSG=${2:-}
+ if [[ ! -x "$BIN" ]]; then
+ error_script $MSG
+ fi
+}
+
+function check_env {
+ ENV=${1:-}
+ MSG=${2:-}
+ if [ -z ${ENV+x} ]; then
+ error_script $MSG
+ fi
+}
+
+function info {
+ INFOMSG=${1:-}
+ echo "Info: $INFOMSG"
+}
+#+end_src
+
+*** Create experimental directory (only functions)
+#+begin_src sh :results output :tangle scripts/create_experiment_dir.sh :tangle-mode (identity #o755)
+#!/bin/bash
+
+# fail often, so we are robust
+set -euo pipefail
+
+# source functions
+DIR=$(dirname $0)
+source $DIR/colors_error_report.sh
+
+# Function to cat a file within an example block
+function org_example_file {
+ FILE=${1:-}
+ if [ -e "$FILE" ]; then
+ echo "#+BEGIN_EXAMPLE"
+ cat $FILE
+ echo "#+END_EXAMPLE"
+ fi
+}
+
+# Function that returns the unique org filename given an experiment unique identifier
+function org_err_file {
+ UNIQUE=${1:-}
+ if [ -z "$UNIQUE" ]; then
+ error_script "Error: Unique key not passed as parameter to org_err_file."
+ fi
+ echo $(basename $UNIQUE .org)_err.org
+}
+
+# Function that returns the unique org filename given an experiment unique identifier
+function org_file {
+ UNIQUE=${1:-}
+ if [ -z "$UNIQUE" ]; then
+ error_script "Error: Unique key not passed as parameter to org_file."
+ fi
+ echo $(basename $UNIQUE .org).org
+}
+
+function org_info_file {
+ UNIQUE=${1:-}
+ if [ -z "$UNIQUE" ]; then
+ error_script "Error: Unique key not passed as parameter to org_file."
+ fi
+ echo $(basename $UNIQUE .org)_get_info.org
+}
+
+# Function that returns the unique directory name given an experiment unique identifier
+function dir_file {
+ UNIQUE=${1:-}
+ if [ -z "$UNIQUE" ]; then
+ error_script "Error: Unique key not passed as parameter to dir_file."
+ fi
+ echo $(basename $UNIQUE .org).dir
+}
+
+function create_experiment_dir {
+ UNIQUE=${1:-}
+
+ if [ -z "$UNIQUE" ]; then
+ error_script "Error: Unique key not passed as parameter to create_experiment_directory."
+ fi
+
+ # Preparation of the output (org and dir)
+ OUTPUTORG=$(org_file $UNIQUE)
+ OUTPUTORGDIR=$(dir_file $UNIQUE)
+ if [ -e $OUTPUTORG ]; then
+ error_script "$OUTPUTORG already exists. Please, remove it or use another unique name."
+ usage;
+ exit;
+ else
+ info "Experiment Org file: $OUTPUTORG"
+ fi
+ if [ -d $OUTPUTORGDIR ]; then
+ error_script "$OUTPUTORGDIR directory already exists. Please, remove it or use another unique name."
+ usage;
+ exit;
+ else
+ info "Experiment Directory: $OUTPUTORGDIR"
+ fi
+ # Everything seems okay, let's prepare the terrain
+ touch $OUTPUTORG
+ if [ $? -ne 0 ]; then
+ error_script "Creation of $OUTPUTORG did not work."
+ exit;
+ fi
+ mkdir $OUTPUTORGDIR
+ if [ $? -ne 0 ]; then
+ error_script "Creation of $OUTPUTORGDIR did not work."
+ exit;
+ fi
+}
+#+end_src
+*** Detect Hyperthreading
+Inspired by:
+http://unix.stackexchange.com/questions/33450/checking-if-hyperthreading-is-enabled-or-not/33509#33509
+#+begin_src R :results output :session :exports both :tangle scripts/hyperthreading.sh :tangle-mode (identity #o755)
+#!/bin/bash
+CPUFILE=/proc/cpuinfo
+test -f $CPUFILE || exit 1
+NUMPHYCPU=`grep "physical id" $CPUFILE | sort -u | wc -l`
+NUMLOGCORE=`grep "processor" $CPUFILE | wc -l`
+NUMPHYCORE=`grep "core id" $CPUFILE | sort -u | wc -l`
+TOTALNUMPHYCORE=$(echo "$NUMPHYCPU * $NUMPHYCORE" | bc)
+MODEL=`grep "model name" $CPUFILE | sort -u | cut -d : -f 2- | sed "s/^[[:space:]]*//"`
+echo "This system has $NUMPHYCPU CPUs, of model \"$MODEL\"."
+echo "Each physical CPU is equipped with $NUMPHYCORE physical cores (total is $TOTALNUMPHYCORE)."
+if [ $TOTALNUMPHYCORE -ne $NUMLOGCORE ]; then
+ echo "Hyperthreading is ON. So, there are $NUMLOGCORE logical cores."
+else
+ echo "Hyperthreading is OFF."
+fi
+exit
+#+end_src
+*** Disable Hyperthreading
+#+begin_src sh :results output :tangle scripts/disable_hyperthreading.sh :tangle-mode (identity #o755)
+#!/bin/bash
+DIR=$(dirname $0)
+
+#First, enable all cores
+for PU in `find /sys/devices/system/cpu/ |grep cpu[0-9]*$`; do
+ echo "Enabling $PU now."
+ sudo zsh -c "echo 1 > ${PU}/online"
+done
+
+HYPERTHREADING=`$DIR/hyperthreading.sh | grep -e "Hyperthreading is ON" | wc -l`
+if [ $HYPERTHREADING -eq 0 ]; then
+ echo "Hyperthreading is OFF, so disabling is not necessary."
+ exit
+else
+ echo "Hyperthreading is ON."
+fi
+echo "The number of PUs now is $(hwloc-ls --only PU | wc -l)."
+echo "I will disable hyperthreading now."
+# Disable hyperthreading
+# Only run this if you are sure
+# - Hyperthreading is enabled
+# - Each physical core has two processing units (PU)
+# - hwloc-ls is installed and reports two PU per physical core
+for PU in `hwloc-ls --only PU | cat -n | grep -e "[[:space:]]*[0-9]*[02468][[:space:]]*PU" | sed -e "s/^[^(]*(P#\\([0-9]*\))/\1/"`; do
+ echo "Disabling PU $PU now."
+ sudo zsh -c "echo 0 > /sys/devices/system/cpu/cpu${PU}/online"
+done
+echo "The number of PUs now is $(hwloc-ls --only PU | wc -l)."
+#+end_src
+*** Disable TurboBoost
+#+begin_src sh :results output :tangle scripts/disable_turboboost.sh :tangle-mode (identity #o755)
+#!/bin/bash
+DIR=$(dirname $0)
+
+if [ `lsmod | grep msr | wc -l` -ne 1 ]; then
+ echo "The =msr= module is not loaded. It should be."
+ exit 1;
+fi
+
+# Get the list of online cores
+ONLINECPUS=$(for CPU in $(find /sys/devices/system/cpu/ | grep -v cpu0 | grep cpu[0-9]*$); do [[ $(cat $CPU/online) -eq 1 ]] && echo $CPU; done | grep cpu[0-9]*$ | sed 's/.*cpu//')
+
+# Enable
+for PU in ${ONLINECPUS}; do
+ sudo zsh -c "/usr/sbin/wrmsr -p${PU} 0x1a0 0x850089"
+done
+
+# Disable & Check
+for PU in ${ONLINECPUS}; do
+ echo "Disabling turbo boost mode for PU $PU."
+ sudo zsh -c "/usr/sbin/wrmsr -p${PU} 0x1a0 0x4000850089"
+ TURBOBOOST=$(sudo zsh -c "/usr/sbin/rdmsr -p${PU} 0x1a0 -f 38:38")
+ if [[ "0" = $TURBOBOOST ]]; then
+ echo "Failed to disable turbo boost for PU number $cpu. Aborting."
+ exit 1
+ fi
+done
+#+end_src
+*** CPU Selection
+#+begin_src sh :results output :tangle scripts/cpu_selection.sh :tangle-mode (identity #o755)
+#!/bin/bash
+DIR=$(dirname $0)
+
+function usage()
+{
+ echo "Input: number of CPUs to be used"
+ echo "Output: core identifiers (NUMA-aware)"
+ echo "$0 ";
+}
+
+NCPUS=${1:-}
+if [ -z "$NCPUS" ]; then
+ echo "Error: is empty"
+ usage;
+ exit;
+fi
+
+# Check total number of CPUs
+NUMCPU=$($DIR/hyperthreading.sh | head -n1 | sed 's/^This system has \([0-9]*\) CPUs.*$/\1/')
+if [ $NCPUS -gt $NUMCPU ]; then
+ echo "You request $NCPUS CPUs, but this system has only $NUMCPU. Sorry."
+ exit
+fi
+
+RES=$(lscpu | grep "NUMA\ node[0-9]* " | sed "s/^.*:[[:space:]]*//" | head -n${NCPUS} | tr '\n' ' ' | sed "s/ $//")
+FINAL=$(for i in $RES; do
+ if [ $(echo $i | grep "-" | wc -l) -eq 0 ]; then
+ #just replace commas by spaces
+ echo $i | sed "s/,/ /g"
+ else
+ #use seq to replace core ranges by actual core ids
+ S=$(echo $i | sed "s/-.*$//");
+ E=$(echo $i | sed "s/^.*-//");
+ seq $S $E
+ fi
+done | tr '\n' ' ')
+echo "Use these: $FINAL"
+#+end_src
+*** Detect DVFS driver (acpi-cpufreq)
+#+begin_src sh :results output :tangle scripts/detect_acpidriver.sh :tangle-mode (identity #o755)
+#!/bin/bash
+DIR=$(dirname $0)
+
+function usage()
+{
+ echo "Input: number of CPUs to be used"
+ echo "Output: core identifiers (NUMA-aware)"
+ echo "$0 ";
+}
+
+PRESENT=$(cpufreq-info | grep driver | uniq | grep acpi-cpufreq | wc -l)
+if [ $PRESENT -ne 1 ]; then
+ exit 1;
+fi
+
+exit 0
+#+end_src
+*** Set frequency to maximum
+#+begin_src sh :results output :tangle scripts/set_maximum_frequency.sh :tangle-mode (identity #o755)
+#!/bin/bash
+DIR=$(dirname $0)
+
+MAXFREQ=$(cpufreq-info | grep limits | sed -e "s/.*- //" -e "s/ //g" | uniq)
+
+# Get all online cores
+ONLINECPUS=$(for CPU in $(find /sys/devices/system/cpu/ | grep -v cpu0 | grep cpu[0-9]*$); do [[ $(cat $CPU/online) -eq 1 ]] && echo $CPU; done | grep cpu[0-9]*$ | sed 's/.*cpu//')
+
+# Core 0 is always online
+ONLINECPUS="0 ${ONLINECPUS}"
+
+for PU in ${ONLINECPUS}; do
+ echo "Setting the frequency of PU ${PU} to ${MAXFREQ}"
+ sudo cpufreq-set -c ${PU} -f ${MAXFREQ}
+ echo "After setting to max, the frequency is now $(cat /sys/devices/system/cpu/cpu${PU}/cpufreq/scaling_cur_freq)"
+done
+#+end_src
+** Configure multi-node experimental settings
+A script to control the experiment.
+- [X] Input is a file with node names
+
+For each machine:
+- [X] Disable turboboost
+- [X] Disable hyperthreading
+- [X] The =acpi_cpufreq=
+ - [X] Check for its use
+ - [X] Fix processor frequency to maximum
+- [X] Call the getinfo script after everything
+
+#+begin_src sh :results output :tangle scripts/control_experiment.sh :tangle-mode (identity #o755)
+#!/bin/bash
+DIR=$(dirname $0)
+
+function usage()
+{
+ echo "$0 [dest]";
+ echo " where is a file with a list of hostnames, one per line";
+ echo " where is a directory where captured files should be moved to";
+}
+
+##############################
+# Parameter handling section #
+##############################
+MACHINEFILE=${1:-}
+if [ -z "$MACHINEFILE" ]; then
+ echo "Error: is empty"
+ usage;
+ exit;
+fi
+
+DEST=${2:-}
+if [ -z "$DEST" ]; then
+ DEST="."
+fi
+
+##############################
+# For every host #
+##############################
+while IFS='' read -u10 -r HOST; do
+ # Ignore comments and empty lines
+ [[ "$(echo $HOST | sed 's/ //g')" =~ ^$ ]] && continue
+ [[ "$HOST" =~ ^#.*$ ]] && continue
+
+ # Clear anything after the colon
+ HOST=$(echo $HOST | sed "s/:.*$//g")
+
+ echo "** $HOST"
+ echo "*** passwordless ssh"
+
+ # Check is passwordless ssh is possible
+ X=$(eval ssh -o 'PreferredAuthentications=publickey' ${HOST} "echo -n"; echo $?)
+ if [ $X -ne 0 ]; then
+ echo "passwordless ssh is not possible to ${HOST}."
+ exit ;
+ else
+ echo "${HOST} is acessible through passwordless ssh."
+ fi
+
+ echo "*** Check presence of acpi-cpufreq driver"
+
+ # Check for acpi-cpufreq
+ scp $DIR/detect_acpidriver.sh ${HOST}:/tmp/
+ PRESENT=$(eval ssh ${HOST} "/tmp/detect_acpidriver.sh ; echo $?")
+ if [ $PRESENT -ne 0 ]; then
+ echo "The driver acpi-cpufreq is not available in ${HOST}"
+ exit
+ else
+ echo "I believe the ${HOST} is using the acpi-cpufreq DVFS driver."
+ fi
+
+ echo "*** Disable hyperthreading"
+
+ # Disable hyperthreading
+ scp $DIR/hyperthreading.sh ${HOST}:/tmp/
+ scp $DIR/disable_hyperthreading.sh ${HOST}:/tmp/
+ ssh ${HOST} /tmp/disable_hyperthreading.sh
+
+ echo "*** Disable turboboost"
+
+ # Disable turboboost
+ scp $DIR/disable_turboboost.sh ${HOST}:/tmp/
+ ssh ${HOST} /tmp/disable_turboboost.sh
+
+ echo "*** Set the frequency to maximum"
+
+ # Manually set the frequency to maximum
+ scp $DIR/set_maximum_frequency.sh ${HOST}:/tmp/
+ ssh ${HOST} /tmp/set_maximum_frequency.sh
+
+ echo "*** Call the get_info.sh"
+
+ # After everything, call the get_info.sh script
+ scp $DIR/get_info.sh ${HOST}:/tmp/
+ GETINFO=$(echo ${HOST} | sed "s/\./_/g" | sed "s/$/_get_info.org/")
+ ssh ${HOST} "rm -f /tmp/${GETINFO}"
+ ssh ${HOST} "/tmp/get_info.sh -t 'Get information from the ${HOST}' /tmp/${GETINFO}"
+ scp ${HOST}:/tmp/$GETINFO .
+ echo $GETINFO
+ if [ "$DEST" != "." ]; then
+ mv -f $GETINFO $DEST
+ fi
+
+done 10< ${MACHINEFILE}
+#+end_src
+** Run an Alya experiment
+Input:
+- [X] The machine file with the list of reserved machines
+- [X] The number of MPI processes to launch
+- [X] Path to Alya binary
+- [X] Path to the DAT file
+
+Everything should be ran in =/tmp/= directory, outside of NFS. The
+script will pushd to the directory of the DAT file (that should be
+outside of NFS) and ran from there.
+
+#+begin_src sh :results output :tangle scripts/run_alya_experiment.sh :tangle-mode (identity #o755)
+#!/bin/bash
+DIR=$(dirname $0)
+
+source $DIR/create_experiment_dir.sh
+
+function usage()
+{
+ echo "$0 ";
+ echo " where is a file with a list of hostnames, one per line";
+ echo " where is the number of MPI process";
+ echo " where is the path to the Alya binary";
+ echo " where is the path to the DAT file";
+ echo " where is a comment without white spaces";
+ echo " where (for infiniband) is true or false";
+}
+
+function now() { date +%s; }
+
+##############################
+# Parameter handling section #
+##############################
+MACHINEFILE=${1:-}
+if [ -z "$MACHINEFILE" ]; then
+ echo "Error: is empty"
+ usage;
+ exit;
+fi
+MACHINEFILE=$(readlink -f $MACHINEFILE)
+
+if [ ! -e "$MACHINEFILE" ]; then
+ echo "Error: '$MACHINEFILE' does not exists."
+ usage;
+ exit
+fi
+
+NP=${2:-}
+if [ -z "$NP" ]; then
+ echo "Error: is empty"
+ usage;
+ exit;
+fi
+
+ALYA=${3:-}
+if [ -z "$ALYA" ]; then
+ echo "Error: is empty"
+ usage;
+ exit;
+fi
+ALYA=$(readlink -f $ALYA)
+
+DAT=${4:-}
+if [ -z "$DAT" ]; then
+ echo "Error: is empty"
+ usage;
+ exit;
+fi
+DAT=$(readlink -f $DAT)
+
+UNIQUE=${5:-}
+if [ -z "$UNIQUE" ]; then
+ echo "Error: is empty"
+ usage;
+ exit;
+else
+ # Make sure unique is without spaces
+ UNIQUE=$(echo $UNIQUE | sed "s/ /_/g")
+fi
+
+IB=${6:-}
+if [ -z "$IB" ]; then
+ echo "Error: is empty";
+ exit;
+fi
+
+##############################
+# Create the directory #
+##############################
+# The org file of this script
+OUTPUTORG=$(org_file $UNIQUE)
+OUTPUTORGERR=$(org_err_file $UNIQUE)
+# The getinfo of where this script has been executed
+OUTPUTORGINFO=$(org_info_file $UNIQUE)
+# The place where results will be stored
+OUTPUTORGDIR=$(dir_file $UNIQUE)
+# Create the experimental directory
+create_experiment_dir $UNIQUE
+# Create the two org files
+touch $OUTPUTORG $OUTPUTORGINFO
+# From now on, all standard output goes to $OUTPUTORG
+exec > $OUTPUTORG
+exec 2> $OUTPUTORGERR
+
+DRYRUN=false
+
+##############################
+# Prepare the experiment #
+##############################
+echo "* Prepare the environment"
+
+# Call control the experiment
+if [ "$DRYRUN" = "false" ]; then
+ $DIR/control_experiment.sh ${MACHINEFILE} ${OUTPUTORGDIR}
+else
+ echo "This is a dry run, not calling the control experiment"
+fi
+
+# Save the machinefile forever
+cp $MACHINEFILE ${OUTPUTORGDIR}
+
+echo "* Alya"
+echo "** Binary"
+echo "The alya binary: $ALYA"
+ls -lh $ALYA
+# ldd is rather important
+ldd $ALYA
+# Heinrich told me he saves the binary, let's do the same
+cp $ALYA $OUTPUTORGDIR
+echo "** Input"
+
+echo "*** DAT"
+echo "The contents of the dat file:"
+ls -lh $DAT
+org_example_file $DAT
+DATDIR=$(dirname $DAT)
+DATBASE=$(basename $DAT .dat)
+echo "The DAT dir is ${DATDIR}, the DAT base is ${DATBASE}"
+
+echo "*** DOM.DAT"
+DOMDAT=${DATDIR}/${DATBASE}.dom.dat
+echo "The contents of the $DOMDAT"
+ls -lh $DOMDAT
+org_example_file $DOMDAT
+
+echo "*** KER.DAT"
+KERDAT=${DATDIR}/${DATBASE}.ker.dat
+echo "The contents of the $KERDAT"
+ls -lh $KERDAT
+org_example_file $KERDAT
+
+echo "*** NSI.DAT"
+NSIDAT=${DATDIR}/${DATBASE}.ker.dat
+echo "The contents of the $NSIDAT"
+ls -lh $NSIDAT
+org_example_file $NSIDAT
+
+if [ "$DRYRUN" = "false" ]; then
+ $DIR/get_info.sh -t "Get information from launcher host $(hostname)" $OUTPUTORGINFO
+else
+ echo "This is a dry run, not calling the get info script"
+fi
+
+##############################
+# Run the experiment #
+##############################
+echo "* ScoreP directory"
+SCOREPDIR="$HOME/scorep_${UNIQUE}"
+echo "$SCOREPDIR (finally moved to this experiment directory)"
+rm -rf $SCOREPDIR
+
+echo "* The Experiment"
+if [ "$IB" = "true" ]; then
+ OPENIB=""
+else
+ OPENIB="OMPI_MCA_btl=\"tcp,self,sm\""
+fi
+
+# btl_base_verbose is set to 40 "info"
+COMMAND="${OPENIB} mpirun --mca btl_base_verbose 40 -x SCOREP_MPI_ENABLE_GROUPS=no -x SCOREP_VERBOSE=true -x SCOREP_TOTAL_MEMORY=4G -x SCOREP_EXPERIMENT_DIRECTORY=${SCOREPDIR} -x SCOREP_OVERWRITE_EXPERIMENT_DIRECTORY=true -x SCOREP_ENABLE_TRACING=true --bind-to core --report-bindings -np $NP -machinefile $MACHINEFILE $ALYA ${DATDIR}/${DATBASE}"
+echo "The command to be executed: '$COMMAND'"
+echo "The epoch time now, before the execution, is: $(now)"
+echo "#+BEGIN_EXAMPLE"
+if [ "$DRYRUN" = "false" ]; then
+ eval $COMMAND
+fi
+echo "#+END_EXAMPLE"
+echo "The epoch time now, after the execution, is: $(now)"
+
+##############################
+# Copy LOGs #
+##############################
+echo "* The Logs"
+ls -lh ${DATDIR}/*.log
+cp ${DATDIR}/*.log $OUTPUTORGDIR
+
+echo "* Copy other small produced data"
+ls -lh ${DATDIR}/*.cvg ${DATDIR}/*.msh ${DATDIR}/*.res ${DATDIR}/*.sol ${DATDIR}/*.set
+cp ${DATDIR}/*.cvg ${DATDIR}/*.msh ${DATDIR}/*.res ${DATDIR}/*.sol ${DATDIR}/*.set $OUTPUTORGDIR
+
+echo "* Copy results of our manual instrumentation of alya code"
+ls -lh results_NPOIN_NELEM_NELEW_NBOUN.log
+mv -f results_NPOIN_NELEM_NELEW_NBOUN.log $OUTPUTORGDIR
+
+##############################
+# Save in the directory #
+##############################
+# Move scorep directory to
+mv -f $SCOREPDIR $OUTPUTORGDIR
+
+# Move org file to directory
+mv -f $OUTPUTORG $OUTPUTORGDIR
+
+# Move org err file to directory
+mv -f $OUTPUTORGERR $OUTPUTORGDIR
+
+# Move get_info org to directory
+mv -f $OUTPUTORGINFO $OUTPUTORGDIR
+
+# Copy myself to directory
+cp -f $0 $OUTPUTORGDIR
+
+# Copy all scripts to directory
+cp -f $DIR/*.sh $OUTPUTORGDIR
+
+# Exit graciously
+exit 0
+#+end_src
+
+#+RESULTS:
+** Convert traces.otf2 to CSV (version with =pj_dump=)
+#+begin_src sh :results output :tangle scripts/otf22csv.sh :tangle-mode (identity #o755)
+#!/bin/bash
+DIR=$(dirname $0)
+
+otf22csv() {
+ pushd $(dirname $otf2)
+ otf22paje traces.otf2 | pj_dump | grep ^State | cut -d, -f2,4,5,8 | sed -e "s/ //g" -e "s/MPIRank//" | gzip > traces.csv.gz
+ popd
+}
+
+function usage()
+{
+ echo "$0 "
+ echo " where is a directory that contains .otf2 files";
+}
+
+##############################
+# Parameter handling section #
+##############################
+OTF2DIR=${1:-}
+if [ -z "$OTF2DIR" ]; then
+ echo "Error: is empty"
+ usage;
+ exit;
+fi
+OTF2DIR=$(readlink -f $OTF2DIR)
+
+if [ ! -d "$OTF2DIR" ]; then
+ echo "Error: $OTF2DIR is not a directory"
+ usage
+ exit
+fi
+
+##############################
+# Check for necessary tools #
+##############################
+if [ -z "$(which pj_dump)" ]; then
+ echo "Error: pj_dump is not in the PATH"
+ exit
+fi
+if [ -z "$(which otf22paje)" ]; then
+ echo "Error: otf22paje is not in the PATH"
+ exit
+fi
+
+# Files already converted (whose CSV size is not zero)
+EXISTINGFILE=$(tempfile)
+OTF2FILE=$(tempfile)
+find $OTF2DIR -not -empty | grep csv.gz$ | sed -e "s/.csv.gz$//" | sort > $EXISTINGFILE
+find $OTF2DIR | grep \.otf2$ | sed -e "s/.otf2$//" | sort > $OTF2FILE
+
+for otf2 in $(comm -3 $OTF2FILE $EXISTINGFILE | sed "s/$/.otf2/"); do
+ echo $otf2
+ otf22csv $otf2
+done
+#+end_src
+** Convert traces.otf2 to CSV (version *without* =pj_dump=, only *otf22csv*)
+#+begin_src sh :results output :tangle scripts/otf22csv_faster.sh :tangle-mode (identity #o755)
+#!/bin/bash
+DIR=$(dirname $0)
+
+otf22csv_faster() {
+ pushd $(dirname $otf2)
+ otf22csv traces.otf2 | gzip > traces.csv.gz
+ popd
+}
+
+function usage()
+{
+ echo "$0 "
+ echo " where is a directory that contains .otf2 files";
+}
+
+##############################
+# Parameter handling section #
+##############################
+OTF2DIR=${1:-}
+if [ -z "$OTF2DIR" ]; then
+ echo "Error: is empty"
+ usage;
+ exit;
+fi
+OTF2DIR=$(readlink -f $OTF2DIR)
+
+if [ ! -d "$OTF2DIR" ]; then
+ echo "Error: $OTF2DIR is not a directory"
+ usage
+ exit
+fi
+
+##############################
+# Check for necessary tools #
+##############################
+if [ -z "$(which otf22csv)" ]; then
+ echo "Error: otf22csv is not in the PATH"
+ exit
+fi
+
+# Files already converted (whose CSV size is not zero)
+EXISTINGFILE=$(tempfile)
+OTF2FILE=$(tempfile)
+find $OTF2DIR -not -empty | grep csv.gz$ | sed -e "s/.csv.gz$//" | sort > $EXISTINGFILE
+find $OTF2DIR | grep \.otf2$ | sed -e "s/.otf2$//" | sort > $OTF2FILE
+
+for otf2 in $(comm -3 $OTF2FILE $EXISTINGFILE | sed "s/$/.otf2/"); do
+ echo $otf2
+ otf22csv_faster $otf2
+done
+#+end_src
+** Post-processing in R
+#+begin_src R :results output :session :exports both :tangle scripts/trace2summary.R :tangle-mode (identity #o755)
+#!/usr/bin/Rscript
+library(readr);
+library(dplyr);
+alya_scorep_trace_read <- function(filename)
+{
+ meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[3], "_"));
+ read_csv(filename,
+ col_names=c("Rank", "Start", "End", "Value"),
+ progress=TRUE) %>%
+ # Transform Value to factor
+ mutate(Value = as.factor(Value)) %>%
+ # Detect begin and end of iterations
+ mutate(Iteration = case_when(
+ grepl("timste", .$Value) ~ 1,
+ grepl("endste", .$Value) ~ -1,
+ TRUE ~ 0)) %>%
+ group_by(Rank) %>%
+ # Create a logical to detect observations within iterations
+ mutate(Iteration = as.logical(cumsum(Iteration))) %>%
+ # Get only observations that belongs to some iteration
+ filter(Iteration == TRUE) %>%
+ ungroup() %>%
+ # Create the iteration by cumsum
+ mutate(Iteration = case_when(
+ grepl("timste", .$Value) ~ 1,
+ TRUE ~ 0)) %>%
+ group_by(Rank) %>%
+ mutate(Iteration = as.integer(cumsum(Iteration))) %>%
+ ungroup() %>%
+ # Define metadata
+ mutate(EID = meta[2],
+ Platform = meta[3],
+ Nodes = meta[4],
+ NP = meta[5],
+ Partitioning = meta[6],
+ Infiniband = as.logical(meta[7]));
+}
+
+alya_scorep_trace_iterations <- function(filename)
+{
+ alya_scorep_trace_read(filename) %>%
+ group_by(Rank, Iteration, Platform, Nodes, NP, Partitioning, EID, Infiniband) %>%
+ filter(grepl("MPI_", Value)) %>%
+ summarize(N=n(), S=min(Start), E=max(End), Comm=sum(End-Start), Comp=(E-S)-Comm);
+}
+
+args = commandArgs(trailingOnly=TRUE)
+if (length(args) < 1) {
+ stop("Usage: trace2summary.R ", call.=FALSE)
+}
+
+#
+# parse arguments
+#
+outputFilename = args[[1]];
+inputs = unlist(args[2:length(args)]);
+
+df <- do.call("rbind", lapply(inputs, function(x) { alya_scorep_trace_iterations(x) }));
+write.csv(df, outputFilename);
+#+end_src
+* Guilherme entries
+* Meeting
+** 2018-04-18 Arnaud and Lucas
+
+At http://rendez-vous.renater.fr/alyalbsfc
+
+Rick's goal: 10K cores in MN4
+
+Eeach iteration has: Assemblage and Solver
+
+Rick has made a perfect LB in the Assembly phase only.
+
+The idea now is to make him repeat our observation methodology.
+
+** 2018-04-18 Paper Outline (Discussion @ Barcelona w. Arnaud and Ricard)
+*** General information
+- Target: Physics journal/conf, multi-physics simulation
+ developers. Note that I think this kind of work would totally make
+ sense in CCPE or JPDC.
+- Thoughts about our work:
+ - Nothing specific to Alya
+ - Load balance in-depth study and mitigation is interesting as
+ everyone relies on METIS or SFC.
+ - Profiling technique not new for HPC performance experts but
+ interesting for physics simulation developers.
+ - 1D load balancing strategy not new from an algorithmic perspective
+ - Refinement process expected to work but could have failed as
+ well...
+ - Doing a good load estimation and a good load balance in
+ practice on a real code is not that easy common.
+- Goal: explain how this can be done as it could apply to many other
+ codes
+*** Outline
+**** Intro
+**** Context :Guillaume:Ricard:
+ - Alya structure
+ - 3 typical test cases)
+ - respiratory (up to ~1 K cpus - ~25M elem. mesh)
+ - combustor (up to ~3K cpus - ~50M elem. mesh)
+ - plane (up to ~6K cpus - 160M elem. mesh)
+ - Scalability not satifying regardless of the partitioner in use
+ (METIS, SFC).
+**** In-depth Load Analysis of one test case (mostly :Lucas:Arnaud:
+ - Tracing is intrusive and cumbersome but an informed profiling with
+ e.g., Score-P is cheap and requires almost no modification of the
+ application.
+ - Illustrate the load imbalance and regularity of the application
+ across iterations. Illustrate the fact that the load imbalance is
+ different for assembly and for the sparse solver.
+ - Explain the reason behind the bad load balance (e.g., with SFC but
+ this does not depend on the partitioner). There is no magic
+ formula relating geometric characteristics with the actual
+ execution time.
+**** Proposal: iterative improvement of the load distribution on an SFC :Lucas:Arnaud:Ricard:
+ parition based on a few iterations of the application.
+ - Illustration of the quick improvements of the load balance
+**** Evaluation of the technique on the three test cases at different scales :Lucas:Arnaud:Ricard:
+ - Report both:
+ - the parallel execution time
+ - an evaluation of the load imbalance on the assembly and on the
+ solver to know how much gain can still be expected
+ - the amount of communications
+**** Conclusion
+ - The main issue is the load estimation which seems very hard to do
+ without running the code for real.
+ - Our iterative load refinement allows significant gains but
+ requires a few "dry runs". Ideally, it would be possible to
+ rebalance the load at runtime, which is something on which we are
+ working.
+*** Issues to solve (Ordered by priority)
+- Built-in standard profiling :: Ricard will run the experiments as he
+ has more experience in this but we need systematic and precise
+ load profiles for both writing the paper and deciding how to
+ efficiently balance (e.g., balancing the assembly vs. balancing
+ the sparse solver vs. both at the same time). To this end, Lucas
+ should install score-p on MN4 and explain Ricard how to obtain
+ the traces/profiles. Arnaud and Lucas will then do the analysis
+ and graph generation.
+- Granularity problem :: Too coarse granularity is the best
+ explanation (simpler than unstable memory ordering) for the lack
+ of good performance at scale and the "jumps" we observed with
+ Lucas. It seems there was some memory issue preventing Ricard to
+ run with a finer granularity (bin size)
+- Pre-processing :: Solve pre-processing problems to run SFC on large
+ cases.
+*** Possible future work
+- Could DLB be applied with our static iterative improvemet (e.g., by
+ grouping heavily loaded nodes with lightly loaded ones if we cannot
+ solve our granularity problem). Guillaume is not convinced as there
+ is some overhead.
+- Would it make sense to have different bin sizes, i.e., an SFC more
+ or less fine grained ? If so would it be possible to adjust the bin
+ size when reading the data structure ? What about a simple quad-tree
+ that would automatically split when bins are too large ? First, we
+ would record where elements lie, and estimate their density locally
+ (Note that this could be a pre-processing step independant of the
+ execution of Alya...) then, we would merge the density from the
+ different ranks. The density could then be used to determine at
+ which granularity the SFC should be managed.
+
+* Temporal entries
+** Old entries (from my journal)
+*** 2017-02-06 Alya Meeting :MEET:
+ Two alternatives with just one physics model:
+ - METIS partitioning
+ - Space filling curves
+
+ Tasks:
+ 1. Check if there is really a difference
+ 2. Compare the two .par files
+ - With GID or ours (ggplot2)
+
+ Sources of problems:
+ - Load balancing problem
+ - Too many communication
+ - Badly distributed
+
+ Entered on [2017-02-06 Mon 17:46]
+*** 2017-02-07 Installing and Testing GID :BOOT:
+
+http://www.gidhome.com/download/official-versions/linux64/
+
+Download the =tar.gz= version:
+#+begin_src shell :results output
+cd /tmp/
+wget -q ftp://www.gidhome.com/pub/GiD_Official_Versions/Linux/amd64/gid13.0.1-linux-x64.tar.gz
+#+end_src
+
+#+RESULTS:
+
+Extract it
+#+begin_src shell :results output
+cd /tmp/
+tar xfzv gid13.0.1-linux-x64.tar.gz | head
+#+end_src
+
+#+RESULTS:
+#+begin_example
+gid13.0.1-x64/
+gid13.0.1-x64/gid.exe
+gid13.0.1-x64/desktop_icons.tcl
+gid13.0.1-x64/scripts/
+gid13.0.1-x64/scripts/StatusLabel.tcl
+gid13.0.1-x64/scripts/ContourColors.tcl
+gid13.0.1-x64/scripts/units.tcl
+gid13.0.1-x64/scripts/gdi/
+gid13.0.1-x64/scripts/gdi/pkgIndex.tcl
+gid13.0.1-x64/scripts/DataWindows.tcl
+#+end_example
+
+Check if there is a binary:
+#+begin_src shell :results output
+cd /tmp/gid13.0.1-x64/
+file gid
+file gid.exe
+#+end_src
+
+#+RESULTS:
+: gid: POSIX shell script, ASCII text executable
+: gid.exe: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.15, BuildID[sha1]=b5e042b34364226055f5446ec033407d51bac59c, stripped
+
+Let's go for the script, using the terminal.
+
+Tool works in my machine, I can read the examples, =render= to make them
+beautiful. I tried the motor example.
+
+Entered on [2017-02-07 Tue 09:57]
+*** 2017-02-07 Discussions avec Guillaume Houzeaux (Arnaud's Notes) :MEET:
+***** Original notes
+- Ouverture de compte sur marenostrum pour Lucas. Guillaume s'en
+ occupe et nous dit dès que c'est fait. En début de semaine prochaine
+ normalement.
+- Question générale: Pourquoi les space filling curves sont tellement
+ plus efficaces sur leur code que METHIS alors que tous les articles
+ disent le contraire ?
+- Use case: système respiratoire.
+ - visualisation (avec GiD cimne x64, téléchargeable en version démo
+ sans problème) des mesh/sous domaine MPI avec (fichier .par) en 3D
+ pour visualiser qui est qui et où ils sont
+ - Là, le résultat de Methis montrait qu'un des process était
+ responsable de zones très éloignées (au début du tube
+ respiratoire et dans les bronches). Est-ce que ça explique le
+ problème ? Pas clair... Il faudra qu'on regarde ça nous même.
+ - Typical size: 256 cores, 5 secondes pour quelques itérations, ce
+ qui devrait suffire si le partitionnement est déjà fait.
+ - Faire le partitionnement une bonne fois pour toute et faire un
+ restart d'Alya à partir de là.
+ - Par rapport à
+ http://bsccase02.bsc.es/alya/index.html#Solution_procedure, la
+ description de Guillaume correspond à ce qui se passe dans "solve
+ module 1". Il y a juste un module, pas de couplage. (M= moment,
+ S=Schurr).
+ - A priori, pas de lien entre les informations applicatives et ce
+ qui apparaît dans une trace paraver
+ - Outputs:
+ - un .log
+ - -partition.par.post.msh et -partition.par.post.res
+ - Compilation METHIS
+ #+BEGIN_SRC
+ cd Alya
+ cd thirdparties/metis-4.0; make ...
+ cd Extecutables/unix; cat config.h ; # Guillaume will provide a typical one
+ ./configure nastin parall ; make
+ #+END_SRC
+ - Run:
+ - Guillaume will explain us how to do the partitionning and how to
+ reuse them.
+ - See the doc:
+ http://bsccase02.bsc.es/alya/tutorial/partitions.html. It is
+ actually up-to-date
+ - Warning: master slave organization. Hence partition on 255 and
+ run on 256 procs.
+ - Dans gensap-nastin.dat, on peut définir le NUMBER_OF_STEPS
+ - Input:
+***** Follow-up
+I've checked the site:
+http://bsccase02.bsc.es/alya/index.html#Solution_procedure
+
+#+BEGIN_EXAMPLE
+ - Read files, define mesh dependent arrays
+ - Initial solution or read restart
+ - Output and postprocess
+
+ +--- do time steps
+ !
+ | - Compute time step
+ | - Begin a time step: update bc, etc.
+ |
+ | do blocks ---------------------------------+
+ | |
+ | do coupling --------+ | Modules are grouped into blocks
+ | - Solve module1 | |
+ ! - Solve module2 | |
+ ! ... | | Useful if some equations are coupled
+ | - Check coupling cvg | | and others decoupled. Example
+ | end do coupling --------+ | Block 1: Nastin - Turbul
+ | | Block 2: Chemic: species are transported
+ | Goto new block |
+ | end do blocks ---------------------------------+
+ |
+ | - End time step
+ | - Output and postprocess
+ |
+ +--- end do time steps
+
+ - Output and postprocess
+ - End the run
+#+END_EXAMPLE
+
+Then, I've checked the step-by-step tutorial to partition:
+http://bsccase02.bsc.es/alya/tutorial/partitions.html
+
+The key line in my simple view now is this one:
+#+BEGIN_EXAMPLE
+TASK: ONLY_PREPROCESS, BINARY, SUBDOMAIN=num_domains
+#+END_EXAMPLE
+present in the =case.dat= configuration file for ALYA.
+Run the MPI program without =mpirun=, using a single process.
+
+Then, the resulting pre-processing stage can be registered in files
+that are only read, and the actual simulation can take place.
+*** 2017-02-24 Marenostrum account creation :MN3:
+***** E-mails from Guillaume Houzeaux :ATTACH:
+ :PROPERTIES:
+ :Attachments: Screen%20Shot%202017-02-01%20at%2014.49.05.png Screen%20Shot%202017-02-01%20at%2014.49.31.png
+ :ID: 789c7373-448e-44a3-9c5a-a0afa244280d
+ :END:
+Two screen shots joined here.
+***** Paper about load balancing in Alya :ATTACH:
+ :PROPERTIES:
+ :Attachments: IJHPCA.pdf
+ :ID: 5439e62d-768a-4a98-a77f-4890b829c6ff
+ :END:
+
+Load balancing of an MPI parallel unstructured CFD code using DLB and
+OpenMP. Marta Garcia-Gasulla 1 , Guillaume Houzeaux 1 , Antoni
+Artigues 1 , Jesús Labarta 12 and Mariano Vázquez 1.
+
+- unstructured meshes
+- the assembly step
+ - a loop over the elements of the mesh to compute element matrix and right-hand side
+ - an then to assemble them into global matrix and right-hand side
+- load balancing is an issue
+ - mesh partitioning is never perfect
+ - hardware can also be a random source of imbalance
+- approach
+ - dynamic load balance library (OpenMP) \to idle MPI tasks by others
+
+***** E-mail of account creation :ATTACH:
+ :PROPERTIES:
+ :Attachments: user_responsibilities.pdf 0769_001.pdf
+ :ID: b16bcfa4-f3db-40da-b443-8db078c817e9
+ :END:
+We would like to inform you that the accounts of your activity in the
+supercomputer Marenostrum III have been created. Here you have the basic
+information to access and use the supercomputer.
+
+When accessing to the machine for the first time, an error message
+regarding ssh keys may appear. This is normal and you have to erase (mn1,mn2,mn3)
+entries from:
+=$HOME/.ssh/known_hosts=
+
+We have 3 login nodes for access to MareNostrum III machine
+{mn1,mn2,mn3}.bsc.es,all connections must be done through ssh
+(Secure SHell).
+
+#+BEGIN_EXAMPLE
+ssh username@mn1.bsc.es
+ssh username@mn2.bsc.es
+#+END_EXAMPLE
+
+The usernames associated to your project leaded by Jose M. Cela are:
+
+ =Lucas Schnorr:bsc21835=
+
+The password will be sent in a separate mail with no subject unless you already
+had one for dl01.bsc.es. In order to change your password, you have to login at:
+
+=dl01.bsc.es=
+
+with the same username and password than in the cluster. Then, you have to use
+the 'passwd' command. The new password will become effective 10 minutes after
+the change. For security reasons the password must be changed the
+first time every user accesses the machine.
+
+MareNostrum III provides 5 interactive nodes. In order to gain access to
+each of them you need a ssh connection
+
+#+BEGIN_EXAMPLE
+ssh login5
+#+END_EXAMPLE
+
+The login nodes serve as front ends and are used typically for
+editing, compiling, preparation and submition of batch executions.
+It is not permitted the execution of cpu-bound programs on these nodes,
+if some execution needs more cputime than the permitted, this needs
+to be done through the batch queue system.
+
+The basic limits in queues by job are 48 "wall clock" hours and 2048 cpus. If
+you need to submit a job exceeding 1024 cpus, then you must contact the
+support team to study the scalability and schedule its execution.
+
+No connections are allowed from inside Marenostrum III to the outside world,
+so all the file transfers have to be executed from your local machines
+and not inside Marenostrum III.
+
+Example to copy files or directories from MN to external machine:
+
+#+BEGIN_EXAMPLE
+scp -r {username}@dt01.bsc.es:"SOURCE_MN_directory" "DEST_directory"
+#+END_EXAMPLE
+
+Example to copy files or directories from external machine to MN:
+
+#+BEGIN_EXAMPLE
+scp -r "SOURCE_directory" {username}@dt01.bsc.es:"DEST_MN_directory"
+#+END_EXAMPLE
+
+Each project needs to inform us regularly about their evolution to let us
+know the status of each project running on the machine.
+
+All accounts will be for single user access and they must not be shared
+among several users. If your project needs more resources or accounts,
+only the project manager must do this kind of request. You can find
+the user's guide of MareNostrum III at the following link:
+
+ http://www.bsc.es/support/MareNostrum3-ug.pdf
+
+Remember to send us the User Responsibilities document. If we do not receive it
+15 days after the account is accepted, the access will be disabled.
+
+All BSC's staff is eager to help you in the development and success of
+your project, please contact us for any question / suggestion:
+
+ e-mail: support@bsc.es
+ Barcelona Supercomputing Center
+ C/ Jordi Girona, 31
+ 08034 Barcelona
+
+Entered on [2017-02-24 Fri 10:46]
+*** 2017-02-24 Trying to compile Alya, preprocess, postprocess :ALYA:
+**** Preliminaries (with Alya compilation)
+***** Local SSH configuration for BSC supercomputer
+Put this in your =.ssh/config=
+#+BEGIN_EXAMPLE
+Host bscdata
+ Hostname dt01.bsc.es
+ User bsc21835
+Host bsc
+ Hostname mn3.bsc.es
+ User bsc21835
+#+END_EXAMPLE
+***** Source code identification
+Currently in my Dropbox. It corresponds to a tarball Guillaume
+Houseaux sent me a couple of months ago (the same used by M. Camelo
+in his PFE).
+#+begin_src shell :results output
+cd ~/Downloads/
+md5sum AlyaR1Source.tar.gz
+#+end_src
+
+#+RESULTS:
+: e8d20de91b56c64030d04ed1a4b1c559 AlyaR1Source.tar.gz
+***** Copy Alya Sources to MN3
+#+begin_src shell :results output
+rsync -avuzP ~/Downloads/AlyaR1Source.tar.gz bscdata:.
+ssh bscdata md5sum AlyaR1Source.tar.gz
+#+end_src
+
+#+RESULTS:
+: sending incremental file list
+: AlyaR1Source.tar.gz
+:
32,768 0% 0.00kB/s 0:00:00
21,856,256 21% 20.81MB/s 0:00:03
56,590,336 55% 26.98MB/s 0:00:01
93,487,104 92% 29.72MB/s 0:00:00
101,349,570 100% 29.91MB/s 0:00:03 (xfr#1, to-chk=0/1)
+:
+: sent 91,884,154 bytes received 34 bytes 20,418,708.44 bytes/sec
+: total size is 101,349,570 speedup is 1.10
+: e8d20de91b56c64030d04ed1a4b1c559 AlyaR1Source.tar.gz
+***** Alya Compilation
+Execute the following commands there.
+
+#+begin_src shell :results output
+tar xfz AlyaR1Source.tar.gz
+cd ~/AlyaR1Source/Thirdparties/metis-4.0; make
+cd ~/AlyaR1Source/Executables/unix/;
+cp configure.in/config_ifort.in config.in
+./configure nastin parall ; make
+#+end_src
+
+I've got two binaries (g for Debug):
+
+#+BEGIN_EXAMPLE
+-rwxr-xr-x 1 bsc21835 bsc21 8.8M Feb 24 14:09 Alya.x
+-rwxr-xr-x 1 bsc21835 bsc21 27M Feb 24 14:10 Alya.g
+#+END_EXAMPLE
+**** Preprocess the =4_tufan=
+***** Copy the test case to there
+#+begin_src shell :results output
+md5sum ~/Downloads/4_tufan_run.tar.gz
+rsync -avuzP ~/Downloads/4_tufan_run.tar.gz bscdata:.
+ssh bscdata md5sum 4_tufan_run.tar.gz
+#+end_src
+
+#+RESULTS:
+: b4127cb22bed9309d1234827f41135c9 /home/schnorr/Downloads/4_tufan_run.tar.gz
+: sending incremental file list
+: 4_tufan_run.tar.gz
+:
32,768 0% 0.00kB/s 0:00:00
20,709,376 5% 19.72MB/s 0:00:19
52,789,248 12% 25.16MB/s 0:00:13
82,837,504 20% 26.33MB/s 0:00:12
115,376,128 28% 27.51MB/s 0:00:10
149,848,064 36% 30.80MB/s 0:00:08
183,861,248 45% 31.27MB/s 0:00:07
216,989,696 53% 31.99MB/s 0:00:05
249,724,928 61% 32.04MB/s 0:00:04
283,770,880 69% 31.93MB/s 0:00:03
317,849,600 77% 31.95MB/s 0:00:02
347,111,424 84% 31.02MB/s 0:00:01
377,749,504 92% 30.53MB/s 0:00:00
408,525,195 100% 30.11MB/s 0:00:12 (xfr#1, to-chk=0/1)
+:
+: sent 396,055,909 bytes received 34 bytes 27,314,202.97 bytes/sec
+: total size is 408,525,195 speedup is 1.03
+: b4127cb22bed9309d1234827f41135c9 4_tufan_run.tar.gz
+***** Unpack
+#+begin_src shell :results output
+tar xfz 4_tufan_run.tar.gz
+cd 4_tufan_run/c/
+#+end_src
+***** Preprocess using METIS 4.0
+Everything is done locally, in my notebook, because I don't know yet
+how to submit jobs in MN and Alya.x fails to run on =login= nodes.
+
+See: http://bsccase02.bsc.es/alya/tutorial/partitions.html
+- Partion preprocess step
+
+I am using Metis 4, but the tutorial tells me to use version 5.
+
+Let's proceed anyway.
+
+Create the directories to hold the partitioned meshes (for 16 domains)
+
+#+begin_src shell :results output
+cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
+../../Utils/user/alya-hierarchy 116
+#+end_src
+
+#+RESULTS:
+:
+: --| alya-hierarchy |--
+: --|
+: --| 2 directories have been created
+: --|
+: --| Bye.
+: --|
+: --|
+
+This is the DAT file for preprocessing.
+
+#+begin_src shell :results output
+cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
+cat lucas_preprocess_case.dat
+#+end_src
+
+#+RESULTS:
+#+begin_example
+$-------------------------------------------------------------------
+RUN_DATA
+ ALYA: sq_cyl
+ RUN_TYPE: noCONTI , PRELIMINARY, FREQUENCY=100
+ LATEX_INFO_FILE: Yes
+ LIVE_INFORMATION: Screen
+END_RUN_DATA
+$-------------------------------------------------------------------
+PROBLEM_DATA
+ TIME_COUPLING: GLOBAL, PRESCR
+ TIME_INTERVAL= 0.0,100000.0
+ TIME_STEP_SIZE= 0.00025
+ NUMBER_OF_STEPS= 100000
+ MAXIMUM_NUMBER_GLOBAL= 1
+ NASTIN_MODULE: On
+ END_NASTIN_MODULE
+ PARALL_SERVICE: On
+ PARTITION_TYPE: FACES
+ FILE_HIERARCHY: ON
+ FILE_OPEN_CLOSE: Yes
+ VIRTUAL_FILE: On, MAXIMUM_MEMORY=0.5
+ TASK: ONLY_PREPROCESS, BINARY, SUBDOMAIN=16
+ END_PARALL_SERVICE
+END_PROBLEM_DATA
+$-------------------------------------------------------------------
+#+end_example
+
+Note the customized =SUBDOMAIN=16= setting.
+
+Local execution gives this as output:
+
+#+begin_src shell :results output
+cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
+mpirun -np 1 ../../Executables/unix/Alya.x lucas_preprocess_case
+#+end_src
+
+#+RESULTS:
+#+begin_example
+--|
+--| ALYA START ALYA FOR PROBLEM: sq_cyl
+--|
+--| ALYA READ PROBLEM DATA
+--| ALYA START CONSTRUCT DOMAIN
+--| ALYA READ MESH DATA
+--| ALYA READ MESH ARRAYS
+--| ALYA READ ELEMENT TYPES
+--| ALYA CHECK ELEMENT TYPES
+--| ALYA READ ELEMENT CONNECTIVITY
+--| ALYA CHECK ELEMENT CONNECTIVITY
+--| ALYA READ COORDINATES
+--| ALYA READ BOUNDARY CONNECTIVITY, TYPES AND ELEMENT CONNECTIVITY
+--| ALYA CHECK BOUNDARY TYPES
+--| ALYA CHECK BOUNDARY CONNECTIVITY
+--| ALYA CHECK CONNECTIVITY BOUNDARY/ELEMENT
+--| ALYA READ BOUNDARY CONDITIONS
+--| ALYA WARNING: KERNEL WILL EXTRAPOLATE FROM BOUNDARY CODES TO NODE CODES
+--| ALYA WARNING: KERNEL READS BOUNDARY CODES ON BOUNDARIES
+--| ALYA END READ MESH ARRAYS
+--| ALYA COMPUTE GRAPH
+--| ALYA KERNEL: COMPUTE GROUPS OF DEFLATED CG
+--| ALYA START MESH PARTITION (# OF SUBDOMAINS= 16)
+--| ALYA PARALL PREPROCESS --> ALLOCATE MEMORY...
+--| ALYA PARALL PREPROCESS --> SET WEIGHTS...
+--| ALYA MASTER COMPUTES ELEMENT GRAPH (# EDGES= 6214430, MAX # EDGES/ELEMENT= 6)
+--| ALYA MASTER PARTITIONS ELEMENT GRAPH USING FACE CONNECTIVITY
+--| ALYA MASTER COMPUTES COMMUNICATION STRATEGY
+--| ALYA MASTER ORDERS INTERIOR AND BOUNDARY NODES
+--| ALYA MASTER COMPUTES PERMUTATION ARRAYS
+--| ALYA MASTER WRITES MESH AND PARTITION DATA IN RESTART FILES WITH HIERARCHY
+--| ALYA PARALL PREPROCESS --> S/R 1 - TOT 16
+--| ALYA PARALL PREPROCESS --> S/R 2 - TOT 16
+--| ALYA PARALL PREPROCESS --> S/R 3 - TOT 16
+--| ALYA PARALL PREPROCESS --> S/R 4 - TOT 16
+--| ALYA PARALL PREPROCESS --> S/R 5 - TOT 16
+--| ALYA PARALL PREPROCESS --> S/R 6 - TOT 16
+--| ALYA PARALL PREPROCESS --> S/R 7 - TOT 16
+--| ALYA PARALL PREPROCESS --> S/R 8 - TOT 16
+--| ALYA PARALL PREPROCESS --> S/R 9 - TOT 16
+--| ALYA PARALL PREPROCESS --> S/R 10 - TOT 16
+--| ALYA PARALL PREPROCESS --> S/R 11 - TOT 16
+--| ALYA PARALL PREPROCESS --> S/R 12 - TOT 16
+--| ALYA PARALL PREPROCESS --> S/R 13 - TOT 16
+--| ALYA PARALL PREPROCESS --> S/R 14 - TOT 16
+--| ALYA PARALL PREPROCESS --> S/R 15 - TOT 16
+--| ALYA PARALL PREPROCESS --> S/R 16 - TOT 16
+--| ALYA PARALL PREPROCESS --> SEND/REC 16 - TOTAL 16
+--| ALYA PARALL PREPROCESS --> COORD, LTYPE, LNODS, LTYPB, LNODB...
+--| ALYA PARALL PREPROCESS --> LBOEL...
+--| ALYA PARALL PREPROCESS --> KFL_FIELD, XFIEL, TIME_FIELD,...
+--| ALYA PARALL PREPROCESS --> SET DATA...
+--| ALYA PARALL PREPROCESS --> BC DATA...
+--| ALYA PARALL PREPROCESS --> COMMUNICATION ARRAYS...
+--| ALYA PARALL PREPROCESS --> SEND DATA TO SLAVES... DONE.
+--| ALYA END MESH PARTITION
+--|
+--| ALYA CALCULATIONS CORRECT
+--|
+#+end_example
+
+Here are the contents of the =PAR*= dirs:
+
+#+begin_src shell :results output
+cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
+ls -1hl PAR000*
+#+end_src
+
+#+RESULTS:
+#+begin_example
+PAR000000:
+total 4.0K
+-rw-r--r-- 1 schnorr schnorr 2.9K Feb 24 16:02 lucas_preprocess_case.par.rst0
+
+PAR000100:
+total 102M
+-rw-r--r-- 1 schnorr schnorr 6.4M Feb 24 16:02 lucas_preprocess_case.par.rst1
+-rw-r--r-- 1 schnorr schnorr 6.4M Feb 24 16:02 lucas_preprocess_case.par.rst10
+-rw-r--r-- 1 schnorr schnorr 6.5M Feb 24 16:02 lucas_preprocess_case.par.rst11
+-rw-r--r-- 1 schnorr schnorr 6.4M Feb 24 16:02 lucas_preprocess_case.par.rst12
+-rw-r--r-- 1 schnorr schnorr 6.3M Feb 24 16:02 lucas_preprocess_case.par.rst13
+-rw-r--r-- 1 schnorr schnorr 6.7M Feb 24 16:02 lucas_preprocess_case.par.rst14
+-rw-r--r-- 1 schnorr schnorr 6.6M Feb 24 16:02 lucas_preprocess_case.par.rst15
+-rw-r--r-- 1 schnorr schnorr 6.3M Feb 24 16:02 lucas_preprocess_case.par.rst16
+-rw-r--r-- 1 schnorr schnorr 6.3M Feb 24 16:02 lucas_preprocess_case.par.rst2
+-rw-r--r-- 1 schnorr schnorr 6.5M Feb 24 16:02 lucas_preprocess_case.par.rst3
+-rw-r--r-- 1 schnorr schnorr 6.4M Feb 24 16:02 lucas_preprocess_case.par.rst4
+-rw-r--r-- 1 schnorr schnorr 6.3M Feb 24 16:02 lucas_preprocess_case.par.rst5
+-rw-r--r-- 1 schnorr schnorr 6.5M Feb 24 16:02 lucas_preprocess_case.par.rst6
+-rw-r--r-- 1 schnorr schnorr 6.3M Feb 24 16:02 lucas_preprocess_case.par.rst7
+-rw-r--r-- 1 schnorr schnorr 6.3M Feb 24 16:02 lucas_preprocess_case.par.rst8
+-rw-r--r-- 1 schnorr schnorr 6.6M Feb 24 16:02 lucas_preprocess_case.par.rst9
+#+end_example
+
+Okay, now let's try to use them avoiding the preprocess.
+**** Postprocess the =4_tufan=
+***** Postprocess
+Still local (notebook), for the same reasons.
+
+See: http://bsccase02.bsc.es/alya/tutorial/partitions.html
+- Partion postprocess step
+
+#+begin_src shell :results output
+cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
+cat lucas_postprocess_case.dat
+#+end_src
+
+#+RESULTS:
+#+begin_example
+$-------------------------------------------------------------------
+RUN_DATA
+ ALYA: sq_cyl
+ RUN_TYPE: noCONTI , PRELIMINARY, FREQUENCY=100
+ LATEX_INFO_FILE: Yes
+ LIVE_INFORMATION: Screen
+END_RUN_DATA
+$-------------------------------------------------------------------
+PROBLEM_DATA
+ TIME_COUPLING: GLOBAL, PRESCR
+ TIME_INTERVAL= 0.0,100000.0
+ TIME_STEP_SIZE= 0.00025
+ NUMBER_OF_STEPS=2
+ MAXIMUM_NUMBER_GLOBAL= 1
+ NASTIN_MODULE: On
+ END_NASTIN_MODULE
+
+$ this is the section that does the postprocess
+ PARALL_SERVICE: On
+ PARTITION_TYPE: FACES
+ FILE_HIERARCHY: ON
+ FILE_OPEN_CLOSE: Yes
+ VIRTUAL_FILE: On, MAXIMUM_MEMORY=0.5
+ TASK: READ_PREPROCESS, BINARY
+ END_PARALL_SERVICE
+
+$ This part has been removed to only preprocess
+$ PARALL_SERVICE: On
+$ OUTPUT_FILE: OFF
+$ POSTPROCESS: MASTER
+$ PARTITION_TYPE: FACES
+$ $ COMMUNICATION: ASYNCRONOUS
+$ END_PARALL_SERVICE
+END_PROBLEM_DATA
+$-------------------------------------------------------------------
+#+end_example
+
+The machine file I have:
+
+#+begin_src shell :results output
+cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
+cat machinefile
+#+end_src
+
+#+RESULTS:
+: localhost slots=32 max_slots=32
+
+Now, let's run again with 17 procs:
+
+#+begin_src shell :results output
+cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
+mpirun -machinefile machinefile -np 17 ../../Executables/unix/Alya.x lucas_postprocess_case
+#+end_src
+
+#+RESULTS:
+
+This has failed. The log tells me why:
+
+#+begin_src shell :results output
+cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
+cat lucas_postprocess_case.log
+#+end_src
+
+#+RESULTS:
+#+begin_example
+
+
+
+ --------
+ |- A L Y A: sq_cyl
+ --------
+
+
+
+
+
+ --------------------------------------------------------
+
+ |- AN ERROR HAS BEEN DETECTED:
+
+ ERROR WHEN OPENING THE PARALL RESTART FILE: PAR000000/lucas_postprocess_case.par.rst0
+
+#+end_example
+
+Looks like I can't change the DAT filename.
+
+Let's rename it.
+
+Next code block is non-reproducible.
+
+#+begin_src shell :results output
+cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
+#cp lucas_preprocess_case.dat lucas_preprocess_case.dat.bak
+#cp lucas_postprocess_case.dat lucas_preprocess_case.dat
+cat lucas_preprocess_case.dat
+#+end_src
+
+#+RESULTS:
+#+begin_example
+$-------------------------------------------------------------------
+RUN_DATA
+ ALYA: sq_cyl
+ RUN_TYPE: noCONTI , PRELIMINARY, FREQUENCY=100
+ LATEX_INFO_FILE: Yes
+ LIVE_INFORMATION: Screen
+END_RUN_DATA
+$-------------------------------------------------------------------
+PROBLEM_DATA
+ TIME_COUPLING: GLOBAL, PRESCR
+ TIME_INTERVAL= 0.0,100000.0
+ TIME_STEP_SIZE= 0.00025
+ NUMBER_OF_STEPS=2
+ MAXIMUM_NUMBER_GLOBAL= 1
+ NASTIN_MODULE: On
+ END_NASTIN_MODULE
+
+$ this is the section that does the postprocess
+ PARALL_SERVICE: On
+ OUTPUT_FILE: OFF
+ PARTITION_TYPE: FACES
+ FILE_HIERARCHY: ON
+ FILE_OPEN_CLOSE: Yes
+ VIRTUAL_FILE: On, MAXIMUM_MEMORY=0.5
+ TASK: READ_PREPROCESS, BINARY
+ END_PARALL_SERVICE
+
+END_PROBLEM_DATA
+$-------------------------------------------------------------------
+#+end_example
+
+Okay.
+
+Let's run it:
+
+#+begin_src shell :results output
+cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
+mpirun -machinefile machinefile -np 17 ../../Executables/unix/Alya.x lucas_preprocess_case 2>&1 > x.log
+#+end_src
+
+#+RESULTS:
+
+#+begin_src shell :results output
+cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
+cat x.log
+#+end_src
+
+#+RESULTS:
+#+begin_example
+--|
+--| ALYA START ALYA FOR PROBLEM: sq_cyl
+--|
+--| ALYA CHECK MPI. 1 MASTER + 16 SUBDOMAINS
+--| ALYA MPI IS WORKING WELL
+--| ALYA READ PROBLEM DATA
+--| ALYA START CONSTRUCT DOMAIN
+--| ALYA MASTER/SLAVES READ MESH AND PARTITION DATA FROM RESTART FILE WITH HIERARCHY
+--| ALYA MASTER/SLAVES: ALL SLAVES HAVE READ THEIR RESTART FILES
+--| ALYA KERMOD: READ DATA
+--| ALYA KERMOD FILE DOES NOT EXITS: USE DEFAULT OPTIONS
+--| ALYA PARALL: COMPUTE INTER-COLOR COMMUNICATORS
+--| ALYA PARALL: COMPUTE BIN STRUCTURE FOR PARTITIONS
+--| ALYA RENUMBER ELEMENTS
+--| ALYA RECOMPUTE GRAPH FOR PERIODICITY
+--| ALYA CHECK ELEMENT ORDERING
+--| ALYA CHECK BOUNDARY ORDERING
+--| ALYA PARALL: GET HALO GEOMETRY
+--| ALYA PARALL: CREATE INTRA-ZONE COMMUNICATION ARRAY
+--| ALYA PARALL: CREATE INTRA-SUBDOMAIN COMMUNICATION ARRAY
+--| ALYA OUTPUT MESH
+--| ALYA COMPUTE LUMPED MASS MATRIX
+--| ALYA COMPUTE CLOSED MASS MATRIX
+--| ALYA COMPUTE EXTERIOR NORMALS
+--| ALYA EXTRAPOLATE BOUNDARY CODES TO NODE CODES
+--| ALYA END CONSTRUCT DOMAIN
+--| ALYA MODULE DATA
+--| ALYA NASTIN: READ DATA
+--| ALYA NASTIN: MASTER SENDS PHYSICAL AND NUMERICAL DATA TO SLAVES
+--| ALYA NASTIN: MASTER SENDS PHYSICAL AND NUMERICAL ARRAYS TO SLAVES
+--| ALYA WARNINGS HAVE BEEN FOUND IN MODULE NASTIN
+--|
+--| ALYA ABORTED IN |----> 1 ERROR HAS BEEN FOUND IN NASTIN DATA FILE
+--|
+#+end_example
+
+**** Next steps
+- Get an updated version of Alya
+- Fix the partition postprocess step
+- Test case
+ - The mesh for the respiratory system
+ - Adapt preprocess + postprocess to this case
+- Find out how to activate the second method
+ for mesh partitioning (space filling curves)
+- Reconfirm the fact that SFC are faster than METIS
+** 2017-03-27 Trying to compile SVN version of Alya locally
+*** Configure with gfortran
+#+begin_src shell :results output
+cd ~/misc/alya/alya-bsc/Executables/unix/
+rm config.in
+ln -s ./configure.in/config_gfortran.in config.in
+./configure nastin parall
+#+end_src
+
+#+RESULTS:
+#+begin_example
+
+--| Alya configure |--
+
+--| Build Alya makefile
+
+ Linux OS identified.
+ Using Intel Fortran Compiler ifort.
+
+ Creating makefile for the modules demanded, when the source folder exists:
+
+ Name: adapti ( service # 59 ) --> compile = no iffold = yes iffake = no
+ Name: alefor ( module # 7 ) --> compile = no iffold = yes iffake = no
+ Name: apelme ( module # 9 ) --> compile = no iffold = no iffake = no
+ Name: casimi ( module # 23 ) --> compile = no iffold = no iffake = no
+ Name: cgns24 ( service # 56 ) --> compile = no iffold = yes iffake = yes
+ Name: chemic ( module # 19 ) --> compile = no iffold = yes iffake = no
+ Name: codire ( module # 3 ) --> compile = no iffold = yes iffake = no
+ Name: commdo ( service # 65 ) --> compile = no iffold = no iffake = no
+ Name: coupli ( service # 64 ) --> compile = no iffold = no iffake = no
+ Name: dodeme ( service # 57 ) --> compile = no iffold = yes iffake = yes
+ Name: exmedi ( module # 5 ) --> compile = no iffold = yes iffake = no
+ Name: gidpos ( service # 53 ) --> compile = no iffold = yes iffake = yes
+ Name: gotita ( module # 11 ) --> compile = no iffold = yes iffake = no
+ Name: handfp ( service # 54 ) --> compile = no iffold = yes iffake = yes
+ Name: hdfpos ( service # 62 ) --> compile = no iffold = yes iffake = no
+ Name: helmoz ( module # 20 ) --> compile = no iffold = yes iffake = no
+ Name: immbou ( module # 21 ) --> compile = no iffold = yes iffake = no
+ Name: latbol ( module # 8 ) --> compile = no iffold = no iffake = no
+ Name: levels ( module # 14 ) --> compile = no iffold = yes iffake = no
+ Name: magnet ( module # 16 ) --> compile = no iffold = yes iffake = no
+ Name: nasedg ( module # 18 ) --> compile = no iffold = yes iffake = no
+ Name: nastal ( module # 6 ) --> compile = no iffold = yes iffake = no
+ Name: nastin ( module # 1 ) --> compile = yes iffold = yes iffake = no --> OK to compile!
+ Name: neutro ( module # 26 ) --> compile = no iffold = yes iffake = no
+ Name: optsol ( service # 63 ) --> compile = no iffold = yes iffake = yes
+ Name: parall ( service # 55 ) --> compile = yes iffold = yes iffake = yes --> OK to compile!
+ Name: partis ( module # 17 ) --> compile = no iffold = yes iffake = no
+ Name: porous ( module # 24 ) --> compile = no iffold = yes iffake = no
+ Name: quanty ( module # 15 ) --> compile = no iffold = yes iffake = no
+ Name: radiat ( module # 22 ) --> compile = no iffold = yes iffake = no
+ Name: shapar ( service # 51 ) --> compile = no iffold = no iffake = no
+ Name: solidz ( module # 10 ) --> compile = no iffold = yes iffake = no
+ Name: solmum ( service # 52 ) --> compile = no iffold = yes iffake = yes
+ Name: solpls ( service # 58 ) --> compile = no iffold = yes iffake = no
+ Name: temper ( module # 2 ) --> compile = no iffold = yes iffake = no
+ Name: turbul ( module # 4 ) --> compile = no iffold = yes iffake = no
+ Name: wavequ ( module # 12 ) --> compile = no iffold = yes iffake = no
+ Name: xxxxxx ( module # 25 ) --> compile = no iffold = no iffake = no
+
+ Makefile will make the following binaries:
+
+ Alya.x --> Production (non debugger) binary
+ Alya.g --> Debugger binary
+
+
+--| Makefile created. Ready to make.
+--| The command you typed has been saved in file conf.log
+--| Bye.
+
+
+#+end_example
+*** Make
+#+begin_src shell :results output
+cd ~/misc/alya/alya-bsc/ThirdParties/metis-4.0/; make -j 4 2>&1
+cd ~/misc/alya/alya-bsc/Executables/unix/; make -j 4 2>&1
+#+end_src
+
+#+RESULTS:
+** 2017-03-28 OAR Job Allocation
+#+begin_src shell :results output
+oarsub -p "cluster='taurus'" -l "nodes=4/walltime=03:00:00" -r "$(date '+%Y-%m-%d %H:%M:%S')" -t deploy
+oarsub -C 852259
+#taurus-1.lyon.grid5000.fr
+#taurus-16.lyon.grid5000.fr
+#taurus-2.lyon.grid5000.fr
+#taurus-7.lyon.grid5000.fr
+kadeploy3 --env-file images/stretch_energy.env --file ${OAR_NODE_FILE} -k
+cd ~/alya-bsc/Thirdparties/metis-4.0/; make -j 4
+cd ~/alya-bsc/Executables/unix/; ln -s ./configure.in/config_gfortran.in config.in; make -j 4
+#+end_src
+** 2017-03-29 Experiments to execute
+#+begin_src shell :results output
+for CASE in sfc metis; do
+ for NP in 32 64 96 120; do
+ NP=$((NP+1))
+ UNIQUE=taurus-np${NP}-${CASE}
+ echo "./scripts/run_alya_experiment.sh machine-file ${NP} ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE"
+ echo "mv $UNIQUE /home/lschnorr/"
+ done
+done > script.sh
+cat script.sh
+#+end_src
+
+#+RESULTS:
+#+begin_example
+./scripts/run_alya_experiment.sh machine-file 33 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np33-sfc
+mv taurus-np33-sfc /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 65 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np65-sfc
+mv taurus-np65-sfc /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 97 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np97-sfc
+mv taurus-np97-sfc /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 121 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np121-sfc
+mv taurus-np121-sfc /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 33 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np33-metis
+mv taurus-np33-metis /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 65 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np65-metis
+mv taurus-np65-metis /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 97 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np97-metis
+mv taurus-np97-metis /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 121 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np121-metis
+mv taurus-np121-metis /home/lschnorr/
+#+end_example
+** 2017-03-29 Quick analysis of first results
+I got these:
+
+#+name: first-results
+#+begin_src shell :results output org table
+for case in $(find $HOME/segundo/taurus-np* | grep -e "sfc.org" -e "metis.org"); do echo -n "$case "; cat $case | grep epoch | sed "s/^.*: //" | tr '\n' '-' | sed -e "s/-$//" -e "s/-/ - /" -e "s/$/\n/" | bc -l; done | sed -e "s/^.*np//" -e "s/.org//" -e "s/-/ /g" -e "s/ */|/g"
+#+end_src
+
+#+RESULTS: first-results
+#+BEGIN_SRC org
+| 121 | metis | 0 |
+| 121 | sfc | 1 |
+| 33 | metis | 1160 |
+| 33 | sfc | 938 |
+| 65 | metis | 653 |
+| 65 | sfc | 612 |
+| 97 | metis | 503 |
+| 97 | sfc | 470 |
+#+END_SRC
+
+#+header: :var df=first-results
+#+begin_src R :results output graphics :file img/first-results.png :exports both :width 600 :height 400 :session
+library(dplyr);
+library(ggplot2);
+df %>%
+ rename(NP = V1,
+ Case = V2,
+ Time = V3) %>%
+ filter(NP != 121) %>%
+ ggplot(aes(x=NP, y=Time, color=Case)) +
+ theme_bw() +
+ ylim(0,NA) +
+ geom_point() +
+ geom_line() +
+ theme(legend.position = "top");
+#+end_src
+
+#+RESULTS:
+[[file:img/first-results.png]]
+** 2017-03-29 Planning the tracing with Scorep
+
+Score-P 3.0 allows the selective instrumentation of any function using
+a GCC plugin. Do to so, refer to the Section 3.1 of ScoreP user
+manual. The flag is =--instrument-filter=, much better described in
+Section 5.3 of the same document. This method is preferred since it
+reduces the intrusion. To make such method available in a scorep
+installation, we need to previously install the package
+=gcc-6-plugin-dev= or something similar.
+
+It is interesting to disable everything else which is not MPI and the
+necessary to trace filtered user functions. Guilherme, for example,
+had used this line to instrument Alya:
+
+=scorep --nopomp --nocompiler --nocuda --nopdt --nouser --noopencl=
+
+We would have to keep only:
+
+=scorep --nopomp --nocuda --noopencl --nopdt=
+
+This is the filter that Guilherme used dynamically. We want to use it
+statically.
+
+#+begin_src shell :results output :tangle alya-timestep-filters.scorep
+SCOREP_REGION_NAMES_BEGIN
+EXCLUDE *
+INCLUDE MPI COM
+INCLUDE endste_ timste_
+SCOREP_REGION_NAMES_END
+#+end_src
+
+The way to use this filter in runtime is to use the appropriate
+environment variable this way:
+
+#+BEGIN_EXAMPLE
+SCOREP_FILTERING_FILE=name_of_the_file
+#+END_EXAMPLE
+
+_Summary_ of that has worked:
+
+Finally, I've managed to get it working with the following steps.
+
+1. Configure ScoreP with the following line
+#+BEGIN_EXAMPLE
+./configure --without-gui --prefix=/home/lschnorr/install/nova/scorep-3.0-alya/ --with-shmem=no --with-opari2=no
+#+END_EXAMPLE
+- Do not forget to install the gcc plugin package first
+
+2. Configure Alya's =config.in= to use scorep this way
+#+BEGIN_EXAMPLE
+SCOREP=scorep --compiler --instrument-filter=/home/lschnorr/alya-bsc/Executables/unix/alya-timestep-filters.scorep
+#+END_EXAMPLE
+Where the contents of the filter are like this
+#+BEGIN_EXAMPLE
+#SCOREP_FILE_NAMES_BEGIN
+#EXCLUDE *
+#INCLUDE */Alya.f90
+#SCOREP_FILE_NAMES_END
+
+SCOREP_REGION_NAMES_BEGIN
+EXCLUDE *
+INCLUDE endste
+INCLUDE timste
+SCOREP_REGION_NAMES_END
+#+END_EXAMPLE
+
+3. Finally, run the program this way
+#+BEGIN_EXAMPLE
+mpirun -x SCOREP_FILTERING_FILE=/home/lschnorr/alya-bsc/Executables/unix/alya-timestep-filters.scorep -x SCOREP_EXPERIMENT_DIRECTORY=/home/lschnorr/xxx-scorep7 -x SCOREP_OVERWRITE_EXPERIMENT_DIRECTORY=true -x SCOREP_ENABLE_TRACING=true --bind-to core -np 12 -machinefile /tmp/Alya-Perf/machine-file /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap
+#+END_EXAMPLE
+- The traces need to be put in the HOME NFS.
+- Filtering is not obligatory here since we instrumented in compile time.
+** 2017-04-05 Infiniband problems
+In all machines that belong to an Infiniband experiment, do:
+
+1. In file =/etc/security/limits.conf=, add these lines:
+ #+BEGIN_EXAMPLE
+ * hard memlock unlimited
+ * soft memlock unlimited
+ #+END_EXAMPLE
+
+2. In file =/etc/systemd/system.conf=, make the line looks like this:
+ #+BEGIN_EXAMPLE
+ DefaultLimitMEMLOCK=infinity
+ #+END_EXAMPLE
+** 2017-04-06 Nose check
+#+name: nose_table
+ | Rank | X | Y | Z |
+ |------+--------------+--------------+--------------|
+ | 1 | 0.919827E-01 | 0.115893E+00 | -.639115E-01 |
+ | 2 | 0.103425E+00 | 0.149886E+00 | -.123160E+00 |
+ | 3 | 0.103773E+00 | 0.169636E+00 | -.146991E+00 |
+ | 4 | 0.961346E-01 | 0.202625E+00 | -.216010E+00 |
+ | 5 | 0.139644E+00 | 0.204323E+00 | -.223233E+00 |
+ | 6 | 0.101879E+00 | 0.850899E-01 | 0.235732E-01 |
+ | 7 | 0.933165E-01 | 0.458008E-01 | 0.261210E-01 |
+ | 8 | 0.913297E-01 | 0.682040E-01 | 0.290498E-01 |
+ | 9 | 0.952650E-01 | 0.111200E+00 | 0.149948E-01 |
+ | 10 | 0.950596E-01 | 0.122169E+00 | -.234605E-01 |
+ | 11 | -.403059E+00 | -.400000E+00 | -.489343E+00 |
+ | 12 | 0.596941E+00 | -.400000E+00 | -.489343E+00 |
+ | 13 | 0.596941E+00 | 0.240397E+00 | -.489343E+00 |
+ | 14 | -.403059E+00 | 0.240397E+00 | -.489343E+00 |
+ | 15 | -.403059E+00 | -.400000E+00 | 0.510656E+00 |
+ | 16 | 0.596941E+00 | -.400000E+00 | 0.510656E+00 |
+ | 17 | 0.596941E+00 | 0.240397E+00 | 0.510656E+00 |
+ | 18 | -.403059E+00 | 0.240397E+00 | 0.510656E+00 |
+
+#+begin_src R :results output :session :exports both :var df=nose_table
+df;
+#+end_src
+
+#+RESULTS:
+#+begin_example
+ Rank X Y Z
+1 1 0.0919827 0.1158930 -0.0639115
+2 2 0.1034250 0.1498860 -0.1231600
+3 3 0.1037730 0.1696360 -0.1469910
+4 4 0.0961346 0.2026250 -0.2160100
+5 5 0.1396440 0.2043230 -0.2232330
+6 6 0.1018790 0.0850899 0.0235732
+7 7 0.0933165 0.0458008 0.0261210
+8 8 0.0913297 0.0682040 0.0290498
+9 9 0.0952650 0.1112000 0.0149948
+10 10 0.0950596 0.1221690 -0.0234605
+11 11 -0.4030590 -0.4000000 -0.4893430
+12 12 0.5969410 -0.4000000 -0.4893430
+13 13 0.5969410 0.2403970 -0.4893430
+14 14 -0.4030590 0.2403970 -0.4893430
+15 15 -0.4030590 -0.4000000 0.5106560
+16 16 0.5969410 -0.4000000 0.5106560
+17 17 0.5969410 0.2403970 0.5106560
+18 18 -0.4030590 0.2403970 0.5106560
+#+end_example
+
+#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 600 :height 400 :session
+library(ggplot2);
+plot(df);
+#+end_src
+
+#+RESULTS:
+[[file:/tmp/babel-22618hTf/figure22618i_y.png]]
+
+** 2017-04-06 Alya modifications to get more info, set balance to 1 :EXP10:
+#+BEGIN_EXAMPLE
+lschnorr@fnancy:~/alya-bsc$ git diff
+diff --git a/Executables/unix/configure.in/config_gfortran.in b/Executables/unix/configure.in/config_gfortran.in
+index 1c73a1a..90de238 100644
+--- a/Executables/unix/configure.in/config_gfortran.in
++++ b/Executables/unix/configure.in/config_gfortran.in
+@@ -4,16 +4,17 @@
+ #module load gcc/5.1.0 openmpi/1.8.5 #
+ ###################################################################
+
++SCOREP=scorep --compiler --instrument-filter=/home/lschnorr/alya-bsc/Executables/unix/alya-timestep-filters.scorep
+
+-F77 = mpif90
+-F90 = mpif90
+-FCOCC = mpicc -c
++F77 = $(SCOREP) mpif90
++F90 = $(SCOREP) mpif90
++FCOCC = $(SCOREP) mpicc -c
+ FCFLAGS = -c -J$O -I$O -ffree-line-length-none -fimplicit-none
+ FPPFLAGS = -x f95-cpp-input
+ EXTRALIB = -lc
+ EXTRAINC =
+-fa2p = mpif90 -c -x f95-cpp-input -DMPI_OFF -J../../Utils/user/alya2pos -I../../Utils/user/alya2pos
+-fa2plk = mpif90 -lc
++fa2p = $(SCOREP) mpif90 -c -x f95-cpp-input -DMPI_OFF -J../../Utils/user/alya2pos -I../../Utils/user/alya2pos
++fa2plk = $(SCOREP) mpif90 -lc
+
+ ###################################################################
+ # PERFORMANCE FLAGS #
+diff --git a/Sources/kernel/parall/mod_par_partit_sfc.f90 b/Sources/kernel/parall/mod_par_partit_sfc.f90
+index ee6fd1d..e55185d 100644
+--- a/Sources/kernel/parall/mod_par_partit_sfc.f90
++++ b/Sources/kernel/parall/mod_par_partit_sfc.f90
+@@ -383,7 +383,7 @@ contains
+
+ iboxf = lboxf(ielem)
+ iboxl = lboxl(ielem)
+- iweig = real(ngaus(abs(ltype_par(ielem))),rp)
++ iweig = 1_rp !real(ngaus(abs(ltype_par(ielem))),rp)
+
+ if( PAR_MY_PARMETIS_RANK /= iboxc-1 )then
+ if(bufwei(iboxl*2) == 0_rp) then
+diff --git a/Sources/services/parall/par_outinf.f90 b/Sources/services/parall/par_outinf.f90
+index 6664b93..301522a 100644
+--- a/Sources/services/parall/par_outinf.f90
++++ b/Sources/services/parall/par_outinf.f90
+@@ -15,6 +15,7 @@ subroutine par_outinf()
+ use mod_parall
+ implicit none
+ integer(ip) :: isubd,ksmin(2),ksmax(2),kaver,koutp,nb
++ integer, parameter :: out_unit_npoin=20
+
+ if( IMASTER ) then
+
+@@ -27,7 +27,11 @@ subroutine par_outinf()
+ ksmin = huge(1_ip)
+ ksmax = -huge(1_ip)
+ kaver = 0
++ ! LUCAS: manual instrumentation starts here, all code marked by LUCAS
++ open (unit=out_unit_npoin,file="results_NPOIN_NELEM_NELEW_NBOUN.log",action="write",status="replace")
+ do isubd=1,npart_par
++ ! LUCAS: Here I need to print: npoin_par(isubd)
++ write (out_unit_npoin,*) "Rank ", isubd ," NPOIN ", npoin_par(isubd), " NELEM ", nelem_par(isubd), " NELEW ", nelew_par(isubd), " NBOUN ", nboun_par(isubd), " NNEIG ", lneig_par(isubd), " NBBOU ", npoin_par(isubd)
+ if(npoin_par(isubd)%
+ select(-X1) %>%
+ rename(Start = S, End = E) %>%
+ mutate(Duration = End - Start) %>%
+ mutate(Platform = as.factor(Platform),
+ Partitioning = as.factor(Partitioning),
+ Nodes = as.factor(Nodes));
+
+df2 <- read_csv("data/11/326040-b8e2-41f3-8479-c378f70864d9/exp_11-v1_grimoire_8.csv.gz") %>%
+ select(-X1) %>%
+ rename(Start = S, End = E) %>%
+ mutate(Duration = End - Start) %>%
+ mutate(Platform = as.factor(Platform),
+ Partitioning = as.factor(Partitioning),
+ Nodes = as.factor(Nodes));
+
+df <- rbind(df1, df2);
+df %>% head;
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Parsed with column specification:
+cols(
+ X1 = col_integer(),
+ Rank = col_integer(),
+ Iteration = col_integer(),
+ Platform = col_character(),
+ Nodes = col_integer(),
+ NP = col_integer(),
+ Partitioning = col_character(),
+ EID = col_character(),
+ Infiniband = col_logical(),
+ N = col_integer(),
+ S = col_double(),
+ E = col_double(),
+ Comm = col_double(),
+ Comp = col_double()
+)
+Warning message:
+Missing column names filled in: 'X1' [1]
+Parsed with column specification:
+cols(
+ X1 = col_integer(),
+ Rank = col_integer(),
+ Iteration = col_integer(),
+ Platform = col_character(),
+ Nodes = col_integer(),
+ NP = col_integer(),
+ Partitioning = col_character(),
+ EID = col_character(),
+ Infiniband = col_logical(),
+ N = col_integer(),
+ S = col_double(),
+ E = col_double(),
+ Comm = col_double(),
+ Comp = col_double()
+)
+Warning message:
+Missing column names filled in: 'X1' [1]
+# A tibble: 6 × 14
+ Rank Iteration Platform Nodes NP Partitioning EID Infiniband N
+
+1 0 1 grimoire 8 128 sfc 10-v2 TRUE 5454
+2 0 2 grimoire 8 128 sfc 10-v2 TRUE 4752
+3 0 3 grimoire 8 128 sfc 10-v2 TRUE 4873
+4 0 4 grimoire 8 128 sfc 10-v2 TRUE 4697
+5 0 5 grimoire 8 128 sfc 10-v2 TRUE 4660
+6 0 6 grimoire 8 128 sfc 10-v2 TRUE 4544
+# ... with 5 more variables: Start , End , Comm , Comp ,
+# Duration
+#+end_example
+**** Plot Comp
+#+begin_src R :results output graphics :file img/exp_10-v2_exp_11-v1_comp.png :exports both :width 500 :height 400 :session
+library(tidyr);
+library(ggplot2);
+df %>%
+ select(-Start, -End, -N, -Duration) %>%
+ mutate(Total = Comm+Comp) %>%
+ gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID, -Infiniband) %>%
+ filter(Variable == "Comp", Iteration != 10, Rank != 0) %>%
+ ggplot(aes(x=as.factor(Iteration), y=Value, color=Infiniband)) +
+ theme_bw(base_size=12) +
+ geom_point(alpha=.3) +
+ ylim(0,NA) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
+ theme(legend.position="top") +
+ facet_grid(EID~Partitioning)
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_10-v2_exp_11-v1_comp.png]]
+*** Partition data
+**** Process
+#+begin_src shell :results output
+EDIR="exp_11-v1_grimoire_8 exp_10-v2_grimoire_8"
+for file in $(find $EDIR | grep results | grep log$); do
+ OUTPUT=$(dirname $file)/$(basename $file .log).csv
+ cat $file | tr -s " " | sed "s/^ //" | tr ' ' ',' | cut -d"," -f1,3,5,7,9,11,13 | uniq > $OUTPUT
+ cat $file | tr -s " " | sed "s/^ //" | tr ' ' ',' | cut -d"," -f2,4,6,8,10,12,14 >> $OUTPUT
+ echo "=> $file <="
+ head $OUTPUT
+done
+#+end_src
+
+#+RESULTS:
+#+begin_example
+=> exp_11-v1_grimoire_8/11-v1_grimoire_8_128_sfc_true.dir/results_NPOIN_NELEM_NELEW_NBOUN.log <=
+Rank,NPOIN,NELEM,NELEW,NBOUN,NNEIG,NBBOU
+1,32182,174168,174611,2973,16,32182
+2,20919,53529,174758,5381,15,20919
+3,19727,49972,175172,5020,11,19727
+4,21918,69049,174054,4277,23,21918
+5,21683,64345,174280,4439,14,21683
+6,22205,64122,175002,4582,18,22205
+7,21361,63432,174587,4436,11,21361
+8,22162,75699,174459,3914,8,22162
+9,21389,66339,174799,4332,8,21389
+=> exp_11-v1_grimoire_8/11-v1_grimoire_8_128_metis_true.dir/results_NPOIN_NELEM_NELEW_NBOUN.log <=
+Rank,NPOIN,NELEM,NELEW,NBOUN,NNEIG,NBBOU
+1,22405,85371,172276,3477,4,22405
+2,22923,86049,179769,3742,5,22923
+3,24316,99541,175416,3029,5,24316
+4,22907,90905,170125,3170,4,22907
+5,22636,86964,173194,3448,5,22636
+6,22774,88416,171711,3333,6,22774
+7,23564,91625,177865,3452,7,23564
+8,22211,83394,173364,3612,5,22211
+9,23427,89980,179685,3588,4,23427
+=> exp_10-v2_grimoire_8/10-v2_grimoire_8_128_sfc_true.dir/results_NPOIN_NELEM_NELEW_NBOUN.log <=
+Rank,NPOIN,NELEM,NELEW,NBOUN,NNEIG,NBBOU
+1,14976,72536,72796,2384,17,14976
+2,14641,72392,72575,448,12,14641
+3,22264,72647,167221,4446,16,22264
+4,28129,72464,251684,7187,11,28129
+5,23239,72488,178813,4341,25,23239
+6,26093,72525,209615,5640,14,26093
+7,22996,72391,183401,4422,10,22996
+8,21190,72393,160643,3505,7,21190
+9,25178,72612,201462,5195,10,25178
+=> exp_10-v2_grimoire_8/10-v2_grimoire_8_128_metis_true.dir/results_NPOIN_NELEM_NELEW_NBOUN.log <=
+Rank,NPOIN,NELEM,NELEW,NBOUN,NNEIG,NBBOU
+1,22405,85371,172276,3477,4,22405
+2,22923,86049,179769,3742,5,22923
+3,24316,99541,175416,3029,5,24316
+4,22907,90905,170125,3170,4,22907
+5,22636,86964,173194,3448,5,22636
+6,22774,88416,171711,3333,6,22774
+7,23564,91625,177865,3452,7,23564
+8,22211,83394,173364,3612,5,22211
+9,23427,89980,179685,3588,4,23427
+#+end_example
+**** Read
+#+begin_src R :results output :session :exports both
+library(readr);
+library(dplyr);
+library(tidyr);
+
+read_npoin <- function(filename)
+{
+ meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[2], "_"));
+ meta <- gsub(".dir", "", meta);
+ read_csv(filename) %>%
+ gather(Variable, Value, -Rank) %>%
+ mutate(EID = meta[1],
+ Platform = meta[2],
+ Nodes = meta[3],
+ NP = meta[4],
+ Partitioning = meta[5],
+ Infiniband = as.logical(meta[6]));
+
+}
+files <- list.files("exp_10-v2_grimoire_8", pattern="results_NPOIN_NELEM_NELEW_NBOUN.csv", recursive=TRUE, full.names=TRUE);
+files <- c(files, list.files("exp_11-v1_grimoire_8", pattern="results_NPOIN_NELEM_NELEW_NBOUN.csv", recursive=TRUE, full.names=TRUE));
+dfp <- do.call("rbind", lapply(files, function(x) { read_npoin(x) }))
+dfp %>% filter(Rank == 111, Variable == "NELEW")
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ NPOIN = col_integer(),
+ NELEM = col_integer(),
+ NELEW = col_integer(),
+ NBOUN = col_integer(),
+ NNEIG = col_integer(),
+ NBBOU = col_integer()
+)
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ NPOIN = col_integer(),
+ NELEM = col_integer(),
+ NELEW = col_integer(),
+ NBOUN = col_integer(),
+ NNEIG = col_integer(),
+ NBBOU = col_integer()
+)
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ NPOIN = col_integer(),
+ NELEM = col_integer(),
+ NELEW = col_integer(),
+ NBOUN = col_integer(),
+ NNEIG = col_integer(),
+ NBBOU = col_integer()
+)
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ NPOIN = col_integer(),
+ NELEM = col_integer(),
+ NELEW = col_integer(),
+ NBOUN = col_integer(),
+ NNEIG = col_integer(),
+ NBBOU = col_integer()
+)
+# A tibble: 4 × 9
+ Rank Variable Value EID Platform Nodes NP Partitioning Infiniband
+
+1 111 NELEW 172833 10-v2 grimoire 8 128 metis TRUE
+2 111 NELEW 258786 10-v2 grimoire 8 128 sfc TRUE
+3 111 NELEW 172833 11-v1 grimoire 8 128 metis TRUE
+4 111 NELEW 173994 11-v1 grimoire 8 128 sfc TRUE
+#+end_example
+*** Merge
+Get only the first iteration
+#+begin_src R :results output :session :exports both
+dfz <- df %>% filter(Iteration == 1, Rank != 0) %>%
+ select(-Platform, -Nodes, -NP, -Infiniband, -N, -Start, -End, -Comm, -Duration, -Iteration) %>%
+ gather(Variable, Value, Comp, -EID, -Partitioning) %>%
+ select(EID, Partitioning, Rank, Variable, Value);
+dfpz <- dfp %>% select(-Platform, -Nodes, -NP, -Infiniband) %>%
+ select(EID, Partitioning, Rank, Variable, Value);
+dfm <- rbind(dfz, dfpz);
+dfm %>% head;
+#+end_src
+
+#+RESULTS:
+: # A tibble: 6 × 5
+: EID Partitioning Rank Variable Value
+:
+: 1 10-v2 sfc 1 Comp 8.811678
+: 2 10-v2 sfc 2 Comp 8.560127
+: 3 10-v2 sfc 3 Comp 12.641175
+: 4 10-v2 sfc 4 Comp 15.811419
+: 5 10-v2 sfc 5 Comp 13.183812
+: 6 10-v2 sfc 6 Comp 14.227162
+*** Plot
+#+begin_src R :results output :session :exports both
+dfm %>% .$Variable %>% unique
+#+end_src
+
+#+RESULTS:
+: [1] "Comp" "NPOIN" "NELEM" "NELEW" "NBOUN" "NNEIG" "NBBOU"
+
+#+begin_src R :results output graphics :file img/exp_10-v2_exp_11-v1_comp_partition.png :exports both :width 800 :height 400 :session
+dfm %>%
+ mutate(EID = case_when(
+ grepl("10-v2", .$EID) ~ gsub("$", "(Equal Weight)", .$EID),
+ grepl("11-v1", .$EID) ~ gsub("$", "(Original Weight)", .$EID),
+ TRUE ~ "Undefined")) %>%
+ filter(Partitioning != "metis") %>%
+ filter(Variable %in% c("Comp", "NELEM", "NELEW")) %>%
+ ggplot(aes(x=Rank, y=Value, color=EID)) +
+ theme_bw(base_size=12) +
+ geom_point() +
+ ylim(0,NA) +
+ ggtitle ("SFC only, NP=128, 8-nodes@grimoire, Infiniband") +
+ theme(legend.position="top") +
+ facet_grid(Variable~EID, scales="free_y")
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_10-v2_exp_11-v1_comp_partition.png]]
+
+** 2017-04-07 Check alternative Alya instrumentation :EXP12:
+*** First try - no imbrication
+#+begin_src shell :results output
+ssh nancy.g5k cat ./alya-bsc/Executables/unix/all.scorep
+#+end_src
+
+#+RESULTS:
+#+begin_example
+#SCOREP_FILE_NAMES_BEGIN
+#EXCLUDE *
+#INCLUDE */Alya.f90
+#SCOREP_FILE_NAMES_END
+
+SCOREP_REGION_NAMES_BEGIN
+EXCLUDE *
+INCLUDE endste
+INCLUDE timste
+INCLUDE doiter
+INCLUDE concou
+INCLUDE conblk
+INCLUDE begzon
+INCLUDE endzon
+INCLUDE moduls
+INCLUDE nastin
+INCLUDE parall
+SCOREP_REGION_NAMES_END
+#+end_example
+
+#+begin_src shell :results output
+scp nancy.g5k:./scorep_12-v1_grimoire_6_96_metis_true/traces.csv.gz .
+#+end_src
+
+#+RESULTS:
+
+#+begin_src R :results output :session :exports both
+library(readr);
+library(dplyr);
+df <- read_csv("traces.csv.gz",
+ col_names=c("Rank", "Start", "End", "Value"),
+ progress=FALSE);
+df %>% group_by(Value) %>% summarize(N=n())
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ Start = col_double(),
+ End = col_double(),
+ Value = col_character()
+)
+# A tibble: 10 × 2
+ Value N
+
+1 begzon 96
+2 conblk 96
+3 concou 96
+4 doiter 96
+5 endste 96
+6 endzon 96
+7 moduls 1248
+8 nastin 1152
+9 parall 431331
+10 timste 96
+#+end_example
+
+Let's concentrate in nastin:
+
+#+begin_src R :results output :session :exports both
+df %>% filter(Rank == 61)
+#+end_src
+
+#+RESULTS:
+#+begin_example
+# A tibble: 4,542 × 4
+ Rank Start End Value
+
+1 61 0.010045 0.010054 parall
+2 61 0.012898 0.043018 parall
+3 61 0.043031 0.051135 parall
+4 61 0.051233 0.051235 moduls
+5 61 0.051251 249.036677 parall
+6 61 249.037003 249.094670 parall
+7 61 249.094764 249.094993 parall
+8 61 249.095035 249.095187 parall
+9 61 249.095202 249.095204 parall
+10 61 249.553567 249.709909 parall
+# ... with 4,532 more rows
+#+end_example
+
+I need imbrication to understand that.
+*** With imbrication, plot
+#+begin_src shell :results output
+scp nancy.g5k:./scorep_12-v1_grimoire_6_96_metis_true/traces.csv.gz .
+#+end_src
+
+#+RESULTS:
+
+#+begin_src R :results output :session :exports both
+df <- read_csv("traces.csv.gz",
+ col_names=c("Nature", "Rank", "Type", "Start", "End", "Duration", "Imbrication", "Value"),
+ progress=FALSE);
+df;
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Parsed with column specification:
+cols(
+ Nature = col_character(),
+ Rank = col_integer(),
+ Type = col_character(),
+ Start = col_double(),
+ End = col_double(),
+ Duration = col_double(),
+ Imbrication = col_double(),
+ Value = col_character()
+)
+# A tibble: 857,455 × 8
+ Nature Rank Type Start End Duration Imbrication Value
+
+1 State 95 STATE 0.000267 0.000279 0.000012 0 parall
+2 State 95 STATE 0.000290 0.001318 0.001029 0 parall
+3 State 95 STATE 0.001326 0.032243 0.030917 0 parall
+4 State 95 STATE 0.032352 0.032354 0.000002 0 moduls
+5 State 95 STATE 0.032367 252.062136 252.029768 0 parall
+6 State 95 STATE 252.062477 252.118046 0.055569 0 parall
+7 State 95 STATE 252.118115 252.118401 0.000286 0 parall
+8 State 95 STATE 252.118442 252.118595 0.000153 0 parall
+9 State 95 STATE 252.118610 252.118644 0.000034 0 parall
+10 State 95 STATE 252.722345 252.732470 0.010126 0 parall
+# ... with 857,445 more rows
+#+end_example
+
+#+begin_src R :results output :session :exports both
+dfplot %>% .$Imbrication %>% unique %>% sort
+startTime <- df %>% filter(Value == "timste") %>% arrange(Start) %>% slice(1) %>% .$Start
+startTime
+endTime <- df %>% filter(Value == "endste") %>% arrange(End) %>% slice(n()) %>% .$End
+endTime
+dfplot %>% filter(Start > startTime, End < endTime) %>% filter(Imbrication != 10) %>% filter(Value == "moduls")
+#+end_src
+
+#+RESULTS:
+#+begin_example
+[1] 0
+[1] 259.3502
+[1] 315.6014
+# A tibble: 15 × 8
+ Nature Rank Type Start End Duration Imbrication Value
+
+1 State 4 STATE 259.6966 259.6972 0.000613 0 moduls
+2 State 4 STATE 287.6941 287.7119 0.017868 0 moduls
+3 State 4 STATE 288.0547 288.0553 0.000610 0 moduls
+4 State 3 STATE 259.6966 259.6973 0.000682 0 moduls
+5 State 3 STATE 287.6947 287.7119 0.017238 0 moduls
+6 State 3 STATE 288.0546 288.0553 0.000678 0 moduls
+7 State 2 STATE 259.6966 259.6972 0.000600 0 moduls
+8 State 2 STATE 287.6937 287.7119 0.018168 0 moduls
+9 State 2 STATE 288.0546 288.0552 0.000601 0 moduls
+10 State 1 STATE 259.6966 259.6973 0.000712 0 moduls
+11 State 1 STATE 287.6949 287.7119 0.017059 0 moduls
+12 State 1 STATE 288.0547 288.0554 0.000707 0 moduls
+13 State 0 STATE 259.6968 259.6969 0.000010 0 moduls
+14 State 0 STATE 287.6883 287.7119 0.023637 0 moduls
+15 State 0 STATE 288.0548 288.0548 0.000011 0 moduls
+#+end_example
+
+#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 1200 :height 300 :session
+library(ggplot2);
+
+maxImbrication = df %>% .$Imbrication %>% max;
+plot <- NULL;
+dfplot <- df %>% filter(!(Value %in% c("timste", "endste"))) %>% filter(Rank < 5) %>% filter(Imbrication %in% c(0,1));
+dfplot %>%
+ ggplot() +
+ coord_cartesian(xlim=c(startTime, endTime)) +
+ theme_bw(base_size=12) +
+ scale_fill_brewer(palette = "Set1") -> plot;
+
+for (i in (dfplot %>% .$Imbrication %>% unique %>% sort)){
+ plot <- plot +
+ geom_rect(data=(dfplot %>% filter(Imbrication == i)), aes(fill=Value,
+ xmin=Start,
+ xmax=End,
+ ymin=Rank,
+ ymax=Rank+0.9-(Imbrication*(1/5))));
+}
+plot;
+#+end_src
+
+#+RESULTS:
+[[file:/tmp/babel-24853m4f/figure24853J5Y.png]]
+*** New tracing with MPI + nsi functions
+#+begin_src shell :results output
+ssh nancy.g5k cat ./alya-bsc/Executables/unix/all.scorep
+#+end_src
+
+#+RESULTS:
+#+begin_example
+#SCOREP_FILE_NAMES_BEGIN
+#EXCLUDE *
+#INCLUDE */Alya.f90
+#SCOREP_FILE_NAMES_END
+
+SCOREP_REGION_NAMES_BEGIN
+EXCLUDE *
+INCLUDE endste
+INCLUDE timste
+INCLUDE doiter
+INCLUDE moduls
+INCLUDE nastin
+INCLUDE parall
+INCLUDE nsi_doiter
+INCLUDE nsi_turnon
+INCLUDE nsi_parall
+
+SCOREP_REGION_NAMES_END
+#+end_example
+
+#+begin_src shell :results output
+scp nancy.g5k:./scorep_12-v1_grimoire_6_96_metis_true_v3/traces.csv.gz .
+#+end_src
+
+#+RESULTS:
+
+#+begin_src R :results output :session :exports both
+df <- read_csv("traces.csv.gz",
+ col_names=c("Nature", "Rank", "Type", "Start", "End", "Duration", "Imbrication", "Value"),
+ progress=FALSE);
+df <- df %>% mutate(Value = gsub("MPI_.*", "MPI", Value));
+df;
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Parsed with column specification:
+cols(
+ Nature = col_character(),
+ Rank = col_integer(),
+ Type = col_character(),
+ Start = col_double(),
+ End = col_double(),
+ Duration = col_double(),
+ Imbrication = col_double(),
+ Value = col_character()
+)
+# A tibble: 2,338,005 × 8
+ Nature Rank Type Start End Duration Imbrication Value
+
+1 State 95 STATE 1.770337 1.770339 2.0e-06 0 MPI
+2 State 95 STATE 1.770339 1.770340 1.0e-06 0 MPI
+3 State 95 STATE 1.770348 1.770353 5.0e-06 0 MPI
+4 State 95 STATE 1.770357 1.770449 9.3e-05 0 MPI
+5 State 95 STATE 1.770450 1.770690 2.4e-04 0 MPI
+6 State 95 STATE 1.770691 1.770691 0.0e+00 0 MPI
+7 State 95 STATE 1.770691 1.770691 0.0e+00 0 MPI
+8 State 95 STATE 1.770692 1.770693 1.0e-06 0 MPI
+9 State 95 STATE 1.770693 1.770723 3.0e-05 0 MPI
+10 State 95 STATE 1.770724 1.770731 8.0e-06 0 MPI
+# ... with 2,337,995 more rows
+#+end_example
+
+#+begin_src R :results output :session :exports both
+startTime <- df %>% filter(Value == "timste") %>% arrange(Start) %>% slice(1) %>% .$End
+startTime
+endTime <- df %>% filter(Value == "endste") %>% arrange(End) %>% slice(n()) %>% .$End
+endTime
+maxImbrication = df %>% .$Imbrication %>% max;
+maxImbrication
+#+end_src
+
+#+RESULTS:
+: [1] 126.1242
+: [1] 153.975
+: [1] 5
+
+#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 1200 :height 300 :session
+library(ggplot2);
+
+plot <- NULL;
+dfplot <- df %>%
+ filter(Rank != 0) %>%
+ filter(!(Value %in% c("timste", "endste", "nsi_turnon"))) %>%
+ filter(Imbrication != 5) %>%
+ filter(Rank < 5);
+dfplot %>%
+ ggplot() +
+ coord_cartesian(xlim=c(startTime, endTime)) +
+ xlim(startTime, endTime) +
+ theme_bw(base_size=12) +
+ scale_fill_brewer(palette = "Set1") -> plot;
+
+for (i in (dfplot %>% .$Imbrication %>% unique %>% sort)){
+ plot <- plot +
+ geom_rect(data=(dfplot %>% filter(Imbrication == i)), aes(fill=Value,
+ xmin=Start,
+ xmax=End,
+ ymin=Rank,
+ ymax=Rank+0.9-(Imbrication*(1/10))));
+}
+plot;
+#+end_src
+
+#+RESULTS:
+[[file:/tmp/babel-24853m4f/figure24853ygt.png]]
+
+Conclusions:
+- I should take the end =timste= and =endste= because =timste= is long
+- =nsi_doiter= (or the more general =doiter=) state has the calculations, but it has a lot of MPI calls within
+- =nsi_turnon= is not necessary
+
+#+begin_src R :results output :session :exports both
+df %>% filter(Rank == 4, Start > 135, End < 140) %>% filter(!(Value %in% c("timste", "endste", "doiter", "moduls")))
+#+end_src
+
+#+RESULTS:
+#+begin_example
+# A tibble: 2,674 × 8
+ Nature Rank Type Start End Duration Imbrication Value
+
+1 State 4 STATE 135.0003 135.0005 0.000219 4 MPI
+2 State 4 STATE 135.0008 135.0010 0.000210 4 MPI
+3 State 4 STATE 135.0012 135.0015 0.000228 4 MPI
+4 State 4 STATE 135.0017 135.0018 0.000048 4 parall
+5 State 4 STATE 135.0018 135.0018 0.000010 5 MPI
+6 State 4 STATE 135.0018 135.0018 0.000003 5 MPI
+7 State 4 STATE 135.0018 135.0018 0.000004 5 MPI
+8 State 4 STATE 135.0018 135.0018 0.000002 5 MPI
+9 State 4 STATE 135.0018 135.0018 0.000003 5 MPI
+10 State 4 STATE 135.0018 135.0018 0.000001 5 MPI
+# ... with 2,664 more rows
+#+end_example
+
+#+begin_src R :results output :session :exports both
+df %>% filter(Imbrication == 5) %>% .$Value %>% unique
+#+end_src
+
+#+RESULTS:
+: [1] "MPI"
+
+#+begin_src R :results output :session :exports both
+df %>% filter(Value == "nsi_turnon") %>% filter(Rank == 4)
+#+end_src
+
+#+RESULTS:
+: # A tibble: 1 × 8
+: Nature Rank Type Start End Duration Imbrication Value
+:
+: 1 State 4 STATE 125.57 125.6114 0.041432 2 nsi_turnon
+*** New round
+#+begin_src shell :results output
+ssh nancy.g5k cat ./alya-bsc/Executables/unix/all.scorep
+#+end_src
+
+#+RESULTS:
+#+begin_example
+#SCOREP_FILE_NAMES_BEGIN
+#EXCLUDE *
+#INCLUDE */Alya.f90
+#SCOREP_FILE_NAMES_END
+
+SCOREP_REGION_NAMES_BEGIN
+EXCLUDE *
+# Timestep begins when rank leaves the timste state
+INCLUDE timste
+# Timestep ends when rank enters the endste
+INCLUDE endste
+# doiter is the iteration itself
+INCLUDE doiter
+
+# Nastin calls
+INCLUDE nastin
+INCLUDE nsi_turnon
+INCLUDE nsi_timste
+INCLUDE nsi_iniunk
+INCLUDE nsi_begste
+INCLUDE nsi_doiter
+INCLUDE nsi_concou
+INCLUDE nsi_conblk
+INCLUDE nsi_newmsh
+INCLUDE nsi_endste
+INCLUDE nsi_filter
+INCLUDE nsi_output
+INCLUDE nsi_turnof
+SCOREP_REGION_NAMES_END
+#+end_example
+
+
+#+begin_src shell :results output
+scp nancy.g5k:./scorep_12-v1_grimoire_6_96_metis_true_v4/traces.csv.gz .
+#+end_src
+
+#+RESULTS:
+
+#+begin_src R :results output :session :exports both
+df <- read_csv("traces.csv.gz",
+ col_names=c("Nature", "Rank", "Type", "Start", "End", "Duration", "Imbrication", "Value"),
+ progress=FALSE);
+df <- df %>% mutate(Value = gsub("MPI_.*", "MPI", Value));
+startTime <- df %>% filter(Value == "timste") %>% arrange(Start) %>% slice(1) %>% .$End
+startTime
+endTime <- df %>% filter(Value == "endste") %>% arrange(End) %>% slice(n()) %>% .$End
+endTime
+maxImbrication = df %>% .$Imbrication %>% max;
+maxImbrication
+df;
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Parsed with column specification:
+cols(
+ Nature = col_character(),
+ Rank = col_integer(),
+ Type = col_character(),
+ Start = col_double(),
+ End = col_double(),
+ Duration = col_double(),
+ Imbrication = col_double(),
+ Value = col_character()
+)
+[1] 123.4664
+[1] 151.6115
+[1] 3
+# A tibble: 1,906,194 × 8
+ Nature Rank Type Start End Duration Imbrication Value
+
+1 State 95 STATE 1.110072 1.110075 0.000003 0 MPI
+2 State 95 STATE 1.110076 1.110077 0.000001 0 MPI
+3 State 95 STATE 1.110085 1.110090 0.000005 0 MPI
+4 State 95 STATE 1.110091 1.110191 0.000100 0 MPI
+5 State 95 STATE 1.110191 1.110424 0.000232 0 MPI
+6 State 95 STATE 1.110425 1.110425 0.000000 0 MPI
+7 State 95 STATE 1.110425 1.110425 0.000000 0 MPI
+8 State 95 STATE 1.110426 1.110427 0.000001 0 MPI
+9 State 95 STATE 1.110427 1.110463 0.000036 0 MPI
+10 State 95 STATE 1.110463 1.110470 0.000007 0 MPI
+# ... with 1,906,184 more rows
+#+end_example
+
+#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 1200 :height 300 :session
+library(ggplot2);
+
+plot <- NULL;
+dfplot <- df %>%
+ filter(Rank != 0) %>%
+ filter(!(Value %in% c("timste", "endste"))) %>%
+ filter(Imbrication != 5) %>%
+ filter(Rank < 5);
+dfplot %>%
+ ggplot() +
+ coord_cartesian(xlim=c(135, 145)) + #startTime, endTime)) +
+# xlim(startTime, endTime) +
+ theme_bw(base_size=12) +
+ scale_fill_brewer(palette = "Set1") -> plot;
+
+for (i in (dfplot %>% .$Imbrication %>% unique %>% sort)){
+ plot <- plot +
+ geom_rect(data=(dfplot %>% filter(Imbrication == i)), aes(fill=Value,
+ xmin=Start,
+ xmax=End,
+ ymin=Rank,
+ ymax=Rank+0.9-(Imbrication*(1/10))));
+}
+plot;
+#+end_src
+
+#+RESULTS:
+[[file:/tmp/babel-24853m4f/figure24853mXu.png]]
+
+Still not there, I can't see any function in between.
+*** New round with =nsi_solite=
+#+begin_src shell :results output
+ssh nancy.g5k cat ./alya-bsc/Executables/unix/all.scorep
+#+end_src
+
+#+RESULTS:
+#+begin_example
+#SCOREP_FILE_NAMES_BEGIN
+#EXCLUDE *
+#INCLUDE */Alya.f90
+#SCOREP_FILE_NAMES_END
+
+SCOREP_REGION_NAMES_BEGIN
+EXCLUDE *
+# Timestep begins when rank leaves the timste state
+INCLUDE timste
+# Timestep ends when rank enters the endste
+INCLUDE endste
+# doiter is the iteration itself
+INCLUDE doiter
+
+# Nastin calls
+INCLUDE nastin
+INCLUDE nsi_turnon
+INCLUDE nsi_timste
+INCLUDE nsi_iniunk
+INCLUDE nsi_begste
+INCLUDE nsi_doiter
+INCLUDE nsi_concou
+INCLUDE nsi_conblk
+INCLUDE nsi_newmsh
+INCLUDE nsi_endste
+INCLUDE nsi_filter
+INCLUDE nsi_output
+INCLUDE nsi_turnof
+
+# the solver
+INCLUDE nsi_solite
+
+SCOREP_REGION_NAMES_END
+#+end_example
+
+
+#+begin_src shell :results output
+scp nancy.g5k:./scorep_12-v1_grimoire_6_96_metis_true_v5/traces.csv.gz .
+#+end_src
+
+#+RESULTS:
+
+#+begin_src R :results output :session :exports both
+df <- read_csv("traces.csv.gz",
+ col_names=c("Nature", "Rank", "Type", "Start", "End", "Duration", "Imbrication", "Value"),
+ progress=FALSE);
+df <- df %>% mutate(Value = gsub("MPI_.*", "MPI", Value));
+startTime <- df %>% filter(Value == "timste") %>% arrange(Start) %>% slice(1) %>% .$End
+startTime
+endTime <- df %>% filter(Value == "endste") %>% arrange(End) %>% slice(n()) %>% .$End
+endTime
+maxImbrication = df %>% .$Imbrication %>% max;
+maxImbrication
+df;
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Parsed with column specification:
+cols(
+ Nature = col_character(),
+ Rank = col_integer(),
+ Type = col_character(),
+ Start = col_double(),
+ End = col_double(),
+ Duration = col_double(),
+ Imbrication = col_double(),
+ Value = col_character()
+)
+[1] 125.3257
+[1] 153.2368
+[1] 4
+# A tibble: 1,906,674 × 8
+ Nature Rank Type Start End Duration Imbrication Value
+
+1 State 95 STATE 1.025796 1.025799 0.000002 0 MPI
+2 State 95 STATE 1.025800 1.025800 0.000001 0 MPI
+3 State 95 STATE 1.025809 1.025814 0.000005 0 MPI
+4 State 95 STATE 1.025815 1.025919 0.000104 0 MPI
+5 State 95 STATE 1.025919 1.026159 0.000240 0 MPI
+6 State 95 STATE 1.026160 1.026161 0.000000 0 MPI
+7 State 95 STATE 1.026161 1.026161 0.000000 0 MPI
+8 State 95 STATE 1.026162 1.026163 0.000001 0 MPI
+9 State 95 STATE 1.026163 1.026200 0.000037 0 MPI
+10 State 95 STATE 1.026200 1.026206 0.000005 0 MPI
+# ... with 1,906,664 more rows
+#+end_example
+
+#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 1200 :height 300 :session
+library(ggplot2);
+
+plot <- NULL;
+dfplot <- df %>%
+ filter(Rank != 0) %>%
+ filter(!(Value %in% c("timste", "endste", "doiter", "nastin"))) %>%
+ filter(Imbrication != 5) %>%
+ filter(Rank < 5);
+dfplot %>%
+ ggplot() +
+ coord_cartesian(xlim=c(startTime, endTime)) +
+ theme_bw(base_size=12) +
+ scale_fill_brewer(palette = "Set1") -> plot;
+
+for (i in (dfplot %>% .$Imbrication %>% unique %>% sort)){
+ plot <- plot +
+ geom_rect(data=(dfplot %>% filter(Imbrication == i)), aes(fill=Value,
+ xmin=Start,
+ xmax=End,
+ ymin=Rank,
+ ymax=Rank+0.9-(Imbrication*(1/10))));
+}
+plot;
+#+end_src
+
+#+RESULTS:
+[[file:/tmp/babel-24853m4f/figure24853ZbQ.png]]
+
+
+We have five =nsi_solite= which matches:
+
+#+BEGIN_EXAMPLE
+--| ALYA SOLVE NASTIN (MSMSC)
+--| ALYA SOLVE NASTIN (MSMSC)
+--| ALYA SOLVE NASTIN (MSMSC)
+--| ALYA SOLVE NASTIN (MSMSC)
+--| ALYA SOLVE NASTIN (MSMSC)
+#+END_EXAMPLE
+
+There are still some MPI calls in between. Perhaps one of each MSMSC.
+*** With many functions called by nsi solite
+
+#+begin_src shell :results output
+ssh nancy.g5k cat ./alya-bsc/Executables/unix/all.scorep
+#+end_src
+
+#+RESULTS:
+#+begin_example
+#SCOREP_FILE_NAMES_BEGIN
+#EXCLUDE *
+#INCLUDE */Alya.f90
+#SCOREP_FILE_NAMES_END
+
+SCOREP_REGION_NAMES_BEGIN
+EXCLUDE *
+# Timestep begins when rank leaves the timste state
+INCLUDE timste
+# Timestep ends when rank enters the endste
+INCLUDE endste
+# doiter is the iteration itself
+INCLUDE doiter
+
+# Nastin calls
+INCLUDE nastin
+INCLUDE nsi_turnon
+INCLUDE nsi_timste
+INCLUDE nsi_iniunk
+INCLUDE nsi_begste
+INCLUDE nsi_doiter
+INCLUDE nsi_concou
+INCLUDE nsi_conblk
+INCLUDE nsi_newmsh
+INCLUDE nsi_endste
+INCLUDE nsi_filter
+INCLUDE nsi_output
+INCLUDE nsi_turnof
+
+# the solver
+INCLUDE nsi_solite
+
+# functions called by the nsi solver
+INCLUDE nsi_updbcs
+INCLUDE nsi_ifconf
+INCLUDE nsi_updunk
+INCLUDE nsi_solsgs
+INCLUDE nsi_inisol
+INCLUDE nsi_matrix
+INCLUDE nsimatndof
+INCLUDE nsi_agmgsol
+INCLUDE solver
+INCLUDE nsi_multi_step_fs_solution
+INCLUDE nsi_fractional_step_solution
+INCLUDE nsi_schur_complement_solution
+INCLUDE nsi_solver_postprocess
+
+SCOREP_REGION_NAMES_END
+#+end_example
+
+
+#+begin_src shell :results output
+scp nancy.g5k:./scorep_12-v1_grimoire_6_96_metis_true_v6/traces.csv.gz .
+#+end_src
+
+#+RESULTS:
+
+#+begin_src R :results output :session :exports both
+df <- read_csv("traces.csv.gz",
+ col_names=c("Nature", "Rank", "Type", "Start", "End", "Duration", "Imbrication", "Value"),
+ progress=FALSE);
+df <- df %>% mutate(Value = gsub("MPI_.*", "MPI", Value));
+startTime <- df %>% filter(Value == "timste") %>% arrange(Start) %>% slice(1) %>% .$End
+startTime
+endTime <- df %>% filter(Value == "endste") %>% arrange(End) %>% slice(n()) %>% .$End
+endTime
+maxImbrication = df %>% .$Imbrication %>% max;
+maxImbrication
+df;
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Parsed with column specification:
+cols(
+ Nature = col_character(),
+ Rank = col_integer(),
+ Type = col_character(),
+ Start = col_double(),
+ End = col_double(),
+ Duration = col_double(),
+ Imbrication = col_double(),
+ Value = col_character()
+)
+[1] 124.8219
+[1] 152.4296
+[1] 5
+# A tibble: 1,914,065 × 8
+ Nature Rank Type Start End Duration Imbrication Value
+
+1 State 95 STATE 1.014010 1.014012 0.000002 0 MPI
+2 State 95 STATE 1.014013 1.014014 0.000001 0 MPI
+3 State 95 STATE 1.014022 1.014027 0.000004 0 MPI
+4 State 95 STATE 1.014028 1.014128 0.000100 0 MPI
+5 State 95 STATE 1.014128 1.014358 0.000229 0 MPI
+6 State 95 STATE 1.014359 1.014359 0.000000 0 MPI
+7 State 95 STATE 1.014359 1.014359 0.000000 0 MPI
+8 State 95 STATE 1.014360 1.014361 0.000001 0 MPI
+9 State 95 STATE 1.014361 1.014398 0.000037 0 MPI
+10 State 95 STATE 1.014398 1.014405 0.000007 0 MPI
+# ... with 1,914,055 more rows
+#+end_example
+
+#+begin_src R :results output :session :exports both
+df %>% select(Imbrication, Value) %>% unique %>% as.data.frame() %>% filter(Imbrication == 4)
+#+end_src
+
+#+RESULTS:
+: Imbrication Value
+: 1 4 nsi_updbcs
+: 2 4 nsi_ifconf
+: 3 4 nsi_updunk
+: 4 4 nsi_solsgs
+: 5 4 nsi_matrix
+: 6 4 solver
+: 7 4 MPI
+: 8 4 nsi_inisol
+
+#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 1400 :height 300 :session
+library(ggplot2);
+
+plot <- NULL;
+dfplot <- df %>%
+ filter(Rank != 0) %>%
+ filter(Value %in% c(#"MPI",
+ "solver",
+ #"nsi_solite",
+ "nsi_matrix",
+ "nsi_inisol",
+ "nsi_updunk",
+ "nsi_updbcs",
+ "nsi_ifconf",
+ "nsi_updbcs")) %>% #, "nsi_matrix")) %>%
+# filter(!(Value %in% c("timste", "endste", "doiter", "nastin", "nsi_begste", "nsi_endste", "nsi_concou", "nsi_output"))) %>%
+ filter(Imbrication != 5)# %>%
+# filter(Rank < 5);
+dfplot %>%
+ ggplot() +
+ coord_cartesian(xlim=c(startTime, endTime)) +
+ xlim(startTime, endTime) +
+ theme_bw(base_size=12) +
+ scale_fill_brewer(palette = "Set1") -> plot;
+
+for (i in (dfplot %>% .$Imbrication %>% unique %>% sort)){
+ plot <- plot +
+ geom_rect(data=(dfplot %>% filter(Imbrication == i)), aes(fill=Value,
+ xmin=Start,
+ xmax=End,
+ ymin=Rank,
+ ymax=Rank+0.9-(Imbrication*(1/10))));
+}
+plot;
+#+end_src
+
+#+RESULTS:
+[[file:/tmp/babel-24853m4f/figure24853PbT.png]]
+
+
+So, for Nastin, computation time is:
+- =nsi_matrix=
+- =solver=
+- =nsi_inisol= (hardly seen)
+-
+*** Final attempt
+Check analysis of EXP12.
+*** SCOREP Filter File :ATTACH:
+:PROPERTIES:
+:Attachments: all.scorep
+:ID: ef6feaae-5f92-4706-a250-bde856794f47
+:END:
+
+#+begin_src shell :results output
+scp nancy.g5k:./alya-bsc/Executables/unix/all.scorep .
+#+end_src
+
+#+RESULTS:
+
+** 2017-05-04 Meeting D2.4 " *Report on Multilevel Programming and I/O optimizations"
+Presents: Guillaume, Rick, Lucas, Arnaud
+Link: https://rendez-vous.renater.fr/alya
+*** Introduction (Guillaume's Message)
+Guillaume's Message: I'm responsible for a deliverable of HPC4E, D2.4
+"Report on Multilevel Programming and I/O optimizations". We are going
+to report our work on SFC as a mesh partitioner and I was wondering if
+you want to report the performance analysis you did on this as
+well. No need to be extensive, one page would be okay.
+*** Discussed Topics
+**** Load imbalance (and SFC/Metis)
+When the number of points is higher, the time spent in the "solver"
+function is reduce. That surprises us because it should be the
+contrary. The number of points correspond to the number of rows in the
+matrix associated to each rank. Instead, Rick told us that we should
+look instead to the number of entries in each row; that should
+directly correlate with the workload of each rank.
+
+So we should add a new row with the number of entries associated to
+each rank: =img/exp_13_solver_partition.png=.
+
+Load imbalance is independent of Metis/SFC.
+
+At one point, when load imbalance is perfect, we shall look again to
+the communication issue we have previously seen.
+**** The one-page report
+We still have 10 days to finish the delivrable. So we can check the
+number of entries previously mentionned before writing the report.
+** 2017-05-10 Experimental session
+#+begin_src shell :results output
+oarsub -p "cluster='grisou'" -l "nodes=20,walltime=14:00:00" -r "$(date '+%Y-%m-%d 19:00:00')" -t deploy
+kadeploy3 --env-file /home/lschnorr/images/stretch_energy.env -f ${OAR_NODE_FILE} -k
+#+end_src
+** 2018-01-21 Meeting @BSC during HP4E closure
+*** Pre-meeting notes based on e-mails
+**** Partition quality using adaptation strategy: run-time measurements + unity tests (Ricard)
+The SFC partitioning process has shown to be very fast: a mesh of 30M
+elements can be partitioned in 0.05 seconds using 4K CPUs. As you say,
+next steps must be focused on the partition quality. In particular,
+I’m thinking on a partition adaptation strategy based on run-time
+measurements or some initial measurements on unity tests.
+**** Different input to check for variability (Lucas)
+Sure. Back in those days, we have instrumented the kernels that were
+used in one of the simulations to measure their run-time ie. the
+per-iteration load of each rank. The gathered information can be used
+to understand how such load evolves along the execution. As you plan,
+perhaps we can leverage on that to improve the partition quality. I
+wonder if we can play with many different inputs to check for
+variability.
+**** LB with co-execution on hybrid systems where a proper load distribution is hard to guess (Ricard+Arnaud)
+Le fait de lancer plusieurs codes (couplés ou pas) sur les mêmes
+ressources. Par exemple, dans Alya, qui fait de la multi physique, on
+peut activer plusieurs "modules" (un pour l'atmosphère, un pour
+l'océan, un pour ...) qui sont en fait des programmes autonomes mais
+couplés. Tu peux mettre un module sur une partie des ressources, un
+module sur une autre partie des ressources ou les deux modules sur
+l'ensemble des ressources. Pour moi, quand tu les met au même endroit,
+c'est un exemple de co-execution et sinon, ils se partagent quand même
+les ressources réseau. On parle de co-execution aussi au niveau batch
+scheduler quand on s'autorise à exécuter deux applis vraiment
+indépendantes sur les mêmes ressources mais je ne pense pas que ça
+soit ça dont il parle. Là, c'est plusieurs modules au sein de la même
+application et histoire de simplifier les choses ces modules utilisent
+plus ou moins les GPUs, plus ou moins en même temps selon la façon
+dont c'est programmé..
+*** Meeting
+- Partitioning is very fast, so we can run several times until the
+ quality is very good. There are two approaches to improve the
+ quality: the first one is based on using realtime measuremnets from
+ the Alya; the second is with a unity test (mimicing the Alya
+ code). The second has some disavatnages but is pretty much faster.
+- They are using a Hilbert SFC because it provides better locality.
+ - Other approaches: Peano, Morton
+- The current weight: the number of elements, mass center of the element
+ - Other approaches Ricard tried: more sofisticated approaches but it
+ is a manual approach, very slow. A runtime approach with
+ monitoring is better.
+- They read the mesh in parallel
+ - Boxes and subboxes: the distribution based on SFC and the weight
+ of each box is passed throught (acumulated) to other boxes.
+- Test cases:
+ - sniff (17.7 M elements, 6.8M nodes); \approx1K cores (60K elements per core)
+ - in memory occupation: 50K (1GBytes per core)
+ - combustor (28.9 M elements, 7M nodes); \approx2K cores
+ - 30 to 60 number of elements per core
+- Cartesian division from the initial global bounding box
+ - Each rank reads the mesh in parallel and send to each box (another
+ rank is responsible) the number of elements and their weight
+ - Starts with a coarse grid, but a finer cartesion grid used for the
+ SFC gives better initial LB. He does a very thin grid and its
+ good.
+- How to improve the SFC. There are two options
+ - You cut the line before if the weight is too much
+ - Change the cartesian in some axis
+ - Change the granularity of one box with a finer SFC
+- Arnaud (another approach): perturbate the partioning and collect
+ measurements with multiple SFC configurations (with cuts slightly
+ before and after) and then,
+
+** 2018-01-29 Prep data for Arnaud (three stable runs)
+*** Introduction
+
+We are talking about experiences:
+- EXP24 (5 iterations)
+ - It is a bit strange as we previously saw
+ - See figure in to remember
+ #+BEGIN_EXAMPLE
+ ** 4-node grisou (64 cores + 1 core) 1its, new round
+ *** Visualization / Analysis
+ **** Merge the cumsum of exp24 and exp25
+ #+END_EXAMPLE
+- EXP25 (1 iteration)
+- EXP26 (5 iterations)
+
+*** Gather data
+
+#+name: exp24_PREP
+#+header: :var dep0=exp24_enrich
+#+header: :var dep1=exp24_number_of_elements
+#+begin_src R :results output :session :exports both
+exp.ENRICH %>% left_join(exp.ELEMENTS) %>% mutate(Exp = "exp24") -> exp24.PREP
+#+end_src
+
+#+RESULTS: exp24_PREP
+: Joining, by = "Rank"
+
+#+name: exp25_PREP
+#+header: :var dep0=exp25_enrich
+#+header: :var dep1=exp25_number_of_elements
+#+begin_src R :results output :session :exports both
+exp.ENRICH %>% left_join(exp.ELEMENTS) %>% mutate(Exp = "exp25") -> exp25.PREP
+#+end_src
+
+#+RESULTS: exp25_PREP
+: Joining, by = "Rank"
+
+#+name: exp26_PREP
+#+header: :var dep0=exp26_enrich
+#+header: :var dep1=exp26_number_of_elements
+#+begin_src R :results output :session :exports both
+exp.ENRICH %>% left_join(exp.ELEMENTS) %>% mutate(Exp = "exp26") -> exp26.PREP
+#+end_src
+
+#+RESULTS: exp26_PREP
+: Joining, by = "Rank"
+
+#+begin_src R :results output :session :exports both
+exp24.PREP %>%
+ bind_rows(exp25.PREP) %>%
+ bind_rows(exp26.PREP) %>%
+ select(Phase, ID, Rank, NELEM,NPOIN,NBOUN,NPOI32,Exp) -> exp.PREP
+exp.PREP %>%
+ select(Phase, Exp) %>% unique
+#+end_src
+
+#+RESULTS:
+#+begin_example
+# A tibble: 11 x 2
+ Phase Exp
+
+ 1 1 exp24
+ 2 2 exp24
+ 3 3 exp24
+ 4 4 exp24
+ 5 5 exp24
+ 6 1 exp25
+ 7 1 exp26
+ 8 2 exp26
+ 9 3 exp26
+10 4 exp26
+11 5 exp26
+#+end_example
+
+*** Save the data here :ATTACH:
+:PROPERTIES:
+:Attachments: exp24-exp25-exp26.csv.gz
+:ID: 2c325c57-678e-491d-a7d4-f71be90dbd73
+:END:
+
+#+begin_src R :results output :session :exports both
+write_csv(exp.PREP, "exp24-exp25-exp26.csv.gz")
+#+end_src
+
+#+RESULTS:
+
+** 2018-02-06 New hypothesis Check
+*** PAST: ping-pong effects can be avoided by learning from the past (many rounds)
+
+Arnaud.
+
+*** GRANULARITY: a finer SFC granularity can reveal necessary details to LB
+
+10 iterations of the test case with 128, 256, and 512 workers, for 20
+refinement rounds, with granularity levels for each partitioner
+sub-box of 128_ip (the current def), 256_ip, and 512_ip. Please, let me
+know if you think that is enough. I also intend to keep CRITERIA
+equals to 1 (unchanged) to avoid changing everything at once, except
+if I can allocated another cluster for that specific test.
+
+*** CRITERIA: use a different =CRITERIA = 2_ip= so we consider weighted elements
+
+* Experiments
+** 2017-03-31 Grisou experiments
+#+begin_src shell :results output
+SCRIPT=grisou-script.sh
+rm -f ${SCRIPT}
+echo 'export PATH=$PATH:$HOME/install/nova/scorep-3.0-alya/bin' >> ${SCRIPT}
+echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/install/nova/scorep-3.0-alya/lib' >> ${SCRIPT}
+for CASE in sfc metis; do
+ for NP in 127 112 96 80 64 48 32 16; do
+ NP=$((NP+1))
+ UNIQUE=taurus-np${NP}-${CASE}
+ echo "./scripts/run_alya_experiment.sh machine-file ${NP} ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE"
+ echo "mv $UNIQUE /home/lschnorr/"
+ done
+done >> ${SCRIPT}
+cat ${SCRIPT}
+#+end_src
+
+#+RESULTS:
+#+begin_example
+export PATH=$PATH:$HOME/install/nova/scorep-3.0-alya/bin
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/install/nova/scorep-3.0-alya/lib
+./scripts/run_alya_experiment.sh machine-file 128 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np128-sfc
+mv taurus-np128-sfc /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 113 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np113-sfc
+mv taurus-np113-sfc /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 97 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np97-sfc
+mv taurus-np97-sfc /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 81 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np81-sfc
+mv taurus-np81-sfc /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 65 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np65-sfc
+mv taurus-np65-sfc /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 49 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np49-sfc
+mv taurus-np49-sfc /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 33 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np33-sfc
+mv taurus-np33-sfc /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 17 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np17-sfc
+mv taurus-np17-sfc /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 128 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np128-metis
+mv taurus-np128-metis /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 113 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np113-metis
+mv taurus-np113-metis /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 97 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np97-metis
+mv taurus-np97-metis /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 81 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np81-metis
+mv taurus-np81-metis /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 65 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np65-metis
+mv taurus-np65-metis /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 49 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np49-metis
+mv taurus-np49-metis /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 33 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np33-metis
+mv taurus-np33-metis /home/lschnorr/
+./scripts/run_alya_experiment.sh machine-file 17 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np17-metis
+mv taurus-np17-metis /home/lschnorr/
+#+end_example
+
+Those experiments have failed.
+
+** 2017-03-31 2-node grisou debugging experiments :EXP1:
+
+Job 1224689 @ Nancy
+
+#+begin_src shell :results output
+git clone https://gitlab.inria.fr/schnorr/Alya-Perf.git
+#+end_src
+
+Requirements
+
+- The mesh: =$HOME/node_denis/=
+- The case study: =$HOME/WORK-RICARD/=
+- Scorep 3.0 installed at: =$HOME/install/nova/scorep-3.0-alya/=
+- Alya: =$HOME/alya-bsc/Executables/unix/Alya.x=
+ - Instrumented with the aid of scorep
+ #+BEGIN_EXAMPLE
+ SCOREP=scorep --compiler --instrument-filter=/home/lschnorr/alya-bsc/Executables/unix/alya-timestep-filters.scorep
+ #+END_EXAMPLE
+ Contents of the filter
+ #+BEGIN_EXAMPLE
+SCOREP_REGION_NAMES_BEGIN
+EXCLUDE *
+INCLUDE endste
+INCLUDE timste
+SCOREP_REGION_NAMES_END
+ #+END_EXAMPLE
+
+Experimental design
+
+#+begin_src shell :results output
+CLUSTER=grisou
+NODES=2
+MACHINEFILE=machine-file
+SCRIPT="script_${CLUSTER}_${NODES}_nodes.sh"
+rm -f ${SCRIPT}
+for NP in 31 16; do
+ for CASE in sfc metis; do
+ NPN=$((NP+1))
+ UNIQUE=${CLUSTER}_${NODES}_${NPN}_${CASE}
+ echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE"
+ echo "mv $UNIQUE.dir /home/lschnorr/"
+ done
+done >> ${SCRIPT}
+chmod 755 ${SCRIPT}
+cat ${SCRIPT}
+#+end_src
+
+#+RESULTS:
+: ./scripts/run_alya_experiment.sh machine-file 32 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat grisou_2_32_sfc
+: mv grisou_2_32_sfc.dir /home/lschnorr/
+: ./scripts/run_alya_experiment.sh machine-file 32 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat grisou_2_32_metis
+: mv grisou_2_32_metis.dir /home/lschnorr/
+: ./scripts/run_alya_experiment.sh machine-file 17 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat grisou_2_17_sfc
+: mv grisou_2_17_sfc.dir /home/lschnorr/
+: ./scripts/run_alya_experiment.sh machine-file 17 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat grisou_2_17_metis
+: mv grisou_2_17_metis.dir /home/lschnorr/
+** 2017-03-31 10-node grisou debugging experiments :EXP2:EXP3:
+
+Same as previous entry, but with 10 nodes.
+
+Job 1224741 @ Nancy
+
+Kadeploy the image:
+=/home/lschnorr/images/stretch_energy_v6_2017_03_30.tgz=
+
+Same requirements as previous entry.
+
+Experimental design
+
+#+begin_src shell :results output
+EID=3
+CLUSTER=grisou
+NODES=10
+MACHINEFILE=machine-file
+G5KHOME=/home/lschnorr/
+SCRIPT="script_${CLUSTER}_${NODES}_nodes.sh"
+
+rm -f $SCRIPT
+
+EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
+echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
+echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
+echo " exit" >> $SCRIPT
+echo "fi" >> $SCRIPT
+
+# Create experimental directory in the home
+echo "mkdir -p $EDIR" >> $SCRIPT
+
+# The design
+for NP in 159 144 128 112 96; do
+ for CASE in sfc metis; do
+ NPN=$((NP+1))
+ UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}
+ echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE"
+ echo "cp -prfv $UNIQUE.dir ${EDIR}"
+ done
+done >> ${SCRIPT}
+echo "mv $SCRIPT $EDIR" >> $SCRIPT
+chmod 755 ${SCRIPT}
+cat ${SCRIPT}
+echo "This script is at '${SCRIPT}', commit it."
+#+end_src
+
+#+RESULTS:
+#+begin_example
+if [ -e /home/lschnorr//exp_3_grisou_10 ]; then
+ echo "Directory/File '/home/lschnorr//exp_3_grisou_10' already exists. Remove it or rename it."
+ exit
+fi
+mkdir -p /home/lschnorr//exp_3_grisou_10
+./scripts/run_alya_experiment.sh machine-file 160 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 3_grisou_10_160_sfc
+cp -prfv 3_grisou_10_160_sfc.dir /home/lschnorr//exp_3_grisou_10
+./scripts/run_alya_experiment.sh machine-file 160 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 3_grisou_10_160_metis
+cp -prfv 3_grisou_10_160_metis.dir /home/lschnorr//exp_3_grisou_10
+./scripts/run_alya_experiment.sh machine-file 145 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 3_grisou_10_145_sfc
+cp -prfv 3_grisou_10_145_sfc.dir /home/lschnorr//exp_3_grisou_10
+./scripts/run_alya_experiment.sh machine-file 145 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 3_grisou_10_145_metis
+cp -prfv 3_grisou_10_145_metis.dir /home/lschnorr//exp_3_grisou_10
+./scripts/run_alya_experiment.sh machine-file 129 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 3_grisou_10_129_sfc
+cp -prfv 3_grisou_10_129_sfc.dir /home/lschnorr//exp_3_grisou_10
+./scripts/run_alya_experiment.sh machine-file 129 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 3_grisou_10_129_metis
+cp -prfv 3_grisou_10_129_metis.dir /home/lschnorr//exp_3_grisou_10
+./scripts/run_alya_experiment.sh machine-file 113 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 3_grisou_10_113_sfc
+cp -prfv 3_grisou_10_113_sfc.dir /home/lschnorr//exp_3_grisou_10
+./scripts/run_alya_experiment.sh machine-file 113 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 3_grisou_10_113_metis
+cp -prfv 3_grisou_10_113_metis.dir /home/lschnorr//exp_3_grisou_10
+./scripts/run_alya_experiment.sh machine-file 97 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 3_grisou_10_97_sfc
+cp -prfv 3_grisou_10_97_sfc.dir /home/lschnorr//exp_3_grisou_10
+./scripts/run_alya_experiment.sh machine-file 97 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 3_grisou_10_97_metis
+cp -prfv 3_grisou_10_97_metis.dir /home/lschnorr//exp_3_grisou_10
+mv script_grisou_10_nodes.sh /home/lschnorr//exp_3_grisou_10
+This script is at 'script_grisou_10_nodes.sh', commit it.
+#+end_example
+
+Faster way to copy is to tar (without compression) all files, and then
+copy using (not the g5kg)
+#+BEGIN_EXAMPLE
+rsync --progress grisou-51.nancy.g5kg:~/exp_2_grisou_10.tar .
+#+END_EXAMPLE
+
+Two replications have been made:
+- exp 2
+- exp 3
+** 2017-04-03 23-node (368 cores) nova in April 3rd :EXP4:
+*** Preparation for nova (prep-4-v[12])
+
+#+begin_src shell :results output
+EID=prep-4
+CLUSTER=grisou
+NODES=4
+MACHINEFILE=machine-file
+G5KHOME=/home/lschnorr/
+SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
+
+rm -f $SCRIPT
+
+EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
+echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
+echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
+echo " exit" >> $SCRIPT
+echo "fi" >> $SCRIPT
+
+# Create experimental directory in the home
+echo "mkdir -p $EDIR" >> $SCRIPT
+
+# The design
+for NP in 62 48 32 17; do
+ for CASE in sfc metis; do
+ NPN=$((NP+1))
+ UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}
+ echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE"
+ echo "cp -prfv $UNIQUE.dir ${EDIR}"
+ done
+done >> ${SCRIPT}
+echo "mv $SCRIPT $EDIR" >> $SCRIPT
+chmod 755 ${SCRIPT}
+cat ${SCRIPT}
+echo "This script is at '${SCRIPT}', commit it."
+#+end_src
+
+#+RESULTS:
+#+begin_example
+if [ -e /home/lschnorr//exp_prep-4_grisou_4 ]; then
+ echo "Directory/File '/home/lschnorr//exp_prep-4_grisou_4' already exists. Remove it or rename it."
+ exit
+fi
+mkdir -p /home/lschnorr//exp_prep-4_grisou_4
+./scripts/run_alya_experiment.sh machine-file 63 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat prep-4_grisou_4_63_sfc
+cp -prfv prep-4_grisou_4_63_sfc.dir /home/lschnorr//exp_prep-4_grisou_4
+./scripts/run_alya_experiment.sh machine-file 63 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat prep-4_grisou_4_63_metis
+cp -prfv prep-4_grisou_4_63_metis.dir /home/lschnorr//exp_prep-4_grisou_4
+./scripts/run_alya_experiment.sh machine-file 49 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat prep-4_grisou_4_49_sfc
+cp -prfv prep-4_grisou_4_49_sfc.dir /home/lschnorr//exp_prep-4_grisou_4
+./scripts/run_alya_experiment.sh machine-file 49 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat prep-4_grisou_4_49_metis
+cp -prfv prep-4_grisou_4_49_metis.dir /home/lschnorr//exp_prep-4_grisou_4
+./scripts/run_alya_experiment.sh machine-file 33 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat prep-4_grisou_4_33_sfc
+cp -prfv prep-4_grisou_4_33_sfc.dir /home/lschnorr//exp_prep-4_grisou_4
+./scripts/run_alya_experiment.sh machine-file 33 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat prep-4_grisou_4_33_metis
+cp -prfv prep-4_grisou_4_33_metis.dir /home/lschnorr//exp_prep-4_grisou_4
+./scripts/run_alya_experiment.sh machine-file 18 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat prep-4_grisou_4_18_sfc
+cp -prfv prep-4_grisou_4_18_sfc.dir /home/lschnorr//exp_prep-4_grisou_4
+./scripts/run_alya_experiment.sh machine-file 18 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat prep-4_grisou_4_18_metis
+cp -prfv prep-4_grisou_4_18_metis.dir /home/lschnorr//exp_prep-4_grisou_4
+mv script_prep-4_grisou_4_nodes.sh /home/lschnorr//exp_prep-4_grisou_4
+This script is at 'script_prep-4_grisou_4_nodes.sh', commit it.
+#+end_example
+*** Full 23-node nova
+
+JobId 854496 @ Lyon.
+
+#+begin_src shell :results output
+EID=4
+CLUSTER=grisou
+NODES=23
+MACHINEFILE=machine-file
+G5KHOME=/home/lschnorr/
+SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
+
+rm -f $SCRIPT
+
+EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
+echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
+echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
+echo " exit" >> $SCRIPT
+echo "fi" >> $SCRIPT
+
+# Create experimental directory in the home
+echo "mkdir -p $EDIR" >> $SCRIPT
+
+# The design
+for NP in 367 352 336 272 208 144 80 16; do
+ for CASE in sfc metis; do
+ NPN=$((NP+1))
+ UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}
+ echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE"
+ echo "cp -prfv $UNIQUE.dir ${EDIR}"
+ done
+done >> ${SCRIPT}
+echo "cp $SCRIPT $EDIR" >> $SCRIPT
+chmod 755 ${SCRIPT}
+cat ${SCRIPT}
+echo "This script is at '${SCRIPT}', commit it."
+#+end_src
+
+#+RESULTS:
+#+begin_example
+if [ -e /home/lschnorr//exp_4_grisou_23 ]; then
+ echo "Directory/File '/home/lschnorr//exp_4_grisou_23' already exists. Remove it or rename it."
+ exit
+fi
+mkdir -p /home/lschnorr//exp_4_grisou_23
+./scripts/run_alya_experiment.sh machine-file 368 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 4_grisou_23_368_sfc
+cp -prfv 4_grisou_23_368_sfc.dir /home/lschnorr//exp_4_grisou_23
+./scripts/run_alya_experiment.sh machine-file 368 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 4_grisou_23_368_metis
+cp -prfv 4_grisou_23_368_metis.dir /home/lschnorr//exp_4_grisou_23
+./scripts/run_alya_experiment.sh machine-file 353 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 4_grisou_23_353_sfc
+cp -prfv 4_grisou_23_353_sfc.dir /home/lschnorr//exp_4_grisou_23
+./scripts/run_alya_experiment.sh machine-file 353 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 4_grisou_23_353_metis
+cp -prfv 4_grisou_23_353_metis.dir /home/lschnorr//exp_4_grisou_23
+./scripts/run_alya_experiment.sh machine-file 337 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 4_grisou_23_337_sfc
+cp -prfv 4_grisou_23_337_sfc.dir /home/lschnorr//exp_4_grisou_23
+./scripts/run_alya_experiment.sh machine-file 337 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 4_grisou_23_337_metis
+cp -prfv 4_grisou_23_337_metis.dir /home/lschnorr//exp_4_grisou_23
+./scripts/run_alya_experiment.sh machine-file 273 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 4_grisou_23_273_sfc
+cp -prfv 4_grisou_23_273_sfc.dir /home/lschnorr//exp_4_grisou_23
+./scripts/run_alya_experiment.sh machine-file 273 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 4_grisou_23_273_metis
+cp -prfv 4_grisou_23_273_metis.dir /home/lschnorr//exp_4_grisou_23
+./scripts/run_alya_experiment.sh machine-file 209 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 4_grisou_23_209_sfc
+cp -prfv 4_grisou_23_209_sfc.dir /home/lschnorr//exp_4_grisou_23
+./scripts/run_alya_experiment.sh machine-file 209 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 4_grisou_23_209_metis
+cp -prfv 4_grisou_23_209_metis.dir /home/lschnorr//exp_4_grisou_23
+./scripts/run_alya_experiment.sh machine-file 145 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 4_grisou_23_145_sfc
+cp -prfv 4_grisou_23_145_sfc.dir /home/lschnorr//exp_4_grisou_23
+./scripts/run_alya_experiment.sh machine-file 145 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 4_grisou_23_145_metis
+cp -prfv 4_grisou_23_145_metis.dir /home/lschnorr//exp_4_grisou_23
+./scripts/run_alya_experiment.sh machine-file 81 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 4_grisou_23_81_sfc
+cp -prfv 4_grisou_23_81_sfc.dir /home/lschnorr//exp_4_grisou_23
+./scripts/run_alya_experiment.sh machine-file 81 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 4_grisou_23_81_metis
+cp -prfv 4_grisou_23_81_metis.dir /home/lschnorr//exp_4_grisou_23
+./scripts/run_alya_experiment.sh machine-file 17 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 4_grisou_23_17_sfc
+cp -prfv 4_grisou_23_17_sfc.dir /home/lschnorr//exp_4_grisou_23
+./scripts/run_alya_experiment.sh machine-file 17 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 4_grisou_23_17_metis
+cp -prfv 4_grisou_23_17_metis.dir /home/lschnorr//exp_4_grisou_23
+cp script_4_grisou_23_nodes.sh /home/lschnorr//exp_4_grisou_23
+This script is at 'script_4_grisou_23_nodes.sh', commit it.
+#+end_example
+
+** 2017-04-03 50-node (800 cores) grisou in April 4th :EXP5:
+#+begin_src shell :results output
+EID=5
+VERSION=1
+CLUSTER=grisou
+NODES=50
+MACHINEFILE=machine-file
+G5KHOME=/home/lschnorr/
+SCRIPT="script_${EID}_v${VERSION}_${CLUSTER}_${NODES}_nodes.sh"
+
+rm -f $SCRIPT
+
+EDIR="$G5KHOME/exp_${EID}-v${VERSION}_${CLUSTER}_${NODES}"
+echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
+echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
+echo " exit" >> $SCRIPT
+echo "fi" >> $SCRIPT
+
+# Create experimental directory in the home
+echo "mkdir -p $EDIR" >> $SCRIPT
+
+# The design
+for NP in 800 720 640 560 480 400; do
+ for CASE in sfc metis; do
+ NPN=$((NP))
+ UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}
+ echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE false"
+ echo "cp -prfv $UNIQUE.dir ${EDIR}"
+ done
+done >> ${SCRIPT}
+echo "cp $SCRIPT $EDIR" >> $SCRIPT
+chmod 755 ${SCRIPT}
+cat ${SCRIPT}
+echo "This script is at '${SCRIPT}', commit it."
+#+end_src
+
+#+RESULTS:
+#+begin_example
+if [ -e /home/lschnorr//exp_5-v1_grisou_50 ]; then
+ echo "Directory/File '/home/lschnorr//exp_5-v1_grisou_50' already exists. Remove it or rename it."
+ exit
+fi
+mkdir -p /home/lschnorr//exp_5-v1_grisou_50
+./scripts/run_alya_experiment.sh machine-file 800 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 5_grisou_50_800_sfc false
+cp -prfv 5_grisou_50_800_sfc.dir /home/lschnorr//exp_5-v1_grisou_50
+./scripts/run_alya_experiment.sh machine-file 800 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 5_grisou_50_800_metis false
+cp -prfv 5_grisou_50_800_metis.dir /home/lschnorr//exp_5-v1_grisou_50
+./scripts/run_alya_experiment.sh machine-file 720 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 5_grisou_50_720_sfc false
+cp -prfv 5_grisou_50_720_sfc.dir /home/lschnorr//exp_5-v1_grisou_50
+./scripts/run_alya_experiment.sh machine-file 720 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 5_grisou_50_720_metis false
+cp -prfv 5_grisou_50_720_metis.dir /home/lschnorr//exp_5-v1_grisou_50
+./scripts/run_alya_experiment.sh machine-file 640 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 5_grisou_50_640_sfc false
+cp -prfv 5_grisou_50_640_sfc.dir /home/lschnorr//exp_5-v1_grisou_50
+./scripts/run_alya_experiment.sh machine-file 640 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 5_grisou_50_640_metis false
+cp -prfv 5_grisou_50_640_metis.dir /home/lschnorr//exp_5-v1_grisou_50
+./scripts/run_alya_experiment.sh machine-file 560 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 5_grisou_50_560_sfc false
+cp -prfv 5_grisou_50_560_sfc.dir /home/lschnorr//exp_5-v1_grisou_50
+./scripts/run_alya_experiment.sh machine-file 560 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 5_grisou_50_560_metis false
+cp -prfv 5_grisou_50_560_metis.dir /home/lschnorr//exp_5-v1_grisou_50
+./scripts/run_alya_experiment.sh machine-file 480 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 5_grisou_50_480_sfc false
+cp -prfv 5_grisou_50_480_sfc.dir /home/lschnorr//exp_5-v1_grisou_50
+./scripts/run_alya_experiment.sh machine-file 480 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 5_grisou_50_480_metis false
+cp -prfv 5_grisou_50_480_metis.dir /home/lschnorr//exp_5-v1_grisou_50
+./scripts/run_alya_experiment.sh machine-file 400 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 5_grisou_50_400_sfc false
+cp -prfv 5_grisou_50_400_sfc.dir /home/lschnorr//exp_5-v1_grisou_50
+./scripts/run_alya_experiment.sh machine-file 400 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 5_grisou_50_400_metis false
+cp -prfv 5_grisou_50_400_metis.dir /home/lschnorr//exp_5-v1_grisou_50
+cp script_5_v1_grisou_50_nodes.sh /home/lschnorr//exp_5-v1_grisou_50
+This script is at 'script_5_v1_grisou_50_nodes.sh', commit it.
+#+end_example
+
+** 2017-04-03 7-node (112 cores) grimoire with Infiniband :EXP6:
+#+begin_src shell :results output
+EID=6
+CLUSTER=grimoire
+NODES=7
+MACHINEFILE=machine-file
+G5KHOME=/home/lschnorr/
+SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
+
+rm -f $SCRIPT
+
+EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
+echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
+echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
+echo " exit" >> $SCRIPT
+echo "fi" >> $SCRIPT
+
+# Create experimental directory in the home
+echo "mkdir -p $EDIR" >> $SCRIPT
+
+# The design
+for NP in 111 96; do
+ for CASE in sfc metis; do
+ NPN=$((NP+1))
+ UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}
+ echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE true"
+ echo "cp -prfv $UNIQUE.dir ${EDIR}"
+ done
+done >> ${SCRIPT}
+echo "cp $SCRIPT $EDIR" >> $SCRIPT
+chmod 755 ${SCRIPT}
+cat ${SCRIPT}
+echo "This script is at '${SCRIPT}', commit it."
+#+end_src
+
+#+RESULTS:
+#+begin_example
+if [ -e /home/lschnorr//exp_6_grimoire_7 ]; then
+ echo "Directory/File '/home/lschnorr//exp_6_grimoire_7' already exists. Remove it or rename it."
+ exit
+fi
+mkdir -p /home/lschnorr//exp_6_grimoire_7
+./scripts/run_alya_experiment.sh machine-file 112 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 6_grimoire_7_112_sfc true
+cp -prfv 6_grimoire_7_112_sfc.dir /home/lschnorr//exp_6_grimoire_7
+./scripts/run_alya_experiment.sh machine-file 112 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 6_grimoire_7_112_metis true
+cp -prfv 6_grimoire_7_112_metis.dir /home/lschnorr//exp_6_grimoire_7
+./scripts/run_alya_experiment.sh machine-file 97 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 6_grimoire_7_97_sfc true
+cp -prfv 6_grimoire_7_97_sfc.dir /home/lschnorr//exp_6_grimoire_7
+./scripts/run_alya_experiment.sh machine-file 97 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 6_grimoire_7_97_metis true
+cp -prfv 6_grimoire_7_97_metis.dir /home/lschnorr//exp_6_grimoire_7
+cp script_6_grimoire_7_nodes.sh /home/lschnorr//exp_6_grimoire_7
+This script is at 'script_6_grimoire_7_nodes.sh', commit it.
+#+end_example
+
+** 2017-04-04 4-node (64 cores) grimoire with Infiniband :EXP7:
+
+The goal is simply confirm experimentally that OpenMPI is doing openib
+communications instead of ethernet when told so.
+
+#+begin_src shell :results output
+EID=7
+CLUSTER=grimoire
+NODES=4
+MACHINEFILE=machine-file
+G5KHOME=/home/lschnorr/
+SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
+
+rm -f $SCRIPT
+
+EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
+echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
+echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
+echo " exit" >> $SCRIPT
+echo "fi" >> $SCRIPT
+
+# Create experimental directory in the home
+echo "mkdir -p $EDIR" >> $SCRIPT
+
+# The design
+NPN=112
+for INFINIBAND in true false; do
+ for CASE in sfc metis; do
+ UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}
+ echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
+ echo "cp -prfv $UNIQUE.dir ${EDIR}"
+ done
+done >> ${SCRIPT}
+echo "cp $SCRIPT $EDIR" >> $SCRIPT
+chmod 755 ${SCRIPT}
+cat ${SCRIPT}
+echo "This script is at '${SCRIPT}', commit it."
+#+end_src
+
+#+RESULTS:
+#+begin_example
+if [ -e /home/lschnorr//exp_7_grimoire_4 ]; then
+ echo "Directory/File '/home/lschnorr//exp_7_grimoire_4' already exists. Remove it or rename it."
+ exit
+fi
+mkdir -p /home/lschnorr//exp_7_grimoire_4
+./scripts/run_alya_experiment.sh machine-file 112 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 7_grimoire_4_112_sfc true
+cp -prfv 7_grimoire_4_112_sfc.dir /home/lschnorr//exp_7_grimoire_4
+./scripts/run_alya_experiment.sh machine-file 112 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 7_grimoire_4_112_metis true
+cp -prfv 7_grimoire_4_112_metis.dir /home/lschnorr//exp_7_grimoire_4
+./scripts/run_alya_experiment.sh machine-file 112 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 7_grimoire_4_112_sfc false
+cp -prfv 7_grimoire_4_112_sfc.dir /home/lschnorr//exp_7_grimoire_4
+./scripts/run_alya_experiment.sh machine-file 112 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 7_grimoire_4_112_metis false
+cp -prfv 7_grimoire_4_112_metis.dir /home/lschnorr//exp_7_grimoire_4
+cp script_7_grimoire_4_nodes.sh /home/lschnorr//exp_7_grimoire_4
+This script is at 'script_7_grimoire_4_nodes.sh', commit it.
+#+end_example
+
+** 2017-04-05 2-node (32 cores) and 4-node (64 cores) grimoire to test Infiniband :EXP8:
+#+begin_src shell :results output
+EID=8
+VERSION=4
+CLUSTER=grimoire
+NODES=4
+MACHINEFILE=machine-file
+G5KHOME=/home/lschnorr/
+EID=${EID}-v${VERSION}
+SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
+
+rm -f $SCRIPT
+
+EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
+echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
+echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
+echo " exit" >> $SCRIPT
+echo "fi" >> $SCRIPT
+
+# Create experimental directory in the home
+echo "mkdir -p $EDIR" >> $SCRIPT
+
+# The design
+NP=60
+for INFINIBAND in true false; do
+ for CASE in metis sfc; do
+ NPN=$((NP))
+ UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}_${INFINIBAND}
+ echo "# "
+ echo "# Experiment $UNIQUE"
+ echo "# "
+ echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
+ echo "cp -prfv $UNIQUE.dir ${EDIR}"
+ echo "rm -rf $UNIQUE.dir"
+ done
+done >> ${SCRIPT}
+echo "# " >> ${SCRIPT}
+echo "# Final" >> ${SCRIPT}
+echo "# " >> ${SCRIPT}
+echo "cp $SCRIPT $EDIR" >> $SCRIPT
+chmod 755 ${SCRIPT}
+cat ${SCRIPT}
+echo "This script is at '${SCRIPT}', commit it."
+#+end_src
+
+#+RESULTS:
+#+begin_example
+if [ -e /home/lschnorr//exp_8-v4_grimoire_4 ]; then
+ echo "Directory/File '/home/lschnorr//exp_8-v4_grimoire_4' already exists. Remove it or rename it."
+ exit
+fi
+mkdir -p /home/lschnorr//exp_8-v4_grimoire_4
+#
+# Experiment 8-v4_grimoire_4_60_metis_true
+#
+./scripts/run_alya_experiment.sh machine-file 60 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 8-v4_grimoire_4_60_metis_true true
+cp -prfv 8-v4_grimoire_4_60_metis_true.dir /home/lschnorr//exp_8-v4_grimoire_4
+rm -rf 8-v4_grimoire_4_60_metis_true.dir
+#
+# Experiment 8-v4_grimoire_4_60_sfc_true
+#
+./scripts/run_alya_experiment.sh machine-file 60 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 8-v4_grimoire_4_60_sfc_true true
+cp -prfv 8-v4_grimoire_4_60_sfc_true.dir /home/lschnorr//exp_8-v4_grimoire_4
+rm -rf 8-v4_grimoire_4_60_sfc_true.dir
+#
+# Experiment 8-v4_grimoire_4_60_metis_false
+#
+./scripts/run_alya_experiment.sh machine-file 60 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 8-v4_grimoire_4_60_metis_false false
+cp -prfv 8-v4_grimoire_4_60_metis_false.dir /home/lschnorr//exp_8-v4_grimoire_4
+rm -rf 8-v4_grimoire_4_60_metis_false.dir
+#
+# Experiment 8-v4_grimoire_4_60_sfc_false
+#
+./scripts/run_alya_experiment.sh machine-file 60 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 8-v4_grimoire_4_60_sfc_false false
+cp -prfv 8-v4_grimoire_4_60_sfc_false.dir /home/lschnorr//exp_8-v4_grimoire_4
+rm -rf 8-v4_grimoire_4_60_sfc_false.dir
+#
+# Final
+#
+cp script_8-v4_grimoire_4_nodes.sh /home/lschnorr//exp_8-v4_grimoire_4
+This script is at 'script_8-v4_grimoire_4_nodes.sh', commit it.
+#+end_example
+
+** 2017-04-05 6-node grimoire with Infiniband :EXP9:
+#+begin_src shell :results output
+EID=9
+VERSION=1
+CLUSTER=grimoire
+NODES=6
+MACHINEFILE=machine-file
+G5KHOME=/home/lschnorr/
+EID=${EID}-v${VERSION}
+SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
+
+rm -f $SCRIPT
+
+EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
+echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
+echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
+echo " exit" >> $SCRIPT
+echo "fi" >> $SCRIPT
+
+# Create experimental directory in the home
+echo "mkdir -p $EDIR" >> $SCRIPT
+
+# The design
+NP=96
+for INFINIBAND in true false; do
+ for CASE in metis sfc; do
+ NPN=$((NP))
+ UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}_${INFINIBAND}
+ echo "# "
+ echo "# Experiment $UNIQUE"
+ echo "# "
+ echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
+ echo "cp -prfv $UNIQUE.dir ${EDIR}"
+ echo "rm -rf $UNIQUE.dir"
+ done
+done >> ${SCRIPT}
+echo "# " >> ${SCRIPT}
+echo "# Final" >> ${SCRIPT}
+echo "# " >> ${SCRIPT}
+echo "cp $SCRIPT $EDIR" >> $SCRIPT
+chmod 755 ${SCRIPT}
+cat ${SCRIPT}
+echo "This script is at '${SCRIPT}', commit it."
+#+end_src
+
+#+RESULTS:
+#+begin_example
+if [ -e /home/lschnorr//exp_9-v1_grimoire_6 ]; then
+ echo "Directory/File '/home/lschnorr//exp_9-v1_grimoire_6' already exists. Remove it or rename it."
+ exit
+fi
+mkdir -p /home/lschnorr//exp_9-v1_grimoire_6
+#
+# Experiment 9-v1_grimoire_6_96_metis_true
+#
+./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 9-v1_grimoire_6_96_metis_true true
+cp -prfv 9-v1_grimoire_6_96_metis_true.dir /home/lschnorr//exp_9-v1_grimoire_6
+rm -rf 9-v1_grimoire_6_96_metis_true.dir
+#
+# Experiment 9-v1_grimoire_6_96_sfc_true
+#
+./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 9-v1_grimoire_6_96_sfc_true true
+cp -prfv 9-v1_grimoire_6_96_sfc_true.dir /home/lschnorr//exp_9-v1_grimoire_6
+rm -rf 9-v1_grimoire_6_96_sfc_true.dir
+#
+# Experiment 9-v1_grimoire_6_96_metis_false
+#
+./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 9-v1_grimoire_6_96_metis_false false
+cp -prfv 9-v1_grimoire_6_96_metis_false.dir /home/lschnorr//exp_9-v1_grimoire_6
+rm -rf 9-v1_grimoire_6_96_metis_false.dir
+#
+# Experiment 9-v1_grimoire_6_96_sfc_false
+#
+./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 9-v1_grimoire_6_96_sfc_false false
+cp -prfv 9-v1_grimoire_6_96_sfc_false.dir /home/lschnorr//exp_9-v1_grimoire_6
+rm -rf 9-v1_grimoire_6_96_sfc_false.dir
+#
+# Final
+#
+cp script_9-v1_grimoire_6_nodes.sh /home/lschnorr//exp_9-v1_grimoire_6
+This script is at 'script_9-v1_grimoire_6_nodes.sh', commit it.
+#+end_example
+
+** 2017-04-06 8-node grimoire infiniband-only after Alya modifications :EXP10:
+#+begin_src shell :results output
+EID=10
+VERSION=1
+CLUSTER=grimoire
+NODES=8
+MACHINEFILE=machine-file
+G5KHOME=/home/lschnorr/
+EID=${EID}-v${VERSION}
+SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
+
+rm -f $SCRIPT
+
+EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
+echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
+echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
+echo " exit" >> $SCRIPT
+echo "fi" >> $SCRIPT
+
+# Create experimental directory in the home
+echo "mkdir -p $EDIR" >> $SCRIPT
+
+# The design
+NP=128
+for INFINIBAND in true; do
+ for CASE in metis sfc; do
+ NPN=$((NP))
+ UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}_${INFINIBAND}
+ echo "# "
+ echo "# Experiment $UNIQUE"
+ echo "# "
+ echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
+ echo "cp -prfv $UNIQUE.dir ${EDIR}"
+ echo "rm -rf $UNIQUE.dir"
+ done
+done >> ${SCRIPT}
+echo "# " >> ${SCRIPT}
+echo "# Final" >> ${SCRIPT}
+echo "# " >> ${SCRIPT}
+echo "cp $SCRIPT $EDIR" >> $SCRIPT
+chmod 755 ${SCRIPT}
+cat ${SCRIPT}
+echo "This script is at '${SCRIPT}', commit it."
+#+end_src
+
+#+RESULTS:
+#+begin_example
+if [ -e /home/lschnorr//exp_10-v1_grimoire_8 ]; then
+ echo "Directory/File '/home/lschnorr//exp_10-v1_grimoire_8' already exists. Remove it or rename it."
+ exit
+fi
+mkdir -p /home/lschnorr//exp_10-v1_grimoire_8
+#
+# Experiment 10-v1_grimoire_8_128_metis_true
+#
+./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 10-v1_grimoire_8_128_metis_true true
+cp -prfv 10-v1_grimoire_8_128_metis_true.dir /home/lschnorr//exp_10-v1_grimoire_8
+rm -rf 10-v1_grimoire_8_128_metis_true.dir
+#
+# Experiment 10-v1_grimoire_8_128_sfc_true
+#
+./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 10-v1_grimoire_8_128_sfc_true true
+cp -prfv 10-v1_grimoire_8_128_sfc_true.dir /home/lschnorr//exp_10-v1_grimoire_8
+rm -rf 10-v1_grimoire_8_128_sfc_true.dir
+#
+# Final
+#
+cp script_10-v1_grimoire_8_nodes.sh /home/lschnorr//exp_10-v1_grimoire_8
+This script is at 'script_10-v1_grimoire_8_nodes.sh', commit it.
+#+end_example
+
+** 2017-04-06 8-node grimoire infiniband-only without Alya modifications :EXP11:
+I will compile Alya before the partitioning weight modifications.
+#+begin_src shell :results output
+EID=11
+VERSION=1
+CLUSTER=grimoire
+NODES=8
+MACHINEFILE=machine-file
+G5KHOME=/home/lschnorr/
+EID=${EID}-v${VERSION}
+SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
+
+rm -f $SCRIPT
+
+EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
+echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
+echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
+echo " exit" >> $SCRIPT
+echo "fi" >> $SCRIPT
+
+# Create experimental directory in the home
+echo "mkdir -p $EDIR" >> $SCRIPT
+
+# The design
+NP=128
+for INFINIBAND in true; do
+ for CASE in metis sfc; do
+ NPN=$((NP))
+ UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}_${INFINIBAND}
+ echo "# "
+ echo "# Experiment $UNIQUE"
+ echo "# "
+ echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
+ echo "cp -prfv $UNIQUE.dir ${EDIR}"
+ echo "rm -rf $UNIQUE.dir"
+ done
+done >> ${SCRIPT}
+echo "# " >> ${SCRIPT}
+echo "# Final" >> ${SCRIPT}
+echo "# " >> ${SCRIPT}
+echo "cp $SCRIPT $EDIR" >> $SCRIPT
+chmod 755 ${SCRIPT}
+cat ${SCRIPT}
+echo "This script is at '${SCRIPT}', commit it."
+#+end_src
+
+#+RESULTS:
+#+begin_example
+if [ -e /home/lschnorr//exp_11-v1_grimoire_8 ]; then
+ echo "Directory/File '/home/lschnorr//exp_11-v1_grimoire_8' already exists. Remove it or rename it."
+ exit
+fi
+mkdir -p /home/lschnorr//exp_11-v1_grimoire_8
+#
+# Experiment 11-v1_grimoire_8_128_metis_true
+#
+./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 11-v1_grimoire_8_128_metis_true true
+cp -prfv 11-v1_grimoire_8_128_metis_true.dir /home/lschnorr//exp_11-v1_grimoire_8
+rm -rf 11-v1_grimoire_8_128_metis_true.dir
+#
+# Experiment 11-v1_grimoire_8_128_sfc_true
+#
+./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 11-v1_grimoire_8_128_sfc_true true
+cp -prfv 11-v1_grimoire_8_128_sfc_true.dir /home/lschnorr//exp_11-v1_grimoire_8
+rm -rf 11-v1_grimoire_8_128_sfc_true.dir
+#
+# Final
+#
+cp script_11-v1_grimoire_8_nodes.sh /home/lschnorr//exp_11-v1_grimoire_8
+This script is at 'script_11-v1_grimoire_8_nodes.sh', commit it.
+#+end_example
+
+** 2017-04-07 6-node grimoire (no Alya modifications, full computing instrumentations) :EXP12:
+
+For this experiment, we shall:
+- remove MPI instrumentation completely
+- adopt compiler instrumentation to obtain some application functions
+
+#+begin_src shell :results output
+EID=12
+VERSION=1
+CLUSTER=grimoire
+NODES=6
+MACHINEFILE=machine-file
+G5KHOME=/home/lschnorr/
+EID=${EID}-v${VERSION}
+SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
+
+rm -f $SCRIPT
+
+EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
+echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
+echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
+echo " exit" >> $SCRIPT
+echo "fi" >> $SCRIPT
+
+# Create experimental directory in the home
+echo "mkdir -p $EDIR" >> $SCRIPT
+
+# The design
+for NP in 96 48; do
+ for INFINIBAND in true; do
+ for CASE in metis sfc; do
+ NPN=$((NP))
+ UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}_${INFINIBAND}
+ echo "# "
+ echo "# Experiment $UNIQUE"
+ echo "# "
+ echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
+ echo "cp -prfv $UNIQUE.dir ${EDIR}"
+ echo "rm -rf $UNIQUE.dir"
+ done
+ done
+done >> ${SCRIPT}
+echo "# " >> ${SCRIPT}
+echo "# Final" >> ${SCRIPT}
+echo "# " >> ${SCRIPT}
+echo "cp $SCRIPT $EDIR" >> $SCRIPT
+chmod 755 ${SCRIPT}
+cat ${SCRIPT}
+echo "This script is at '${SCRIPT}', commit it."
+#+end_src
+
+#+RESULTS:
+#+begin_example
+if [ -e /home/lschnorr//exp_12-v1_grimoire_6 ]; then
+ echo "Directory/File '/home/lschnorr//exp_12-v1_grimoire_6' already exists. Remove it or rename it."
+ exit
+fi
+mkdir -p /home/lschnorr//exp_12-v1_grimoire_6
+#
+# Experiment 12-v1_grimoire_6_96_metis_true
+#
+./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 12-v1_grimoire_6_96_metis_true true
+cp -prfv 12-v1_grimoire_6_96_metis_true.dir /home/lschnorr//exp_12-v1_grimoire_6
+rm -rf 12-v1_grimoire_6_96_metis_true.dir
+#
+# Experiment 12-v1_grimoire_6_96_sfc_true
+#
+./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 12-v1_grimoire_6_96_sfc_true true
+cp -prfv 12-v1_grimoire_6_96_sfc_true.dir /home/lschnorr//exp_12-v1_grimoire_6
+rm -rf 12-v1_grimoire_6_96_sfc_true.dir
+#
+# Experiment 12-v1_grimoire_6_48_metis_true
+#
+./scripts/run_alya_experiment.sh machine-file 48 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 12-v1_grimoire_6_48_metis_true true
+cp -prfv 12-v1_grimoire_6_48_metis_true.dir /home/lschnorr//exp_12-v1_grimoire_6
+rm -rf 12-v1_grimoire_6_48_metis_true.dir
+#
+# Experiment 12-v1_grimoire_6_48_sfc_true
+#
+./scripts/run_alya_experiment.sh machine-file 48 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 12-v1_grimoire_6_48_sfc_true true
+cp -prfv 12-v1_grimoire_6_48_sfc_true.dir /home/lschnorr//exp_12-v1_grimoire_6
+rm -rf 12-v1_grimoire_6_48_sfc_true.dir
+#
+# Final
+#
+cp script_12-v1_grimoire_6_nodes.sh /home/lschnorr//exp_12-v1_grimoire_6
+This script is at 'script_12-v1_grimoire_6_nodes.sh', commit it.
+#+end_example
+
+** 2017-04-10 12-node grisou (computing instrumentations, Alya modifs yes/no) :EXP13:
+#+begin_src shell :results output
+EID=13
+VERSION=1
+CLUSTER=grisou
+NODES=12
+MACHINEFILE=machine-file
+G5KHOME=/home/lschnorr/
+EID=${EID}-v${VERSION}
+SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
+
+rm -f $SCRIPT
+
+EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
+echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
+echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
+echo " exit" >> $SCRIPT
+echo "fi" >> $SCRIPT
+
+# Create experimental directory in the home
+echo "mkdir -p $EDIR" >> $SCRIPT
+
+# The design
+ALYAO=Alya.x
+ALYAM=Alya.x.modif
+
+for NP in 192 96; do
+ for INFINIBAND in true; do
+ for CASE in metis sfc; do
+ for ALYA in $ALYAO $ALYAM; do
+ NPN=$((NP))
+ UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}_${INFINIBAND}_${ALYA}
+ echo "# "
+ echo "# Experiment $UNIQUE"
+ echo "# "
+ echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/$ALYA /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
+ echo "cp -prfv $UNIQUE.dir ${EDIR}"
+ echo "rm -rf $UNIQUE.dir"
+ done
+ done
+ done >> ${SCRIPT}
+done
+echo "# " >> ${SCRIPT}
+echo "# Final" >> ${SCRIPT}
+echo "# " >> ${SCRIPT}
+echo "cp $SCRIPT $EDIR" >> $SCRIPT
+chmod 755 ${SCRIPT}
+cat ${SCRIPT}
+echo "This script is at '${SCRIPT}', commit it."
+#+end_src
+
+#+RESULTS:
+#+begin_example
+if [ -e /home/lschnorr//exp_13-v1_grisou_12 ]; then
+ echo "Directory/File '/home/lschnorr//exp_13-v1_grisou_12' already exists. Remove it or rename it."
+ exit
+fi
+mkdir -p /home/lschnorr//exp_13-v1_grisou_12
+#
+# Experiment 13-v1_grisou_12_192_metis_true_Alya.x
+#
+./scripts/run_alya_experiment.sh machine-file 192 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 13-v1_grisou_12_192_metis_true_Alya.x true
+cp -prfv 13-v1_grisou_12_192_metis_true_Alya.x.dir /home/lschnorr//exp_13-v1_grisou_12
+rm -rf 13-v1_grisou_12_192_metis_true_Alya.x.dir
+#
+# Experiment 13-v1_grisou_12_192_metis_true_Alya.x.modif
+#
+./scripts/run_alya_experiment.sh machine-file 192 /home/lschnorr/alya-bsc/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 13-v1_grisou_12_192_metis_true_Alya.x.modif true
+cp -prfv 13-v1_grisou_12_192_metis_true_Alya.x.modif.dir /home/lschnorr//exp_13-v1_grisou_12
+rm -rf 13-v1_grisou_12_192_metis_true_Alya.x.modif.dir
+#
+# Experiment 13-v1_grisou_12_192_sfc_true_Alya.x
+#
+./scripts/run_alya_experiment.sh machine-file 192 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 13-v1_grisou_12_192_sfc_true_Alya.x true
+cp -prfv 13-v1_grisou_12_192_sfc_true_Alya.x.dir /home/lschnorr//exp_13-v1_grisou_12
+rm -rf 13-v1_grisou_12_192_sfc_true_Alya.x.dir
+#
+# Experiment 13-v1_grisou_12_192_sfc_true_Alya.x.modif
+#
+./scripts/run_alya_experiment.sh machine-file 192 /home/lschnorr/alya-bsc/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 13-v1_grisou_12_192_sfc_true_Alya.x.modif true
+cp -prfv 13-v1_grisou_12_192_sfc_true_Alya.x.modif.dir /home/lschnorr//exp_13-v1_grisou_12
+rm -rf 13-v1_grisou_12_192_sfc_true_Alya.x.modif.dir
+#
+# Experiment 13-v1_grisou_12_96_metis_true_Alya.x
+#
+./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 13-v1_grisou_12_96_metis_true_Alya.x true
+cp -prfv 13-v1_grisou_12_96_metis_true_Alya.x.dir /home/lschnorr//exp_13-v1_grisou_12
+rm -rf 13-v1_grisou_12_96_metis_true_Alya.x.dir
+#
+# Experiment 13-v1_grisou_12_96_metis_true_Alya.x.modif
+#
+./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 13-v1_grisou_12_96_metis_true_Alya.x.modif true
+cp -prfv 13-v1_grisou_12_96_metis_true_Alya.x.modif.dir /home/lschnorr//exp_13-v1_grisou_12
+rm -rf 13-v1_grisou_12_96_metis_true_Alya.x.modif.dir
+#
+# Experiment 13-v1_grisou_12_96_sfc_true_Alya.x
+#
+./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 13-v1_grisou_12_96_sfc_true_Alya.x true
+cp -prfv 13-v1_grisou_12_96_sfc_true_Alya.x.dir /home/lschnorr//exp_13-v1_grisou_12
+rm -rf 13-v1_grisou_12_96_sfc_true_Alya.x.dir
+#
+# Experiment 13-v1_grisou_12_96_sfc_true_Alya.x.modif
+#
+./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 13-v1_grisou_12_96_sfc_true_Alya.x.modif true
+cp -prfv 13-v1_grisou_12_96_sfc_true_Alya.x.modif.dir /home/lschnorr//exp_13-v1_grisou_12
+rm -rf 13-v1_grisou_12_96_sfc_true_Alya.x.modif.dir
+#
+# Final
+#
+cp script_13-v1_grisou_12_nodes.sh /home/lschnorr//exp_13-v1_grisou_12
+This script is at 'script_13-v1_grisou_12_nodes.sh', commit it.
+#+end_example
+
+** 2017-05-10 20-node grisou :EXP14:
+*** Introduction :ATTACH:
+:PROPERTIES:
+:Attachments: all.scorep
+:ID: b51a80da-07e4-4d58-844c-a73009092eaa
+:END:
+
+The attached file, =all.scorep=, contains the functions that are
+instrumented during compilation by scorep. Those functions are non-MPI
+codes that represent the computation load.
+
+I am using Trunk.6515.Alya, unmodified, to get =Alya.x.orig=. Then, I
+manually modify Alya the way described here: [[*2017-04-06 Alya modifications to get more info, set balance to 1][2017-04-06 Alya
+modifications to get more info, set balance to 1]] to get another binary
+called =Alya.x.modif= (with elements weight set to 1). It seems that the
+latest version of Alya (revision 6515 at least) has a slightly
+different code to deal with element weight (with the =iweig=
+variable). So, for the =modif= version, I am forcing the =iweig= to be
+=1_rp=, removing the =if= that is now there.
+
+#+BEGIN_EXAMPLE
+lschnorr@grisou-1:~/Trunk.6515.Alya/Executables/unix$ svn diff ../../Sources/kernel/parall/mod_par_partit_sfc.f90
+Index: ../../Sources/kernel/parall/mod_par_partit_sfc.f90
+===================================================================
+--- ../../Sources/kernel/parall/mod_par_partit_sfc.f90 (revision 6515)
++++ ../../Sources/kernel/parall/mod_par_partit_sfc.f90 (working copy)
+@@ -459,6 +459,7 @@
+ else
+ iweig = 1_rp
+ endif
++ iweig = 1_rp
+
+ if( PAR_MY_SFC_RANK_WM /= iboxc-1 )then !if true element must be sent
+ if(bufwei(iboxl*2) == 0_rp) then
+#+END_EXAMPLE
+
+Note that the file =Sources/services/parall/par_outinf.f90= needs to be
+modified to output per-rank weights, points, etc. This is fundamental
+to conduct our analysis. Here's the diff:
+
+#+BEGIN_EXAMPLE
+lschnorr@grisou-1:~/Trunk.6515.Alya/Executables/unix$ svn diff ../../Sources/services/parall/par_outinf.f90
+Index: ../../Sources/services/parall/par_outinf.f90
+===================================================================
+--- ../../Sources/services/parall/par_outinf.f90 (revision 6515)
++++ ../../Sources/services/parall/par_outinf.f90 (working copy)
+@@ -15,6 +15,7 @@
+ use mod_parall
+ implicit none
+ integer(ip) :: isubd,ksmin(2),ksmax(2),kaver,koutp,nb
++ integer, parameter :: out_unit_npoin=20
+
+ if( IMASTER ) then
+
+@@ -27,7 +28,11 @@
+ ksmin = huge(1_ip)
+ ksmax = -huge(1_ip)
+ kaver = 0
++ ! LUCAS: manual instrumentation starts here, all code marked by LUCAS
++ open (unit=out_unit_npoin,file="results_NPOIN_NELEM_NELEW_NBOUN.log",action="write",status="replace")
+ do isubd=1,npart_par
++ ! LUCAS: Here I need to print: npoin_par(isubd)
++ write (out_unit_npoin,*) "Rank ", isubd ," NPOIN ", npoin_par(isubd), " NELEM ", nelem_par(isubd), " NELEW ", nelew_par(isubd), " NBOUN ", nboun_par(isubd), " NNEIG ", lneig_par(isubd), " NBBOU ", npoin_par(isubd)
+ if(npoin_par(isubd)> $SCRIPT
+echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
+echo " exit" >> $SCRIPT
+echo "fi" >> $SCRIPT
+
+# Create experimental directory in the home
+echo "mkdir -p $EDIR" >> $SCRIPT
+
+# The design
+ALYAO=Alya.x.orig
+ALYAM=Alya.x.modif
+ALYAPATH=/home/lschnorr/Trunk.6515.Alya
+
+for NP in 320; do
+ for INFINIBAND in false; do
+ for CASE in metis sfc; do
+ for ALYA in $ALYAO $ALYAM; do
+ NPN=$((NP))
+ UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}_${INFINIBAND}_${ALYA}
+ echo "# "
+ echo "# Experiment $UNIQUE"
+ echo "# "
+ echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} ${ALYAPATH}/Executables/unix/$ALYA /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
+ echo "cp -prfv $UNIQUE.dir ${EDIR}"
+ echo "rm -rf $UNIQUE.dir"
+ done
+ done
+ done >> ${SCRIPT}
+done
+echo "# " >> ${SCRIPT}
+echo "# Final" >> ${SCRIPT}
+echo "# " >> ${SCRIPT}
+echo "cp $SCRIPT $EDIR" >> $SCRIPT
+chmod 755 ${SCRIPT}
+cat ${SCRIPT}
+echo "This script is at '${SCRIPT}', commit it."
+#+end_src
+
+#+RESULTS:
+#+begin_example
+if [ -e /home/lschnorr//exp_14-v1_grisou_20 ]; then
+ echo "Directory/File '/home/lschnorr//exp_14-v1_grisou_20' already exists. Remove it or rename it."
+ exit
+fi
+mkdir -p /home/lschnorr//exp_14-v1_grisou_20
+#
+# Experiment 14-v1_grisou_20_320_metis_false_Alya.x.orig
+#
+./scripts/run_alya_experiment.sh machine-file 320 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.orig /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 14-v1_grisou_20_320_metis_false_Alya.x.orig false
+cp -prfv 14-v1_grisou_20_320_metis_false_Alya.x.orig.dir /home/lschnorr//exp_14-v1_grisou_20
+rm -rf 14-v1_grisou_20_320_metis_false_Alya.x.orig.dir
+#
+# Experiment 14-v1_grisou_20_320_metis_false_Alya.x.modif
+#
+./scripts/run_alya_experiment.sh machine-file 320 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 14-v1_grisou_20_320_metis_false_Alya.x.modif false
+cp -prfv 14-v1_grisou_20_320_metis_false_Alya.x.modif.dir /home/lschnorr//exp_14-v1_grisou_20
+rm -rf 14-v1_grisou_20_320_metis_false_Alya.x.modif.dir
+#
+# Experiment 14-v1_grisou_20_320_sfc_false_Alya.x.orig
+#
+./scripts/run_alya_experiment.sh machine-file 320 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.orig /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 14-v1_grisou_20_320_sfc_false_Alya.x.orig false
+cp -prfv 14-v1_grisou_20_320_sfc_false_Alya.x.orig.dir /home/lschnorr//exp_14-v1_grisou_20
+rm -rf 14-v1_grisou_20_320_sfc_false_Alya.x.orig.dir
+#
+# Experiment 14-v1_grisou_20_320_sfc_false_Alya.x.modif
+#
+./scripts/run_alya_experiment.sh machine-file 320 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 14-v1_grisou_20_320_sfc_false_Alya.x.modif false
+cp -prfv 14-v1_grisou_20_320_sfc_false_Alya.x.modif.dir /home/lschnorr//exp_14-v1_grisou_20
+rm -rf 14-v1_grisou_20_320_sfc_false_Alya.x.modif.dir
+#
+# Final
+#
+cp script_14-v1_grisou_20_nodes.sh /home/lschnorr//exp_14-v1_grisou_20
+This script is at 'script_14-v1_grisou_20_nodes.sh', commit it.
+#+end_example
+
+** 2017-05-11 44-node grisou (same as previous entry) :EXP15:
+*** Design
+
+#+begin_src shell :results output
+EID=15
+VERSION=1
+CLUSTER=grisou
+NODES=44
+MACHINEFILE=machine-file
+G5KHOME=/home/lschnorr/
+EID=${EID}-v${VERSION}
+SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
+
+rm -f $SCRIPT
+
+EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
+echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
+echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
+echo " exit" >> $SCRIPT
+echo "fi" >> $SCRIPT
+
+# Create experimental directory in the home
+echo "mkdir -p $EDIR" >> $SCRIPT
+
+# The design
+ALYAO=Alya.x.orig
+ALYAM=Alya.x.modif
+ALYAPATH=/home/lschnorr/Trunk.6515.Alya
+
+for NP in 704; do
+ for INFINIBAND in false; do
+ for CASE in metis sfc; do
+ for ALYA in $ALYAO $ALYAM; do
+ NPN=$((NP))
+ UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}_${INFINIBAND}_${ALYA}
+ echo "# "
+ echo "# Experiment $UNIQUE"
+ echo "# "
+ echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} ${ALYAPATH}/Executables/unix/$ALYA /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
+ echo "cp -prfv $UNIQUE.dir ${EDIR}"
+ echo "rm -rf $UNIQUE.dir"
+ done
+ done
+ done >> ${SCRIPT}
+done
+echo "# " >> ${SCRIPT}
+echo "# Final" >> ${SCRIPT}
+echo "# " >> ${SCRIPT}
+echo "cp $SCRIPT $EDIR" >> $SCRIPT
+chmod 755 ${SCRIPT}
+cat ${SCRIPT}
+echo "This script is at '${SCRIPT}', commit it."
+#+end_src
+
+#+RESULTS:
+#+begin_example
+if [ -e /home/lschnorr//exp_15-v1_grisou_44 ]; then
+ echo "Directory/File '/home/lschnorr//exp_15-v1_grisou_44' already exists. Remove it or rename it."
+ exit
+fi
+mkdir -p /home/lschnorr//exp_15-v1_grisou_44
+#
+# Experiment 15-v1_grisou_44_704_metis_false_Alya.x.orig
+#
+./scripts/run_alya_experiment.sh machine-file 704 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.orig /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 15-v1_grisou_44_704_metis_false_Alya.x.orig false
+cp -prfv 15-v1_grisou_44_704_metis_false_Alya.x.orig.dir /home/lschnorr//exp_15-v1_grisou_44
+rm -rf 15-v1_grisou_44_704_metis_false_Alya.x.orig.dir
+#
+# Experiment 15-v1_grisou_44_704_metis_false_Alya.x.modif
+#
+./scripts/run_alya_experiment.sh machine-file 704 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 15-v1_grisou_44_704_metis_false_Alya.x.modif false
+cp -prfv 15-v1_grisou_44_704_metis_false_Alya.x.modif.dir /home/lschnorr//exp_15-v1_grisou_44
+rm -rf 15-v1_grisou_44_704_metis_false_Alya.x.modif.dir
+#
+# Experiment 15-v1_grisou_44_704_sfc_false_Alya.x.orig
+#
+./scripts/run_alya_experiment.sh machine-file 704 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.orig /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 15-v1_grisou_44_704_sfc_false_Alya.x.orig false
+cp -prfv 15-v1_grisou_44_704_sfc_false_Alya.x.orig.dir /home/lschnorr//exp_15-v1_grisou_44
+rm -rf 15-v1_grisou_44_704_sfc_false_Alya.x.orig.dir
+#
+# Experiment 15-v1_grisou_44_704_sfc_false_Alya.x.modif
+#
+./scripts/run_alya_experiment.sh machine-file 704 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 15-v1_grisou_44_704_sfc_false_Alya.x.modif false
+cp -prfv 15-v1_grisou_44_704_sfc_false_Alya.x.modif.dir /home/lschnorr//exp_15-v1_grisou_44
+rm -rf 15-v1_grisou_44_704_sfc_false_Alya.x.modif.dir
+#
+# Final
+#
+cp script_15-v1_grisou_44_nodes.sh /home/lschnorr//exp_15-v1_grisou_44
+This script is at 'script_15-v1_grisou_44_nodes.sh', commit it.
+#+end_example
+
+** 2017-05-11 8-node grimoire (as previous, Infiniband and Ethernet) :EXP16:
+*** Design
+
+#+begin_src shell :results output
+EID=16
+VERSION=1
+CLUSTER=grimoire
+NODES=8
+MACHINEFILE=machine-file
+G5KHOME=/home/lschnorr/
+EID=${EID}-v${VERSION}
+SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
+
+rm -f $SCRIPT
+
+EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
+echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
+echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
+echo " exit" >> $SCRIPT
+echo "fi" >> $SCRIPT
+
+# Create experimental directory in the home
+echo "mkdir -p $EDIR" >> $SCRIPT
+
+# The design
+ALYAO=Alya.x.orig
+ALYAM=Alya.x.modif
+ALYAPATH=/home/lschnorr/Trunk.6515.Alya
+
+for NP in 128; do
+ for INFINIBAND in true false; do
+ for CASE in metis sfc; do
+ for ALYA in $ALYAO $ALYAM; do
+ NPN=$((NP))
+ UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}_${INFINIBAND}_${ALYA}
+ echo "# "
+ echo "# Experiment $UNIQUE"
+ echo "# "
+ echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} ${ALYAPATH}/Executables/unix/$ALYA /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
+ echo "cp -prfv $UNIQUE.dir ${EDIR}"
+ echo "rm -rf $UNIQUE.dir"
+ done
+ done
+ done >> ${SCRIPT}
+done
+echo "# " >> ${SCRIPT}
+echo "# Final" >> ${SCRIPT}
+echo "# " >> ${SCRIPT}
+echo "cp $SCRIPT $EDIR" >> $SCRIPT
+chmod 755 ${SCRIPT}
+cat ${SCRIPT}
+echo "This script is at '${SCRIPT}', commit it."
+#+end_src
+
+#+RESULTS:
+#+begin_example
+if [ -e /home/lschnorr//exp_16-v1_grimoire_8 ]; then
+ echo "Directory/File '/home/lschnorr//exp_16-v1_grimoire_8' already exists. Remove it or rename it."
+ exit
+fi
+mkdir -p /home/lschnorr//exp_16-v1_grimoire_8
+#
+# Experiment 16-v1_grimoire_8_128_metis_true_Alya.x.orig
+#
+./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.orig /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 16-v1_grimoire_8_128_metis_true_Alya.x.orig true
+cp -prfv 16-v1_grimoire_8_128_metis_true_Alya.x.orig.dir /home/lschnorr//exp_16-v1_grimoire_8
+rm -rf 16-v1_grimoire_8_128_metis_true_Alya.x.orig.dir
+#
+# Experiment 16-v1_grimoire_8_128_metis_true_Alya.x.modif
+#
+./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 16-v1_grimoire_8_128_metis_true_Alya.x.modif true
+cp -prfv 16-v1_grimoire_8_128_metis_true_Alya.x.modif.dir /home/lschnorr//exp_16-v1_grimoire_8
+rm -rf 16-v1_grimoire_8_128_metis_true_Alya.x.modif.dir
+#
+# Experiment 16-v1_grimoire_8_128_sfc_true_Alya.x.orig
+#
+./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.orig /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 16-v1_grimoire_8_128_sfc_true_Alya.x.orig true
+cp -prfv 16-v1_grimoire_8_128_sfc_true_Alya.x.orig.dir /home/lschnorr//exp_16-v1_grimoire_8
+rm -rf 16-v1_grimoire_8_128_sfc_true_Alya.x.orig.dir
+#
+# Experiment 16-v1_grimoire_8_128_sfc_true_Alya.x.modif
+#
+./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 16-v1_grimoire_8_128_sfc_true_Alya.x.modif true
+cp -prfv 16-v1_grimoire_8_128_sfc_true_Alya.x.modif.dir /home/lschnorr//exp_16-v1_grimoire_8
+rm -rf 16-v1_grimoire_8_128_sfc_true_Alya.x.modif.dir
+#
+# Experiment 16-v1_grimoire_8_128_metis_false_Alya.x.orig
+#
+./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.orig /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 16-v1_grimoire_8_128_metis_false_Alya.x.orig false
+cp -prfv 16-v1_grimoire_8_128_metis_false_Alya.x.orig.dir /home/lschnorr//exp_16-v1_grimoire_8
+rm -rf 16-v1_grimoire_8_128_metis_false_Alya.x.orig.dir
+#
+# Experiment 16-v1_grimoire_8_128_metis_false_Alya.x.modif
+#
+./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 16-v1_grimoire_8_128_metis_false_Alya.x.modif false
+cp -prfv 16-v1_grimoire_8_128_metis_false_Alya.x.modif.dir /home/lschnorr//exp_16-v1_grimoire_8
+rm -rf 16-v1_grimoire_8_128_metis_false_Alya.x.modif.dir
+#
+# Experiment 16-v1_grimoire_8_128_sfc_false_Alya.x.orig
+#
+./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.orig /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 16-v1_grimoire_8_128_sfc_false_Alya.x.orig false
+cp -prfv 16-v1_grimoire_8_128_sfc_false_Alya.x.orig.dir /home/lschnorr//exp_16-v1_grimoire_8
+rm -rf 16-v1_grimoire_8_128_sfc_false_Alya.x.orig.dir
+#
+# Experiment 16-v1_grimoire_8_128_sfc_false_Alya.x.modif
+#
+./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 16-v1_grimoire_8_128_sfc_false_Alya.x.modif false
+cp -prfv 16-v1_grimoire_8_128_sfc_false_Alya.x.modif.dir /home/lschnorr//exp_16-v1_grimoire_8
+rm -rf 16-v1_grimoire_8_128_sfc_false_Alya.x.modif.dir
+#
+# Final
+#
+cp script_16-v1_grimoire_8_nodes.sh /home/lschnorr//exp_16-v1_grimoire_8
+This script is at 'script_16-v1_grimoire_8_nodes.sh', commit it.
+#+end_example
+
+* Analysis
+** 2-node grisou (17 and 32 ranks) :EXP1:
+*** Rough epoch-based execution time
+#+name: exp_1_grisou_2_epoch
+#+begin_src shell :results output org table
+EDIR=exp_1_grisou_2
+for case in $(find $EDIR | grep -e "sfc.org" -e "metis.org"); do
+ echo -n "$(basename $case) "
+ cat $case | grep epoch | sed "s/^.*: //" | tr '\n' '-' | sed -e "s/-$//" -e "s/-/ - /" -e "s/$/\n/" | bc -l;
+done | sed -e "s/.org//" | sed -e "s/_/|/g" -e "s/-//g"
+#+end_src
+
+#+RESULTS: exp_1_grisou_2_epoch
+#+BEGIN_SRC org
+| grisou | 2 | 32 | metis | 951 |
+| grisou | 2 | 32 | sfc | 762 |
+| grisou | 2 | 17 | metis | 1417 |
+| grisou | 2 | 17 | sfc | 1225 |
+#+END_SRC
+
+#+header: :var df=exp_1_grisou_2_epoch
+#+begin_src R :results output graphics :file img/exp_1_grisou_2_epoch.png :exports both :width 600 :height 400 :session
+library(dplyr);
+library(ggplot2);
+df %>%
+ rename(Cluster= V1,
+ Nodes = V2,
+ NP = V3,
+ Case = V4,
+ Time = V5) %>%
+ filter(NP != 121) %>%
+ ggplot(aes(x=NP, y=Time, color=Case)) +
+ theme_bw() +
+ ylim(0,NA) +
+ ylab("Time [s]") +
+ geom_point() +
+ geom_line() +
+ theme(legend.position = "top");
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_1_grisou_2_epoch.png]]
+** 10-node grisou (97, 113, 129, 145, 160) :EXP2:
+*** Rough epoch-based execution time
+#+name: exp_2_grisou_10_epoch
+#+begin_src shell :results output org table
+EDIR=exp_2_grisou_10
+for case in $(find $EDIR | grep -e "sfc.org" -e "metis.org"); do
+ echo -n "$(basename $case) "
+ cat $case | grep epoch | sed "s/^.*: //" | tr '\n' '-' | sed -e "s/-$//" -e "s/-/ - /" -e "s/$/\n/" | bc -l;
+done | sed -e "s/.org//" | sed -e "s/_/|/g" -e "s/-//g"
+#+end_src
+
+#+RESULTS: exp_2_grisou_10_epoch
+#+BEGIN_SRC org
+| grisou | 10 | 160 | metis | 304 |
+| grisou | 10 | 129 | sfc | 337 |
+| grisou | 10 | 97 | metis | 401 |
+| grisou | 10 | 145 | sfc | 315 |
+| grisou | 10 | 129 | metis | 345 |
+| grisou | 10 | 145 | metis | 321 |
+| grisou | 10 | 97 | sfc | 380 |
+| grisou | 10 | 160 | sfc | 301 |
+| grisou | 10 | 113 | metis | 363 |
+| grisou | 10 | 113 | sfc | 359 |
+#+END_SRC
+
+#+header: :var df=exp_2_grisou_10_epoch
+#+begin_src R :results output graphics :file img/exp_2_grisou_10_epoch.png :exports both :width 200 :height 400 :session
+library(dplyr);
+library(ggplot2);
+df %>%
+ rename(Cluster= V1,
+ Nodes = V2,
+ NP = V3,
+ Case = V4,
+ Time = V5) %>%
+ filter(NP != 121) %>%
+ ggplot(aes(x=NP, y=Time, color=Case)) +
+ theme_bw() +
+ ylim(0,NA) +
+ ylab("Time [s]") +
+ geom_point() +
+ geom_line() +
+ theme(legend.position = "top");
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_2_grisou_10_epoch.png]]
+*** Convert to CSV
+I need on the $PATH:
+- akypuera's otf22paje (compiled using OTF2 libraries of ScoreP 3.0)
+- pajeng's =pj_dump=
+
+#+begin_src shell :results output
+export PATH=$PATH:/home/schnorr/install/stow/bin/
+export PATH=$PATH:/home/schnorr/dev/pajeng/b/
+
+convert() {
+ pushd $(dirname $otf2)
+ otf22paje traces.otf2 | pj_dump | grep ^State | cut -d, -f2,4,5,8 | sed -e "s/ //g" -e "s/MPIRank//" > traces.csv
+ popd
+}
+
+EDIR=exp_2_grisou_10
+N=4 # number of parallel batches
+for otf2 in $(find $EDIR | grep otf2$); do
+ ((i=i%N)); ((i++==0)) && wait
+ convert $otf2 &
+done
+#+end_src
+
+#+RESULTS:
+#+begin_example
+/home/schnorr/install/stow/bin//otf22paje
+Usage: otf22paje [OPTION...] ANCHORFILE
+Converts an OTF2 archive to the Paje file format
+
+ -b, --basic Avoid extended events (impoverished trace file)
+ -d, --dummy Read input traces but won't translate (no output)
+ -f, --flat Flat hierarchy (only MPI ranks)
+ -h, --hostfile=FILE MPI hostfile to create system hierarchy
+ -i, --ignore-errors Ignore errors
+ -l, --no-links Don't convert links
+ -m, --comment=COMMENT Comment is echoed to output
+ -n, --commentfile=FILE Comments (from file) echoed to output
+ -o, --only-mpi Only convert MPI states
+ -s, --no-states Don't convert states
+ -z, --normalize-mpi Try to normalize MPI state names
+ -?, --help Give this help list
+ --usage Give a short usage message
+
+Mandatory or optional arguments to long options are also mandatory or optional
+for any corresponding short options.
+~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_160_metis.dir/scorep_grisou_10_160_metis ~/dev/Alya-Perf
+~/dev/Alya-Perf
+~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_129_sfc.dir/scorep_grisou_10_129_sfc ~/dev/Alya-Perf
+~/dev/Alya-Perf
+~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_97_metis.dir/scorep_grisou_10_97_metis ~/dev/Alya-Perf
+~/dev/Alya-Perf
+~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_145_sfc.dir/scorep_grisou_10_145_sfc ~/dev/Alya-Perf
+~/dev/Alya-Perf
+~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_129_metis.dir/scorep_grisou_10_129_metis ~/dev/Alya-Perf
+~/dev/Alya-Perf
+~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_145_metis.dir/scorep_grisou_10_145_metis ~/dev/Alya-Perf
+~/dev/Alya-Perf
+~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_97_sfc.dir/scorep_grisou_10_97_sfc ~/dev/Alya-Perf
+~/dev/Alya-Perf
+~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_160_sfc.dir/scorep_grisou_10_160_sfc ~/dev/Alya-Perf
+~/dev/Alya-Perf
+~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_113_metis.dir/scorep_grisou_10_113_metis ~/dev/Alya-Perf
+~/dev/Alya-Perf
+~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_113_sfc.dir/scorep_grisou_10_113_sfc ~/dev/Alya-Perf
+~/dev/Alya-Perf
+#+end_example
+
+#+begin_src shell :results output
+export PATH=$PATH:/home/schnorr/install/stow/bin/
+export PATH=$PATH:/home/schnorr/dev/pajeng/b/
+
+convert() {
+ pushd $(dirname $otf2)
+ otf22paje traces.otf2 | pj_dump | grep ^State | cut -d, -f2,4,5,8 | sed -e "s/ //g" -e "s/MPIRank//" > traces.csv
+ popd
+}
+
+EDIR=exp_2_grisou_10
+# Files already converted (whose CSV size is not zero)
+EXISTINGFILE=$(tempfile)
+OTF2FILE=$(tempfile)
+find $EDIR -not -empty | grep csv$ | sed -e "s/.csv$//" | sort > $EXISTINGFILE
+find $EDIR | grep otf2 | sed -e "s/.otf2$//" | sort > $OTF2FILE
+
+for otf2 in $(comm -3 $OTF2FILE $EXISTINGFILE | sed "s/$/.otf2/"); do
+ convert $otf2
+done
+#+end_src
+
+#+RESULTS:
+*** Read and analysis
+Check read and analysis of next section.
+** 10-node grisou (97, 113, 129, 145, 160) :EXP3:
+*** Rough epoch-based execution time
+#+name: exp_3_grisou_10_epoch
+#+begin_src shell :results output org table
+EDIR=exp_3_grisou_10
+for case in $(find $EDIR | grep -e "sfc.org" -e "metis.org"); do
+ echo -n "$(basename $case) "
+ cat $case | grep epoch | sed "s/^.*: //" | tr '\n' '-' | sed -e "s/-$//" -e "s/-/ - /" -e "s/$/\n/" | bc -l;
+done | sed -e "s/.org//" | sed -e "s/_/|/g" -e "s/-//g"
+#+end_src
+
+#+RESULTS: exp_3_grisou_10_epoch
+#+BEGIN_SRC org
+| 3 | grisou | 10 | 145 | sfc | 320 |
+| 3 | grisou | 10 | 113 | sfc | 356 |
+| 3 | grisou | 10 | 97 | metis | 394 |
+| 3 | grisou | 10 | 129 | sfc | 336 |
+| 3 | grisou | 10 | 129 | metis | 344 |
+| 3 | grisou | 10 | 160 | sfc | 308 |
+| 3 | grisou | 10 | 145 | metis | 319 |
+| 3 | grisou | 10 | 113 | metis | 362 |
+| 3 | grisou | 10 | 160 | metis | 306 |
+| 3 | grisou | 10 | 97 | sfc | 380 |
+#+END_SRC
+
+#+header: :var df=exp_3_grisou_10_epoch
+#+begin_src R :results output graphics :file img/exp_3_grisou_10_epoch.png :exports both :width 200 :height 400 :session
+library(dplyr);
+library(ggplot2);
+df %>%
+ rename(EID = V1,
+ Cluster= V2,
+ Nodes = V3,
+ NP = V4,
+ Case = V5,
+ Time = V6) %>%
+ filter(NP != 121) %>%
+ ggplot(aes(x=NP, y=Time, color=Case)) +
+ theme_bw() +
+ ylim(0,NA) +
+ ylab("Time [s]") +
+ geom_point() +
+ geom_line() +
+ theme(legend.position = "top");
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_3_grisou_10_epoch.png]]
+*** Convert to CSV
+I need on the $PATH:
+- akypuera's otf22paje (compiled using OTF2 libraries of ScoreP 3.0)
+- pajeng's =pj_dump=
+
+#+begin_src shell :results output
+export PATH=$PATH:/home/schnorr/install/stow/bin/
+export PATH=$PATH:/home/schnorr/dev/pajeng/b/
+
+convert() {
+ pushd $(dirname $otf2)
+ otf22paje traces.otf2 | pj_dump | grep ^State | cut -d, -f2,4,5,8 | sed -e "s/ //g" -e "s/MPIRank//" > traces.csv
+ popd
+}
+
+EDIR=exp_3_grisou_10
+# Files already converted (whose CSV size is not zero)
+EXISTINGFILE=$(tempfile)
+OTF2FILE=$(tempfile)
+find $EDIR -not -empty | grep csv$ | sed -e "s/.csv$//" | sort > $EXISTINGFILE
+find $EDIR | grep otf2 | sed -e "s/.otf2$//" | sort > $OTF2FILE
+
+for otf2 in $(comm -3 $OTF2FILE $EXISTINGFILE | sed "s/$/.otf2/"); do
+ echo $otf2
+ convert $otf2
+done
+#+end_src
+
+#+RESULTS:
+#+begin_example
+exp_3_grisou_10/3_grisou_10_113_metis.dir/scorep_3_grisou_10_113_metis/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_113_metis.dir/scorep_3_grisou_10_113_metis ~/dev/Alya-Perf
+~/dev/Alya-Perf
+exp_3_grisou_10/3_grisou_10_113_sfc.dir/scorep_3_grisou_10_113_sfc/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_113_sfc.dir/scorep_3_grisou_10_113_sfc ~/dev/Alya-Perf
+~/dev/Alya-Perf
+exp_3_grisou_10/3_grisou_10_129_metis.dir/scorep_3_grisou_10_129_metis/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_129_metis.dir/scorep_3_grisou_10_129_metis ~/dev/Alya-Perf
+~/dev/Alya-Perf
+exp_3_grisou_10/3_grisou_10_129_sfc.dir/scorep_3_grisou_10_129_sfc/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_129_sfc.dir/scorep_3_grisou_10_129_sfc ~/dev/Alya-Perf
+~/dev/Alya-Perf
+exp_3_grisou_10/3_grisou_10_145_metis.dir/scorep_3_grisou_10_145_metis/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_145_metis.dir/scorep_3_grisou_10_145_metis ~/dev/Alya-Perf
+~/dev/Alya-Perf
+exp_3_grisou_10/3_grisou_10_145_sfc.dir/scorep_3_grisou_10_145_sfc/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_145_sfc.dir/scorep_3_grisou_10_145_sfc ~/dev/Alya-Perf
+~/dev/Alya-Perf
+exp_3_grisou_10/3_grisou_10_160_metis.dir/scorep_3_grisou_10_160_metis/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_160_metis.dir/scorep_3_grisou_10_160_metis ~/dev/Alya-Perf
+~/dev/Alya-Perf
+exp_3_grisou_10/3_grisou_10_160_sfc.dir/scorep_3_grisou_10_160_sfc/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_160_sfc.dir/scorep_3_grisou_10_160_sfc ~/dev/Alya-Perf
+~/dev/Alya-Perf
+exp_3_grisou_10/3_grisou_10_97_metis.dir/scorep_3_grisou_10_97_metis/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_97_metis.dir/scorep_3_grisou_10_97_metis ~/dev/Alya-Perf
+~/dev/Alya-Perf
+exp_3_grisou_10/3_grisou_10_97_sfc.dir/scorep_3_grisou_10_97_sfc/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_97_sfc.dir/scorep_3_grisou_10_97_sfc ~/dev/Alya-Perf
+~/dev/Alya-Perf
+#+end_example
+*** Read in R
+#+begin_src R :results output :session :exports both :tangle do.R :tangle-mode (identity #o755)
+#!/usr/bin/Rscript
+library(readr);
+library(dplyr);
+alya_scorep_trace_read <- function(filename)
+{
+ meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[3], "_"));
+ read_csv(filename,
+ col_names=c("Rank", "Start", "End", "Value"),
+ progress=TRUE) %>%
+ # Transform Value to factor
+ mutate(Value = as.factor(Value)) %>%
+ # Detect begin and end of iterations
+ mutate(Iteration = case_when(
+ grepl("timste", .$Value) ~ 1,
+ grepl("endste", .$Value) ~ -1,
+ TRUE ~ 0)) %>%
+ group_by(Rank) %>%
+ # Create a logical to detect observations within iterations
+ mutate(Iteration = as.logical(cumsum(Iteration))) %>%
+ # Get only observations that belongs to some iteration
+ filter(Iteration == TRUE) %>%
+ ungroup() %>%
+ # Create the iteration by cumsum-
+ mutate(Iteration = case_when(
+ grepl("timste", .$Value) ~ 1,
+ TRUE ~ 0)) %>%
+ group_by(Rank) %>%
+ mutate(Iteration = cumsum(Iteration)) %>%
+ mutate(EID = meta[2],
+ Platform = meta[3],
+ Nodes = meta[4],
+ NP = meta[5],
+ Partitioning = meta[6]);
+}
+
+alya_scorep_trace_iterations <- function(filename)
+{
+ alya_scorep_trace_read(filename) %>%
+ group_by(Rank, Iteration, Platform, Nodes, NP, Partitioning, EID) %>%
+ summarize(N=n(), S=min(Start), E=max(End));
+}
+
+args = commandArgs(trailingOnly=TRUE)
+print(args);
+df <- do.call("rbind", lapply(args, function(x) { alya_scorep_trace_iterations(x) }));
+write.csv(df, "exp2_exp3_iterations.csv");
+#+end_src
+
+#+RESULTS:
+*** Analysis :ATTACH:
+:PROPERTIES:
+:Attachments: exp2_exp3_iterations.csv
+:ID: d2233039-ad84-42ac-ad70-545877ba7e8e
+:END:
+**** Iteration-only shows SFC is worse than Metis (average per-iteration span)
+#+begin_src R :results output :session :exports both
+library(readr);
+library(dplyr);
+df <- read_csv("data/d2/233039-ad84-42ac-ad70-545877ba7e8e/exp2_exp3_iterations.csv") %>%
+ select(-X1) %>%
+ rename(Start = S, End = E) %>%
+ mutate(Duration = End - Start) %>%
+ mutate(Platform = as.factor(Platform),
+ Partitioning = as.factor(Partitioning),
+ Nodes = as.factor(Nodes));
+df %>% summary;
+#+end_src
+
+#+RESULTS:
+#+begin_example
+
+Attaching package: ‘dplyr’
+
+The following objects are masked from ‘package:stats’:
+
+ filter, lag
+
+The following objects are masked from ‘package:base’:
+
+ intersect, setdiff, setequal, union
+Parsed with column specification:
+cols(
+ X1 = col_integer(),
+ Rank = col_integer(),
+ Iteration = col_integer(),
+ Platform = col_character(),
+ Nodes = col_integer(),
+ NP = col_integer(),
+ Partitioning = col_character(),
+ EID = col_integer(),
+ N = col_integer(),
+ S = col_double(),
+ E = col_double()
+)
+Warning message:
+Missing column names filled in: 'X1' [1]
+ Rank Iteration Platform Nodes NP
+ Min. : 0.00 Min. : 1.0 grisou:25760 10:25760 Min. : 97.0
+ 1st Qu.: 32.00 1st Qu.: 3.0 1st Qu.:113.0
+ Median : 64.00 Median : 5.5 Median :129.0
+ Mean : 65.84 Mean : 5.5 Mean :132.7
+ 3rd Qu.: 96.00 3rd Qu.: 8.0 3rd Qu.:145.0
+ Max. :159.00 Max. :10.0 Max. :160.0
+ Partitioning EID N Start End
+ metis:12880 Min. :2.0 Min. : 3580 Min. :120.2 Min. :139.9
+ sfc :12880 1st Qu.:2.0 1st Qu.:17618 1st Qu.:168.8 1st Qu.:188.9
+ Median :2.5 Median :24816 Median :220.5 Median :241.4
+ Mean :2.5 Mean :28263 Mean :222.6 Mean :243.2
+ 3rd Qu.:3.0 3rd Qu.:37184 3rd Qu.:271.5 3rd Qu.:293.7
+ Max. :3.0 Max. :85082 Max. :375.5 Max. :397.2
+ Duration
+ Min. :13.57
+ 1st Qu.:18.05
+ Median :19.93
+ Mean :20.63
+ 3rd Qu.:23.17
+ Max. :29.10
+#+end_example
+
+Execution time
+
+#+begin_src R :results output graphics :file img/exp2_exp3_iterations_only.png :exports both :width 600 :height 400 :session
+df %>%
+ group_by(Platform, Nodes, NP, Partitioning, EID) %>%
+ summarize(Time = max(End) - min(Start)) %>%
+ group_by(Platform, Nodes, NP, Partitioning) %>%
+ summarize(Time = mean(Time), N=n()) %>%
+ print %>%
+ ggplot(aes(x=NP, y=Time, color=Partitioning)) +
+ theme_bw(base_size=12) +
+ geom_point() +
+ geom_line() +
+ ylim(0, NA)
+#+end_src
+
+#+RESULTS:
+[[file:img/exp2_exp3_iterations_only.png]]
+
+#+begin_src R :results output :session :exports both
+df %>%
+ group_by(NP, Iteration, Partitioning) %>%
+ # calculate the mean of number of operations per iteration
+ summarize(N=mean(N), Z=n());
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Source: local data frame [100 x 5]
+Groups: NP, Iteration [?]
+
+ NP Iteration Partitioning N Z
+
+1 97 1 metis 19204.61 194
+2 97 1 sfc 40975.99 194
+3 97 2 metis 18361.41 194
+4 97 2 sfc 39909.78 194
+5 97 3 metis 18358.87 194
+6 97 3 sfc 39712.10 194
+7 97 4 metis 17679.86 194
+8 97 4 sfc 38238.62 194
+9 97 5 metis 17201.62 194
+10 97 5 sfc 37063.47 194
+# ... with 90 more rows
+#+end_example
+
+#+begin_src R :results output graphics :file img/exp2_exp3_number_of_operations.png :exports both :width 600 :height 400 :session
+library(ggplot2);
+df %>%
+ group_by(NP, Iteration, Partitioning) %>%
+ # calculate the mean of number of operations per iteration
+ summarize(N=mean(N), Z=n()) %>%
+ ungroup() %>%
+ ggplot(aes(x=Iteration, y=N, color=Partitioning)) +
+ theme_bw(base_size=12) +
+ ylim(0,NA) +
+ geom_point() +
+ theme(legend.position="top") +
+ facet_wrap(~NP, nrow=1)
+#+end_src
+
+#+RESULTS:
+[[file:img/exp2_exp3_number_of_operations.png]]
+**** Addressing Rick's questions (on max per-iteration)
+:RICK:
+Could you give us the results shown in =exp2_exp3_iterations_only.png=, but
+considering the maximum time for each CPU-core instead of the average?
+:END:
+
+:LUCAS:
+If I understood correctly your question, you want to see the time each
+CPU-core took to compute the 10 iterations. So, I put Rank in the X
+axis, and on Y, I put the difference of time between the moment where
+a given rank enters the first iteration until the moment it leaves the
+10th iteration. I select only one experiment (exp2) because they are
+very similar (on the same set of machines). The exp2 experiment
+contained 5 executions with different NP (97, 113, 129, 145, and
+160). The five plots are in the attached file
+=exp2_exp3_10iter_time_of_each_rank.png=
+:END:
+
+#+begin_src R :results output graphics :file img/exp2_10iter_time_of_each_rank.png :exports both :width 600 :height 600 :session
+library(ggplot2);
+df %>%
+ filter(EID==2) %>%
+ group_by(Rank, NP, Partitioning, EID) %>%
+ summarize(Time = max(End) - min(Start)) %>%
+ ggplot(aes(x=Rank, y=Time, color=Partitioning)) +
+ theme_bw(base_size=12) +
+ geom_point() +
+ ylim(0, NA) +
+ facet_grid(EID~NP) +
+ theme(legend.position="top")
+#+end_src
+
+#+RESULTS:
+[[file:img/exp2_10iter_time_of_each_rank.png]]
+
+:LUCAS:
+Since I am not sure I understood correctly your request, the plot
+=img/exp2_exp3_iterations_only_max.png= contains the max execution time
+(on Y) of each of the 10 iterations (the horizontal facets) taking
+into account all ranks belonging to each of the five NP experiments
+(on X).
+:END:
+
+#+begin_src R :results output graphics :file img/exp2_exp3_iterations_only_max.png :exports both :width 1200 :height 400 :session
+library(ggplot2);
+df %>%
+ select(-Start, -End) %>%
+ group_by(Iteration, NP, Partitioning) %>%
+ summarize(Time = max(Duration), N=n()) %>%
+ ungroup() %>%
+ mutate(NP=as.factor(NP)) %>%
+ ggplot(aes(x=NP, y=Time, color=Partitioning, group=Partitioning)) +
+ theme_bw(base_size=12) +
+ geom_point() +
+ geom_line() +
+ ylim(0, NA) +
+ facet_grid(~Iteration) +
+ theme(legend.position="top")
+#+end_src
+
+#+RESULTS:
+[[file:img/exp2_exp3_iterations_only_max.png]]
+** 23-node nova (367 352 336 272 208 144 80 16) :EXP4:
+*** Preparation (2-nodes only)
+**** Convert to CSV
+I need on the $PATH:
+- akypuera's otf22paje (compiled using OTF2 libraries of ScoreP 3.0)
+- pajeng's =pj_dump=
+
+#+begin_src shell :results output
+export PATH=$PATH:/home/schnorr/install/stow/bin/
+export PATH=$PATH:/home/schnorr/dev/pajeng/b/
+
+convert() {
+ pushd $(dirname $otf2)
+ otf22paje traces.otf2 | pj_dump | grep ^State | cut -d, -f2,4,5,8 | sed -e "s/ //g" -e "s/MPIRank//" | gzip > traces.csv.gz
+ popd
+}
+
+EDIR=exp_prep-4-v2_grisou_4
+# Files already converted (whose CSV size is not zero)
+EXISTINGFILE=$(tempfile)
+OTF2FILE=$(tempfile)
+find $EDIR -not -empty | grep csv$ | sed -e "s/.csv$//" | sort > $EXISTINGFILE
+find $EDIR | grep otf2 | sed -e "s/.otf2$//" | sort > $OTF2FILE
+
+for otf2 in $(comm -3 $OTF2FILE $EXISTINGFILE | sed "s/$/.otf2/"); do
+ echo $otf2
+ convert $otf2
+done
+#+end_src
+
+#+RESULTS:
+: exp_prep-4-v2_grisou_4/prep-4_grisou_4_63_metis.dir/scorep_prep-4_grisou_4_63_metis/traces.otf2
+: ~/dev/Alya-Perf/exp_prep-4-v2_grisou_4/prep-4_grisou_4_63_metis.dir/scorep_prep-4_grisou_4_63_metis ~/dev/Alya-Perf
+: ~/dev/Alya-Perf
+: exp_prep-4-v2_grisou_4/prep-4_grisou_4_63_sfc.dir/scorep_prep-4_grisou_4_63_sfc/traces.otf2
+: ~/dev/Alya-Perf/exp_prep-4-v2_grisou_4/prep-4_grisou_4_63_sfc.dir/scorep_prep-4_grisou_4_63_sfc ~/dev/Alya-Perf
+: ~/dev/Alya-Perf
+**** Extract iteration information :ATTACH:
+:PROPERTIES:
+:Attachments: exp_prep-4_iterations.csv.gz
+:ID: b0e3978c-4131-4005-b979-a86aad031816
+:END:
+#+begin_src shell :results output
+./do.R \
+exp_prep-4-v1_grisou_4/prep-4_grisou_4_18_sfc.dir/scorep_prep-4_grisou_4_18_sfc/traces.csv.gz \
+exp_prep-4-v1_grisou_4/prep-4_grisou_4_33_sfc.dir/scorep_prep-4_grisou_4_33_sfc/traces.csv.gz \
+exp_prep-4-v1_grisou_4/prep-4_grisou_4_49_metis.dir/scorep_prep-4_grisou_4_49_metis/traces.csv.gz \
+exp_prep-4-v1_grisou_4/prep-4_grisou_4_63_sfc.dir/scorep_prep-4_grisou_4_63_sfc/traces.csv.gz \
+exp_prep-4-v1_grisou_4/prep-4_grisou_4_33_metis.dir/scorep_prep-4_grisou_4_33_metis/traces.csv.gz \
+exp_prep-4-v1_grisou_4/prep-4_grisou_4_18_metis.dir/scorep_prep-4_grisou_4_18_metis/traces.csv.gz \
+exp_prep-4-v1_grisou_4/prep-4_grisou_4_49_sfc.dir/scorep_prep-4_grisou_4_49_sfc/traces.csv.gz \
+exp_prep-4-v1_grisou_4/prep-4_grisou_4_63_metis.dir/scorep_prep-4_grisou_4_63_metis/traces.csv.gz \
+exp_prep-4-v2_grisou_4/prep-4_grisou_4_63_sfc.dir/scorep_prep-4_grisou_4_63_sfc/traces.csv.gz \
+exp_prep-4-v2_grisou_4/prep-4_grisou_4_63_metis.dir/scorep_prep-4_grisou_4_63_metis/traces.csv.gz \
+#+end_src
+
+Results are saved in file compressed CSV file called =exp_prep-4.csv.gz=.
+**** Analysis
+Forthcoming.
+*** 23-Node data
+**** Convert to CSV
+I need on the $PATH:
+- akypuera's otf22paje (compiled using OTF2 libraries of ScoreP 3.0)
+- pajeng's =pj_dump=
+
+#+begin_src shell :results output
+export PATH=$PATH:/home/lschnorr/akypuera/b/
+export PATH=$PATH:/home/lschnorr/pajeng/b/
+
+convert() {
+ pushd $(dirname $otf2)
+ otf22paje traces.otf2 | pj_dump | grep ^State | cut -d, -f2,4,5,8 | sed -e "s/ //g" -e "s/MPIRank//" | gzip > traces.csv.gz
+ popd
+}
+
+EDIR=exp_4_grisou_23_v1/
+# Files already converted (whose CSV size is not zero)
+EXISTINGFILE=$(tempfile)
+OTF2FILE=$(tempfile)
+find $EDIR -not -empty | grep csv.gz$ | sed -e "s/.csv.gz$//" | sort > $EXISTINGFILE
+find $EDIR | grep \.otf2$ | sed -e "s/.otf2$//" | sort > $OTF2FILE
+
+for otf2 in $(comm -3 $OTF2FILE $EXISTINGFILE | sed "s/$/.otf2/"); do
+ echo $otf2
+ convert $otf2
+done
+#+end_src
+
+#+RESULTS:
+#+begin_example
+exp_3_grisou_10/3_grisou_10_113_metis.dir/scorep_3_grisou_10_113_metis/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_113_metis.dir/scorep_3_grisou_10_113_metis ~/dev/Alya-Perf
+~/dev/Alya-Perf
+exp_3_grisou_10/3_grisou_10_113_sfc.dir/scorep_3_grisou_10_113_sfc/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_113_sfc.dir/scorep_3_grisou_10_113_sfc ~/dev/Alya-Perf
+~/dev/Alya-Perf
+exp_3_grisou_10/3_grisou_10_129_metis.dir/scorep_3_grisou_10_129_metis/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_129_metis.dir/scorep_3_grisou_10_129_metis ~/dev/Alya-Perf
+~/dev/Alya-Perf
+exp_3_grisou_10/3_grisou_10_129_sfc.dir/scorep_3_grisou_10_129_sfc/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_129_sfc.dir/scorep_3_grisou_10_129_sfc ~/dev/Alya-Perf
+~/dev/Alya-Perf
+exp_3_grisou_10/3_grisou_10_145_metis.dir/scorep_3_grisou_10_145_metis/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_145_metis.dir/scorep_3_grisou_10_145_metis ~/dev/Alya-Perf
+~/dev/Alya-Perf
+exp_3_grisou_10/3_grisou_10_145_sfc.dir/scorep_3_grisou_10_145_sfc/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_145_sfc.dir/scorep_3_grisou_10_145_sfc ~/dev/Alya-Perf
+~/dev/Alya-Perf
+exp_3_grisou_10/3_grisou_10_160_metis.dir/scorep_3_grisou_10_160_metis/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_160_metis.dir/scorep_3_grisou_10_160_metis ~/dev/Alya-Perf
+~/dev/Alya-Perf
+exp_3_grisou_10/3_grisou_10_160_sfc.dir/scorep_3_grisou_10_160_sfc/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_160_sfc.dir/scorep_3_grisou_10_160_sfc ~/dev/Alya-Perf
+~/dev/Alya-Perf
+exp_3_grisou_10/3_grisou_10_97_metis.dir/scorep_3_grisou_10_97_metis/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_97_metis.dir/scorep_3_grisou_10_97_metis ~/dev/Alya-Perf
+~/dev/Alya-Perf
+exp_3_grisou_10/3_grisou_10_97_sfc.dir/scorep_3_grisou_10_97_sfc/traces.otf2
+~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_97_sfc.dir/scorep_3_grisou_10_97_sfc ~/dev/Alya-Perf
+~/dev/Alya-Perf
+#+end_example
+**** Post-processing in R
+#+begin_src R :results output :session :exports both :tangle do-exp4.R :tangle-mode (identity #o755)
+#!/usr/bin/Rscript
+library(readr);
+library(dplyr);
+alya_scorep_trace_read <- function(filename)
+{
+ meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[3], "_"));
+ read_csv(filename,
+ col_names=c("Rank", "Start", "End", "Value"),
+ progress=TRUE) %>%
+ # Transform Value to factor
+ mutate(Value = as.factor(Value)) %>%
+ # Detect begin and end of iterations
+ mutate(Iteration = case_when(
+ grepl("timste", .$Value) ~ 1,
+ grepl("endste", .$Value) ~ -1,
+ TRUE ~ 0)) %>%
+ group_by(Rank) %>%
+ # Create a logical to detect observations within iterations
+ mutate(Iteration = as.logical(cumsum(Iteration))) %>%
+ # Get only observations that belongs to some iteration
+ filter(Iteration == TRUE) %>%
+ ungroup() %>%
+ # Create the iteration by cumsum
+ mutate(Iteration = case_when(
+ grepl("timste", .$Value) ~ 1,
+ TRUE ~ 0)) %>%
+ group_by(Rank) %>%
+ mutate(Iteration = as.integer(cumsum(Iteration))) %>%
+ ungroup() %>%
+ # Define metadata
+ mutate(EID = meta[2],
+ Platform = meta[3],
+ Nodes = meta[4],
+ NP = meta[5],
+ Partitioning = meta[6]);
+}
+
+alya_scorep_trace_iterations <- function(filename)
+{
+ alya_scorep_trace_read(filename) %>%
+ group_by(Rank, Iteration, Platform, Nodes, NP, Partitioning, EID) %>%
+ filter(grepl("MPI_", Value)) %>%
+ summarize(N=n(), S=min(Start), E=max(End), Comm=sum(End-Start), Comp=(E-S)-Comm);
+}
+
+args = commandArgs(trailingOnly=TRUE)
+print(args);
+df <- do.call("rbind", lapply(args, function(x) { alya_scorep_trace_iterations(x) }));
+write.csv(df, "exp4_iterations.csv");
+#+end_src
+
+**** Read in R :ATTACH:
+:PROPERTIES:
+:Attachments: exp4_iterations.csv.gz
+:ID: 3b264f0c-58ba-47c0-b291-b5729da5e669
+:END:
+
+#+begin_src R :results output :session :exports both
+library(readr);
+library(dplyr);
+df <- read_csv("data/3b/264f0c-58ba-47c0-b291-b5729da5e669/exp4_iterations.csv.gz") %>%
+ select(-X1) %>%
+ rename(Start = S, End = E) %>%
+ mutate(Duration = End - Start) %>%
+ mutate(Platform = as.factor(Platform),
+ Partitioning = as.factor(Partitioning),
+ Nodes = as.factor(Nodes));
+df %>% summary;
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Parsed with column specification:
+cols(
+ X1 = col_integer(),
+ Rank = col_integer(),
+ Iteration = col_integer(),
+ Platform = col_character(),
+ Nodes = col_integer(),
+ NP = col_integer(),
+ Partitioning = col_character(),
+ EID = col_integer(),
+ N = col_integer(),
+ S = col_double(),
+ E = col_double(),
+ Comm = col_double(),
+ Comp = col_double()
+)
+Warning message:
+Missing column names filled in: 'X1' [1]
+ Rank Iteration Platform Nodes NP
+ Min. : 0.0 Min. : 1.000 grisou:19613 23:19613 Min. : 17.0
+ 1st Qu.: 61.0 1st Qu.: 2.000 1st Qu.:209.0
+ Median :132.0 Median : 5.000 Median :337.0
+ Mean :145.2 Mean : 5.091 Mean :291.5
+ 3rd Qu.:221.0 3rd Qu.: 8.000 3rd Qu.:353.0
+ Max. :367.0 Max. :10.000 Max. :368.0
+ Partitioning EID N Start End
+ metis: 1783 Min. :4 Min. : 3579 Min. : 131.6 Min. : 145.8
+ sfc :17830 1st Qu.:4 1st Qu.: 29737 1st Qu.: 151.6 1st Qu.: 166.4
+ Median :4 Median : 38587 Median : 189.7 Median : 201.3
+ Mean :4 Mean : 39148 Mean : 199.2 Mean : 214.1
+ 3rd Qu.:4 3rd Qu.: 47123 3rd Qu.: 223.9 3rd Qu.: 234.6
+ Max. :4 Max. :103825 Max. :1417.6 Max. :1527.0
+ Comm Comp Duration
+ Min. : 0.539 Min. : 0.01607 Min. : 8.054
+ 1st Qu.: 5.304 1st Qu.: 4.92897 1st Qu.: 10.519
+ Median : 5.926 Median : 5.75274 Median : 11.189
+ Mean : 6.596 Mean : 8.34015 Mean : 14.936
+ 3rd Qu.: 7.522 3rd Qu.: 8.05452 3rd Qu.: 15.129
+ Max. :169.634 Max. :167.55322 Max. :169.672
+#+end_example
+**** Plot
+#+begin_src R :results output :session :exports both
+df %>% filter(Partitioning == "metis") %>% tail
+#+end_src
+
+#+RESULTS:
+#+begin_example
+# A tibble: 6 x 13
+ Rank Iteration Platform Nodes NP Partitioning EID N Start
+
+1 331 1 grisou 23 337 metis 4 19477 149.0451
+2 332 1 grisou 23 337 metis 4 31973 149.0405
+3 333 1 grisou 23 337 metis 4 35097 149.0563
+4 334 1 grisou 23 337 metis 4 35097 149.0551
+5 335 1 grisou 23 337 metis 4 41345 149.0725
+6 336 1 grisou 23 337 metis 4 28849 149.0405
+# ... with 4 more variables: End , Comm , Comp , Duration
+#+end_example
+
+#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 500 :height 400 :session
+library(ggplot2);
+library(tidyr);
+df %>%
+ select(-Start, -End, -N, -Duration) %>%
+ mutate(Total = Comm+Comp) %>%
+ gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID) %>%
+ group_by (Iteration, Platform, Nodes, NP, Partitioning, EID, Variable) %>%
+ summarize(Mean = mean(Value), SE = 3*sd(Value)/sqrt(n()), N=n()) %>%
+ filter(NP == 368) %>%
+ ggplot(aes(x=Iteration, y=Mean, ymin=Mean-SE, ymax=Mean+SE, color=Partitioning)) +
+ theme_bw(base_size=12) +
+ ylim(0,NA) +
+ geom_point() +
+ geom_errorbar(width=.3) +
+ geom_line () +
+ theme(legend.position="top") +
+ facet_grid(NP~Variable)
+#+end_src
+
+#+RESULTS:
+[[file:/tmp/babel-24763lPR/figure24763z4R.png]]
+
+
+
+Unfortunately, Metis as misconfigured in lyon, it ran only 1
+iteration.
+**** Plot with min/max
+
+#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 500 :height 400 :session
+library(ggplot2);
+library(tidyr);
+df %>%
+ filter(Rank != 0) %>%
+ filter(Partitioning == "sfc") %>%
+ select(-Start, -End, -N, -Duration) %>%
+ mutate(Total = Comm+Comp) %>%
+ gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID) %>%
+ filter(NP == 368) %>%
+ ggplot(aes(x=Iteration, y=Value, group=Iteration)) +
+ theme_bw(base_size=12) +
+ ylim(0,NA) +
+ geom_boxplot() +
+ theme(legend.position="top") +
+ facet_grid(Partitioning~Variable)
+#+end_src
+
+#+RESULTS:
+[[file:/tmp/babel-24763lPR/figure24763AKM.png]]
+**** Gantt to check MPI calls
+***** Read the trace
+
+#+begin_src R :results output :session :exports both
+library(readr);
+library(dplyr);
+alya_scorep_trace_read <- function(filename)
+{
+ meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[3], "_"));
+ read_csv(filename,
+ col_names=c("Rank", "Start", "End", "Value"),
+ progress=TRUE) %>%
+ # Transform Value to factor
+ mutate(Value = as.factor(Value)) %>%
+ # Detect begin and end of iterations
+ mutate(Iteration = case_when(
+ grepl("timste", .$Value) ~ 1,
+ grepl("endste", .$Value) ~ -1,
+ TRUE ~ 0)) %>%
+ group_by(Rank) %>%
+ # Create a logical to detect observations within iterations
+ mutate(Iteration = as.logical(cumsum(Iteration))) %>%
+ # Get only observations that belongs to some iteration
+ filter(Iteration == TRUE) %>%
+ ungroup() %>%
+ # Create the iteration by cumsum
+ mutate(Iteration = case_when(
+ grepl("timste", .$Value) ~ 1,
+ TRUE ~ 0)) %>%
+ group_by(Rank) %>%
+ mutate(Iteration = as.integer(cumsum(Iteration))) %>%
+ ungroup() %>%
+ # Define metadata
+ mutate(EID = meta[2],
+ Platform = meta[3],
+ Nodes = meta[4],
+ NP = meta[5],
+ Partitioning = meta[6]);
+}
+TRACE <- "exp_prep-4-v1_grisou_4/prep-4_grisou_4_63_sfc.dir/scorep_prep-4_grisou_4_63_sfc/traces.csv.gz"
+df4.mpi <- alya_scorep_trace_read(TRACE);
+#+end_src
+
+#+RESULTS:
+: Parsed with column specification:
+: cols(
+: Rank = col_integer(),
+: Start = col_double(),
+: End = col_double(),
+: Value = col_character()
+: )
+:
|= | 2% 22 MB
|= | 2% 22 MB
|= | 2% 22 MB
|= | 2% 22 MB
|= | 2% 22 MB
|= | 2% 22 MB
|= | 2% 22 MB
|= | 2% 23 MB
|= | 2% 23 MB
|= | 3% 23 MB
|= | 3% 23 MB
|= | 3% 23 MB
|= | 3% 23 MB
|= | 3% 23 MB
|= | 3% 23 MB
|= | 3% 23 MB
|== | 3% 23 MB
|== | 3% 23 MB
|== | 3% 23 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 30 MB
|== | 3% 30 MB
|== | 3% 30 MB
|== | 3% 30 MB
|== | 3% 30 MB
|== | 3% 30 MB
|== | 3% 30 MB
|== | 3% 30 MB
|== | 3% 30 MB
|== | 3% 30 MB
|== | 4% 30 MB
|== | 4% 30 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 35 MB
|== | 4% 35 MB
|== | 4% 35 MB
|== | 4% 35 MB
|== | 4% 35 MB
|== | 4% 35 MB
|== | 4% 35 MB
|=== | 4% 35 MB
|=== | 4% 35 MB
|=== | 4% 35 MB
|=== | 4% 35 MB
|=== | 4% 35 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 38 MB
|=== | 4% 38 MB
|=== | 4% 38 MB
|=== | 4% 38 MB
|=== | 4% 38 MB
|=== | 4% 38 MB
|=== | 4% 38 MB
|=== | 5% 38 MB
|=== | 5% 38 MB
|=== | 5% 38 MB
|=== | 5% 38 MB
|=== | 5% 38 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 46 MB
|=== | 5% 46 MB
|=== | 5% 46 MB
|=== | 6% 46 MB
|=== | 6% 46 MB
|=== | 6% 46 MB
|=== | 6% 46 MB
|=== | 6% 46 MB
|=== | 6% 46 MB
|=== | 6% 46 MB
|=== | 6% 46 MB
|=== | 6% 46 MB
|=== | 6% 47 MB
|=== | 6% 47 MB
|=== | 6% 47 MB
|=== | 6% 47 MB
|=== | 6% 47 MB
|==== | 6% 47 MB
|==== | 6% 47 MB
|==== | 6% 47 MB
|==== | 6% 47 MB
|==== | 6% 47 MB
|==== | 6% 47 MB
|==== | 6% 47 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 59 MB
|==== | 7% 59 MB
|==== | 7% 59 MB
|==== | 7% 59 MB
|===== | 7% 59 MB
|===== | 7% 59 MB
|===== | 7% 59 MB
|===== | 7% 59 MB
|===== | 7% 59 MB
|===== | 7% 59 MB
|===== | 7% 59 MB
|===== | 7% 59 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 61 MB
|===== | 7% 61 MB
|===== | 7% 61 MB
|===== | 7% 61 MB
|===== | 7% 61 MB
|===== | 7% 61 MB
|===== | 7% 61 MB
|===== | 7% 61 MB
|===== | 8% 61 MB
|===== | 8% 61 MB
|===== | 8% 61 MB
|===== | 8% 61 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 69 MB
|===== | 8% 69 MB
|===== | 8% 69 MB
|===== | 8% 69 MB
|===== | 8% 69 MB
|===== | 9% 69 MB
|===== | 9% 69 MB
|===== | 9% 69 MB
|===== | 9% 69 MB
|===== | 9% 69 MB
|===== | 9% 69 MB
|===== | 9% 69 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 71 MB
|===== | 9% 71 MB
|====== | 9% 71 MB
|====== | 9% 71 MB
|====== | 9% 71 MB
|====== | 9% 71 MB
|====== | 9% 71 MB
|====== | 9% 71 MB
|====== | 9% 71 MB
|====== | 9% 71 MB
|====== | 9% 71 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 77 MB
|====== | 9% 77 MB
|====== | 10% 77 MB
|====== | 10% 77 MB
|====== | 10% 77 MB
|====== | 10% 77 MB
|====== | 10% 77 MB
|====== | 10% 77 MB
|====== | 10% 77 MB
|====== | 10% 77 MB
|====== | 10% 77 MB
|====== | 10% 77 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 84 MB
|======= | 10% 84 MB
|======= | 10% 84 MB
|======= | 10% 84 MB
|======= | 10% 84 MB
|======= | 10% 84 MB
|======= | 10% 84 MB
|======= | 10% 84 MB
|======= | 10% 84 MB
|======= | 10% 84 MB
|======= | 11% 84 MB
|======= | 11% 84 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 92 MB
|======= | 11% 92 MB
|======= | 11% 92 MB
|======= | 11% 92 MB
|======= | 11% 92 MB
|======= | 11% 92 MB
|======= | 12% 92 MB
|======= | 12% 92 MB
|======= | 12% 92 MB
|======= | 12% 92 MB
|======= | 12% 92 MB
|======= | 12% 92 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======== | 12% 94 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 100 MB
|======== | 12% 100 MB
|======== | 12% 100 MB
|======== | 13% 100 MB
|======== | 13% 100 MB
|======== | 13% 100 MB
|======== | 13% 100 MB
|======== | 13% 100 MB
|======== | 13% 100 MB
|======== | 13% 100 MB
|======== | 13% 100 MB
|======== | 13% 100 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 106 MB
|======== | 13% 106 MB
|======== | 13% 106 MB
|======== | 13% 106 MB
|======== | 13% 106 MB
|======== | 13% 106 MB
|======== | 13% 106 MB
|======== | 13% 106 MB
|======== | 13% 106 MB
|========= | 13% 106 MB
|========= | 13% 106 MB
|========= | 13% 106 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 115 MB
|========= | 14% 115 MB
|========= | 14% 115 MB
|========= | 14% 115 MB
|========= | 14% 115 MB
|========= | 14% 115 MB
|========= | 14% 115 MB
|========= | 14% 115 MB
|========= | 15% 115 MB
|========= | 15% 115 MB
|========= | 15% 115 MB
|========= | 15% 115 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 118 MB
|========= | 15% 118 MB
|========= | 15% 118 MB
|========= | 15% 118 MB
|========= | 15% 118 MB
|========= | 15% 118 MB
|========= | 15% 118 MB
|========= | 15% 118 MB
|========== | 15% 118 MB
|========== | 15% 118 MB
|========== | 15% 118 MB
|========== | 15% 118 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 123 MB
|========== | 15% 123 MB
|========== | 15% 123 MB
|========== | 15% 123 MB
|========== | 15% 123 MB
|========== | 16% 123 MB
|========== | 16% 123 MB
|========== | 16% 123 MB
|========== | 16% 123 MB
|========== | 16% 123 MB
|========== | 16% 123 MB
|========== | 16% 123 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 130 MB
|========== | 16% 130 MB
|========== | 16% 130 MB
|========== | 16% 130 MB
|========== | 16% 130 MB
|========== | 16% 130 MB
|=========== | 16% 130 MB
|=========== | 16% 130 MB
|=========== | 16% 130 MB
|=========== | 16% 130 MB
|=========== | 16% 130 MB
|=========== | 16% 131 MB
|=========== | 16% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 138 MB
|=========== | 17% 138 MB
|=========== | 17% 138 MB
|=========== | 17% 138 MB
|=========== | 17% 138 MB
|=========== | 17% 138 MB
|=========== | 17% 138 MB
|=========== | 17% 138 MB
|=========== | 17% 138 MB
|=========== | 18% 138 MB
|=========== | 18% 138 MB
|=========== | 18% 138 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 142 MB
|=========== | 18% 142 MB
|=========== | 18% 142 MB
|=========== | 18% 142 MB
|============ | 18% 142 MB
|============ | 18% 142 MB
|============ | 18% 142 MB
|============ | 18% 142 MB
|============ | 18% 142 MB
|============ | 18% 142 MB
|============ | 18% 142 MB
|============ | 18% 142 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 146 MB
|============ | 18% 146 MB
|============ | 18% 146 MB
|============ | 18% 146 MB
|============ | 18% 146 MB
|============ | 18% 146 MB
|============ | 19% 146 MB
|============ | 19% 146 MB
|============ | 19% 146 MB
|============ | 19% 146 MB
|============ | 19% 146 MB
|============ | 19% 146 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 154 MB
|============ | 19% 154 MB
|============ | 19% 154 MB
|============= | 20% 154 MB
|============= | 20% 154 MB
|============= | 20% 154 MB
|============= | 20% 154 MB
|============= | 20% 154 MB
|============= | 20% 154 MB
|============= | 20% 154 MB
|============= | 20% 154 MB
|============= | 20% 154 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 21% 161 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 169 MB
|============== | 21% 169 MB
|============== | 21% 169 MB
|============== | 21% 169 MB
|============== | 21% 169 MB
|============== | 21% 169 MB
|============== | 21% 169 MB
|============== | 22% 169 MB
|============== | 22% 169 MB
|============== | 22% 169 MB
|============== | 22% 169 MB
|============== | 22% 169 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 177 MB
|============== | 22% 177 MB
|============== | 22% 177 MB
|============== | 22% 177 MB
|============== | 22% 177 MB
|============== | 23% 177 MB
|============== | 23% 177 MB
|============== | 23% 177 MB
|============== | 23% 177 MB
|============== | 23% 177 MB
|============== | 23% 177 MB
|============== | 23% 177 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 189 MB
|=============== | 24% 189 MB
|=============== | 24% 189 MB
|=============== | 24% 189 MB
|=============== | 24% 189 MB
|=============== | 24% 189 MB
|=============== | 24% 189 MB
|=============== | 24% 189 MB
|=============== | 24% 189 MB
|=============== | 24% 189 MB
|================ | 24% 189 MB
|================ | 24% 189 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 192 MB
|================ | 24% 192 MB
|================ | 24% 192 MB
|================ | 24% 192 MB
|================ | 24% 192 MB
|================ | 24% 192 MB
|================ | 24% 192 MB
|================ | 24% 192 MB
|================ | 24% 192 MB
|================ | 25% 192 MB
|================ | 25% 192 MB
|================ | 25% 192 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 200 MB
|================ | 25% 200 MB
|================ | 25% 200 MB
|================ | 25% 200 MB
|================ | 25% 200 MB
|================ | 25% 200 MB
|================ | 26% 200 MB
|================ | 26% 200 MB
|================ | 26% 200 MB
|================ | 26% 200 MB
|================ | 26% 200 MB
|================ | 26% 200 MB
|================ | 26% 201 MB
|================ | 26% 201 MB
|================ | 26% 201 MB
|================ | 26% 201 MB
|================ | 26% 201 MB
|================ | 26% 201 MB
|================ | 26% 201 MB
|================ | 26% 201 MB
|================= | 26% 201 MB
|================= | 26% 201 MB
|================= | 26% 201 MB
|================= | 26% 201 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 208 MB
|================= | 26% 208 MB
|================= | 26% 208 MB
|================= | 27% 208 MB
|================= | 27% 208 MB
|================= | 27% 208 MB
|================= | 27% 208 MB
|================= | 27% 208 MB
|================= | 27% 208 MB
|================= | 27% 208 MB
|================= | 27% 208 MB
|================= | 27% 208 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 213 MB
|================= | 27% 213 MB
|================= | 27% 213 MB
|================= | 27% 213 MB
|================= | 27% 213 MB
|================= | 27% 213 MB
|================= | 27% 213 MB
|================== | 27% 213 MB
|================== | 27% 213 MB
|================== | 27% 213 MB
|================== | 27% 213 MB
|================== | 27% 213 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 28% 215 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 223 MB
|================== | 28% 223 MB
|================== | 28% 223 MB
|================== | 28% 223 MB
|================== | 28% 223 MB
|================== | 28% 223 MB
|================== | 28% 223 MB
|================== | 29% 223 MB
|================== | 29% 223 MB
|================== | 29% 223 MB
|================== | 29% 223 MB
|================== | 29% 223 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 225 MB
|================== | 29% 225 MB
|================== | 29% 225 MB
|================== | 29% 225 MB
|=================== | 29% 225 MB
|=================== | 29% 225 MB
|=================== | 29% 225 MB
|=================== | 29% 225 MB
|=================== | 29% 225 MB
|=================== | 29% 225 MB
|=================== | 29% 225 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 231 MB
|=================== | 29% 231 MB
|=================== | 29% 231 MB
|=================== | 29% 231 MB
|=================== | 30% 231 MB
|=================== | 30% 231 MB
|=================== | 30% 231 MB
|=================== | 30% 231 MB
|=================== | 30% 231 MB
|=================== | 30% 231 MB
|=================== | 30% 231 MB
|=================== | 30% 231 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 237 MB
|=================== | 30% 237 MB
|=================== | 30% 237 MB
|==================== | 30% 237 MB
|==================== | 30% 237 MB
|==================== | 30% 237 MB
|==================== | 30% 237 MB
|==================== | 30% 237 MB
|==================== | 30% 237 MB
|==================== | 30% 237 MB
|==================== | 30% 237 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 246 MB
|==================== | 31% 246 MB
|==================== | 31% 246 MB
|==================== | 31% 246 MB
|==================== | 31% 246 MB
|==================== | 31% 246 MB
|==================== | 31% 246 MB
|==================== | 31% 246 MB
|==================== | 31% 246 MB
|==================== | 32% 246 MB
|==================== | 32% 246 MB
|==================== | 32% 246 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 249 MB
|==================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 254 MB
|===================== | 32% 254 MB
|===================== | 32% 254 MB
|===================== | 32% 254 MB
|===================== | 32% 254 MB
|===================== | 32% 254 MB
|===================== | 33% 254 MB
|===================== | 33% 254 MB
|===================== | 33% 254 MB
|===================== | 33% 254 MB
|===================== | 33% 254 MB
|===================== | 33% 254 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 262 MB
|====================== | 33% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 35% 269 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 272 MB
|====================== | 35% 272 MB
|====================== | 35% 272 MB
|====================== | 35% 272 MB
|====================== | 35% 272 MB
|====================== | 35% 272 MB
|====================== | 35% 272 MB
|====================== | 35% 272 MB
|====================== | 35% 272 MB
|====================== | 35% 272 MB
|======================= | 35% 272 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 277 MB
|======================= | 35% 277 MB
|======================= | 35% 277 MB
|======================= | 35% 277 MB
|======================= | 35% 277 MB
|======================= | 35% 277 MB
|======================= | 35% 277 MB
|======================= | 36% 277 MB
|======================= | 36% 277 MB
|======================= | 36% 277 MB
|======================= | 36% 277 MB
|======================= | 36% 277 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 284 MB
|======================= | 36% 284 MB
|======================= | 36% 284 MB
|======================= | 36% 284 MB
|======================= | 36% 284 MB
|======================= | 36% 284 MB
|======================= | 36% 284 MB
|======================= | 36% 284 MB
|======================= | 36% 284 MB
|======================== | 36% 284 MB
|======================== | 36% 284 MB
|======================== | 36% 284 MB
|======================== | 36% 285 MB
|======================== | 36% 285 MB
|======================== | 36% 285 MB
|======================== | 36% 285 MB
|======================== | 37% 285 MB
|======================== | 37% 285 MB
|======================== | 37% 285 MB
|======================== | 37% 285 MB
|======================== | 37% 285 MB
|======================== | 37% 285 MB
|======================== | 37% 285 MB
|======================== | 37% 285 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 296 MB
|======================== | 38% 296 MB
|======================== | 38% 296 MB
|======================== | 38% 296 MB
|======================== | 38% 296 MB
|======================== | 38% 296 MB
|======================== | 38% 296 MB
|========================= | 38% 296 MB
|========================= | 38% 296 MB
|========================= | 38% 296 MB
|========================= | 38% 296 MB
|========================= | 38% 296 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 300 MB
|========================= | 38% 300 MB
|========================= | 38% 300 MB
|========================= | 38% 300 MB
|========================= | 38% 300 MB
|========================= | 38% 300 MB
|========================= | 38% 300 MB
|========================= | 38% 300 MB
|========================= | 38% 300 MB
|========================= | 39% 300 MB
|========================= | 39% 300 MB
|========================= | 39% 300 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 308 MB
|========================= | 39% 308 MB
|========================= | 39% 308 MB
|========================= | 39% 308 MB
|========================= | 39% 308 MB
|========================== | 40% 308 MB
|========================== | 40% 308 MB
|========================== | 40% 308 MB
|========================== | 40% 308 MB
|========================== | 40% 308 MB
|========================== | 40% 308 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 320 MB
|========================== | 41% 320 MB
|========================== | 41% 320 MB
|========================== | 41% 320 MB
|=========================== | 41% 320 MB
|=========================== | 41% 320 MB
|=========================== | 41% 320 MB
|=========================== | 41% 320 MB
|=========================== | 41% 320 MB
|=========================== | 41% 320 MB
|=========================== | 41% 320 MB
|=========================== | 41% 320 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 42% 323 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 331 MB
|=========================== | 42% 331 MB
|=========================== | 42% 331 MB
|=========================== | 42% 331 MB
|=========================== | 42% 331 MB
|=========================== | 42% 331 MB
|=========================== | 42% 331 MB
|=========================== | 43% 331 MB
|=========================== | 43% 331 MB
|=========================== | 43% 331 MB
|=========================== | 43% 331 MB
|=========================== | 43% 331 MB
|=========================== | 43% 332 MB
|=========================== | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 339 MB
|============================ | 43% 339 MB
|============================ | 43% 339 MB
|============================ | 43% 339 MB
|============================ | 44% 339 MB
|============================ | 44% 339 MB
|============================ | 44% 339 MB
|============================ | 44% 339 MB
|============================ | 44% 339 MB
|============================ | 44% 339 MB
|============================ | 44% 339 MB
|============================ | 44% 339 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 354 MB
|============================= | 45% 354 MB
|============================= | 45% 354 MB
|============================= | 45% 354 MB
|============================= | 45% 354 MB
|============================= | 45% 354 MB
|============================= | 45% 354 MB
|============================= | 45% 354 MB
|============================= | 46% 354 MB
|============================= | 46% 354 MB
|============================= | 46% 354 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================== | 46% 355 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 362 MB
|============================== | 46% 362 MB
|============================== | 46% 362 MB
|============================== | 46% 362 MB
|============================== | 46% 362 MB
|============================== | 47% 362 MB
|============================== | 47% 362 MB
|============================== | 47% 362 MB
|============================== | 47% 362 MB
|============================== | 47% 362 MB
|============================== | 47% 362 MB
|============================== | 47% 362 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 367 MB
|============================== | 47% 367 MB
|============================== | 47% 367 MB
|============================== | 47% 367 MB
|============================== | 47% 367 MB
|============================== | 47% 367 MB
|============================== | 47% 367 MB
|============================== | 47% 367 MB
|=============================== | 47% 367 MB
|=============================== | 47% 367 MB
|=============================== | 47% 367 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 370 MB
|=============================== | 47% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 377 MB
|=============================== | 48% 377 MB
|=============================== | 48% 377 MB
|=============================== | 48% 377 MB
|=============================== | 48% 377 MB
|=============================== | 48% 377 MB
|=============================== | 48% 377 MB
|=============================== | 48% 377 MB
|=============================== | 48% 377 MB
|=============================== | 48% 377 MB
|=============================== | 49% 377 MB
|=============================== | 49% 377 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 379 MB
|=============================== | 49% 379 MB
|=============================== | 49% 379 MB
|=============================== | 49% 379 MB
|=============================== | 49% 379 MB
|=============================== | 49% 379 MB
|=============================== | 49% 379 MB
|=============================== | 49% 379 MB
|================================ | 49% 379 MB
|================================ | 49% 379 MB
|================================ | 49% 379 MB
|================================ | 49% 379 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 385 MB
|================================ | 49% 385 MB
|================================ | 49% 385 MB
|================================ | 49% 385 MB
|================================ | 49% 385 MB
|================================ | 49% 385 MB
|================================ | 50% 385 MB
|================================ | 50% 385 MB
|================================ | 50% 385 MB
|================================ | 50% 385 MB
|================================ | 50% 385 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 391 MB
|================================ | 50% 391 MB
|================================ | 50% 391 MB
|================================ | 50% 391 MB
|================================ | 50% 391 MB
|================================ | 50% 391 MB
|================================= | 50% 391 MB
|================================= | 50% 391 MB
|================================= | 50% 391 MB
|================================= | 50% 391 MB
|================================= | 50% 391 MB
|================================= | 50% 391 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 393 MB
|================================= | 50% 393 MB
|================================= | 50% 393 MB
|================================= | 51% 393 MB
|================================= | 51% 393 MB
|================================= | 51% 393 MB
|================================= | 51% 393 MB
|================================= | 51% 393 MB
|================================= | 51% 393 MB
|================================= | 51% 393 MB
|================================= | 51% 393 MB
|================================= | 51% 393 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 403 MB
|================================= | 52% 403 MB
|================================= | 52% 403 MB
|================================= | 52% 403 MB
|================================== | 52% 403 MB
|================================== | 52% 403 MB
|================================== | 52% 403 MB
|================================== | 52% 403 MB
|================================== | 52% 403 MB
|================================== | 52% 403 MB
|================================== | 52% 403 MB
|================================== | 52% 403 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 408 MB
|================================== | 52% 408 MB
|================================== | 52% 408 MB
|================================== | 52% 408 MB
|================================== | 52% 408 MB
|================================== | 52% 408 MB
|================================== | 52% 408 MB
|================================== | 52% 408 MB
|================================== | 53% 408 MB
|================================== | 53% 408 MB
|================================== | 53% 408 MB
|================================== | 53% 408 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 415 MB
|================================== | 53% 415 MB
|================================== | 53% 415 MB
|=================================== | 53% 415 MB
|=================================== | 53% 415 MB
|=================================== | 53% 415 MB
|=================================== | 53% 415 MB
|=================================== | 53% 415 MB
|=================================== | 53% 415 MB
|=================================== | 53% 415 MB
|=================================== | 53% 415 MB
|=================================== | 53% 415 MB
|=================================== | 53% 416 MB
|=================================== | 53% 416 MB
|=================================== | 53% 416 MB
|=================================== | 53% 416 MB
|=================================== | 54% 416 MB
|=================================== | 54% 416 MB
|=================================== | 54% 416 MB
|=================================== | 54% 416 MB
|=================================== | 54% 416 MB
|=================================== | 54% 416 MB
|=================================== | 54% 416 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 424 MB
|=================================== | 54% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 431 MB
|==================================== | 55% 431 MB
|==================================== | 55% 431 MB
|==================================== | 55% 431 MB
|==================================== | 55% 431 MB
|==================================== | 55% 431 MB
|==================================== | 55% 431 MB
|==================================== | 55% 431 MB
|==================================== | 55% 431 MB
|==================================== | 55% 431 MB
|==================================== | 56% 431 MB
|==================================== | 56% 431 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|===================================== | 56% 438 MB
|===================================== | 56% 439 MB
|===================================== | 56% 439 MB
|===================================== | 56% 439 MB
|===================================== | 56% 439 MB
|===================================== | 56% 439 MB
|===================================== | 56% 439 MB
|===================================== | 57% 439 MB
|===================================== | 57% 439 MB
|===================================== | 57% 439 MB
|===================================== | 57% 439 MB
|===================================== | 57% 439 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 447 MB
|===================================== | 57% 447 MB
|===================================== | 57% 447 MB
|===================================== | 58% 447 MB
|===================================== | 58% 447 MB
|===================================== | 58% 447 MB
|===================================== | 58% 447 MB
|===================================== | 58% 447 MB
|===================================== | 58% 447 MB
|===================================== | 58% 447 MB
|===================================== | 58% 447 MB
|===================================== | 58% 447 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 450 MB
|===================================== | 58% 450 MB
|===================================== | 58% 450 MB
|===================================== | 58% 450 MB
|===================================== | 58% 450 MB
|===================================== | 58% 450 MB
|===================================== | 58% 450 MB
|===================================== | 58% 450 MB
|===================================== | 58% 450 MB
|====================================== | 58% 450 MB
|====================================== | 58% 450 MB
|====================================== | 58% 450 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 59% 454 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 462 MB
|====================================== | 59% 462 MB
|====================================== | 59% 462 MB
|====================================== | 59% 462 MB
|====================================== | 59% 462 MB
|====================================== | 59% 462 MB
|====================================== | 59% 462 MB
|====================================== | 59% 462 MB
|======================================= | 60% 462 MB
|======================================= | 60% 462 MB
|======================================= | 60% 462 MB
|======================================= | 60% 462 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 470 MB
|======================================= | 60% 470 MB
|======================================= | 60% 470 MB
|======================================= | 60% 470 MB
|======================================= | 61% 470 MB
|======================================= | 61% 470 MB
|======================================= | 61% 470 MB
|======================================= | 61% 470 MB
|======================================= | 61% 470 MB
|======================================= | 61% 470 MB
|======================================= | 61% 470 MB
|======================================= | 61% 470 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 474 MB
|======================================= | 61% 474 MB
|======================================= | 61% 474 MB
|======================================= | 61% 474 MB
|======================================= | 61% 474 MB
|======================================= | 61% 474 MB
|======================================== | 61% 474 MB
|======================================== | 61% 474 MB
|======================================== | 61% 474 MB
|======================================== | 61% 474 MB
|======================================== | 61% 474 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 478 MB
|======================================== | 61% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 485 MB
|======================================== | 62% 485 MB
|======================================== | 62% 485 MB
|======================================== | 62% 485 MB
|======================================== | 62% 485 MB
|======================================== | 62% 485 MB
|======================================== | 62% 485 MB
|======================================== | 62% 485 MB
|======================================== | 62% 485 MB
|======================================== | 62% 485 MB
|======================================== | 63% 485 MB
|======================================== | 63% 485 MB
|======================================== | 63% 486 MB
|======================================== | 63% 486 MB
|======================================== | 63% 486 MB
|======================================== | 63% 486 MB
|======================================== | 63% 486 MB
|========================================= | 63% 486 MB
|========================================= | 63% 486 MB
|========================================= | 63% 486 MB
|========================================= | 63% 486 MB
|========================================= | 63% 486 MB
|========================================= | 63% 486 MB
|========================================= | 63% 486 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 493 MB
|========================================= | 63% 493 MB
|========================================= | 63% 493 MB
|========================================= | 63% 493 MB
|========================================= | 63% 493 MB
|========================================= | 63% 493 MB
|========================================= | 64% 493 MB
|========================================= | 64% 493 MB
|========================================= | 64% 493 MB
|========================================= | 64% 493 MB
|========================================= | 64% 493 MB
|========================================= | 64% 493 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 498 MB
|========================================= | 64% 498 MB
|========================================= | 64% 498 MB
|========================================== | 64% 498 MB
|========================================== | 64% 498 MB
|========================================== | 64% 498 MB
|========================================== | 64% 498 MB
|========================================== | 64% 498 MB
|========================================== | 64% 498 MB
|========================================== | 64% 498 MB
|========================================== | 64% 498 MB
|========================================== | 64% 498 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 501 MB
|========================================== | 64% 501 MB
|========================================== | 65% 501 MB
|========================================== | 65% 501 MB
|========================================== | 65% 501 MB
|========================================== | 65% 501 MB
|========================================== | 65% 501 MB
|========================================== | 65% 501 MB
|========================================== | 65% 501 MB
|========================================== | 65% 501 MB
|========================================== | 65% 501 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 66% 508 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 516 MB
|=========================================== | 66% 516 MB
|=========================================== | 66% 516 MB
|=========================================== | 66% 516 MB
|=========================================== | 66% 516 MB
|=========================================== | 66% 516 MB
|=========================================== | 66% 516 MB
|=========================================== | 66% 516 MB
|=========================================== | 67% 516 MB
|=========================================== | 67% 516 MB
|=========================================== | 67% 516 MB
|=========================================== | 67% 516 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|============================================ | 67% 521 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 524 MB
|============================================ | 67% 524 MB
|============================================ | 67% 524 MB
|============================================ | 67% 524 MB
|============================================ | 68% 524 MB
|============================================ | 68% 524 MB
|============================================ | 68% 524 MB
|============================================ | 68% 524 MB
|============================================ | 68% 524 MB
|============================================ | 68% 524 MB
|============================================ | 68% 524 MB
|============================================ | 68% 524 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 533 MB
|============================================ | 69% 533 MB
|============================================ | 69% 533 MB
|============================================ | 69% 533 MB
|============================================ | 69% 533 MB
|============================================ | 69% 533 MB
|============================================ | 69% 533 MB
|============================================ | 69% 533 MB
|============================================ | 69% 533 MB
|============================================ | 69% 533 MB
|============================================= | 69% 533 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 539 MB
|============================================= | 69% 539 MB
|============================================= | 69% 539 MB
|============================================= | 69% 539 MB
|============================================= | 69% 539 MB
|============================================= | 69% 539 MB
|============================================= | 69% 539 MB
|============================================= | 69% 539 MB
|============================================= | 69% 539 MB
|============================================= | 70% 539 MB
|============================================= | 70% 539 MB
|============================================= | 70% 539 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 545 MB
|============================================= | 70% 545 MB
|============================================= | 70% 545 MB
|============================================= | 70% 545 MB
|============================================= | 70% 545 MB
|============================================= | 70% 545 MB
|============================================= | 70% 545 MB
|============================================= | 70% 545 MB
|============================================== | 70% 545 MB
|============================================== | 70% 545 MB
|============================================== | 70% 545 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 547 MB
|============================================== | 70% 547 MB
|============================================== | 70% 547 MB
|============================================== | 70% 547 MB
|============================================== | 70% 547 MB
|============================================== | 70% 547 MB
|============================================== | 71% 547 MB
|============================================== | 71% 547 MB
|============================================== | 71% 547 MB
|============================================== | 71% 547 MB
|============================================== | 71% 547 MB
|============================================== | 71% 547 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 555 MB
|============================================== | 71% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 557 MB
|============================================== | 72% 557 MB
|============================================== | 72% 557 MB
|============================================== | 72% 557 MB
|============================================== | 72% 557 MB
|============================================== | 72% 557 MB
|============================================== | 72% 557 MB
|=============================================== | 72% 557 MB
|=============================================== | 72% 557 MB
|=============================================== | 72% 557 MB
|=============================================== | 72% 557 MB
|=============================================== | 72% 557 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 73% 562 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 569 MB
|=============================================== | 73% 569 MB
|=============================================== | 73% 569 MB
|=============================================== | 73% 569 MB
|=============================================== | 73% 569 MB
|================================================ | 73% 569 MB
|================================================ | 73% 569 MB
|================================================ | 73% 569 MB
|================================================ | 73% 569 MB
|================================================ | 73% 569 MB
|================================================ | 73% 569 MB
|================================================ | 73% 569 MB
|================================================ | 73% 570 MB
|================================================ | 73% 570 MB
|================================================ | 73% 570 MB
|================================================ | 73% 570 MB
|================================================ | 73% 570 MB
|================================================ | 73% 570 MB
|================================================ | 73% 570 MB
|================================================ | 73% 570 MB
|================================================ | 74% 570 MB
|================================================ | 74% 570 MB
|================================================ | 74% 570 MB
|================================================ | 74% 570 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 578 MB
|================================================ | 74% 578 MB
|================================================ | 74% 578 MB
|================================================ | 74% 578 MB
|================================================ | 75% 578 MB
|================================================ | 75% 578 MB
|================================================ | 75% 578 MB
|================================================ | 75% 578 MB
|================================================ | 75% 578 MB
|================================================ | 75% 578 MB
|================================================ | 75% 578 MB
|================================================ | 75% 578 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 581 MB
|================================================ | 75% 581 MB
|================================================ | 75% 581 MB
|================================================ | 75% 581 MB
|================================================= | 75% 581 MB
|================================================= | 75% 581 MB
|================================================= | 75% 581 MB
|================================================= | 75% 581 MB
|================================================= | 75% 581 MB
|================================================= | 75% 581 MB
|================================================= | 75% 581 MB
|================================================= | 75% 581 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 593 MB
|================================================= | 76% 593 MB
|================================================== | 76% 593 MB
|================================================== | 76% 593 MB
|================================================== | 76% 593 MB
|================================================== | 76% 593 MB
|================================================== | 76% 593 MB
|================================================== | 76% 593 MB
|================================================== | 76% 593 MB
|================================================== | 77% 593 MB
|================================================== | 77% 593 MB
|================================================== | 77% 593 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 601 MB
|================================================== | 77% 601 MB
|================================================== | 77% 601 MB
|================================================== | 77% 601 MB
|================================================== | 77% 601 MB
|================================================== | 77% 601 MB
|================================================== | 78% 601 MB
|================================================== | 78% 601 MB
|================================================== | 78% 601 MB
|================================================== | 78% 601 MB
|================================================== | 78% 601 MB
|================================================== | 78% 601 MB
|================================================== | 78% 601 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 609 MB
|=================================================== | 78% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|==================================================== | 80% 616 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 624 MB
|==================================================== | 80% 624 MB
|==================================================== | 80% 624 MB
|==================================================== | 80% 624 MB
|==================================================== | 80% 624 MB
|==================================================== | 80% 624 MB
|==================================================== | 80% 624 MB
|==================================================== | 80% 624 MB
|==================================================== | 81% 624 MB
|==================================================== | 81% 624 MB
|==================================================== | 81% 624 MB
|==================================================== | 81% 624 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 628 MB
|==================================================== | 81% 628 MB
|==================================================== | 81% 628 MB
|==================================================== | 81% 628 MB
|==================================================== | 81% 628 MB
|==================================================== | 81% 628 MB
|==================================================== | 81% 628 MB
|==================================================== | 81% 628 MB
|==================================================== | 81% 628 MB
|===================================================== | 81% 628 MB
|===================================================== | 81% 628 MB
|===================================================== | 81% 628 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 632 MB
|===================================================== | 81% 632 MB
|===================================================== | 81% 632 MB
|===================================================== | 81% 632 MB
|===================================================== | 82% 632 MB
|===================================================== | 82% 632 MB
|===================================================== | 82% 632 MB
|===================================================== | 82% 632 MB
|===================================================== | 82% 632 MB
|===================================================== | 82% 632 MB
|===================================================== | 82% 632 MB
|===================================================== | 82% 632 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 83% 640 MB
|===================================================== | 83% 640 MB
|===================================================== | 83% 640 MB
|===================================================== | 83% 640 MB
|===================================================== | 83% 640 MB
|===================================================== | 83% 640 MB
|===================================================== | 83% 640 MB
|====================================================== | 83% 640 MB
|====================================================== | 83% 640 MB
|====================================================== | 83% 640 MB
|====================================================== | 83% 640 MB
|====================================================== | 83% 640 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 647 MB
|====================================================== | 83% 647 MB
|====================================================== | 83% 647 MB
|====================================================== | 83% 647 MB
|====================================================== | 83% 647 MB
|====================================================== | 83% 647 MB
|====================================================== | 83% 647 MB
|====================================================== | 83% 647 MB
|====================================================== | 83% 647 MB
|====================================================== | 84% 647 MB
|====================================================== | 84% 647 MB
|====================================================== | 84% 647 MB
|====================================================== | 84% 647 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 652 MB
|====================================================== | 84% 652 MB
|====================================================== | 84% 652 MB
|====================================================== | 84% 652 MB
|====================================================== | 84% 652 MB
|====================================================== | 84% 652 MB
|======================================================= | 84% 652 MB
|======================================================= | 84% 652 MB
|======================================================= | 84% 652 MB
|======================================================= | 84% 652 MB
|======================================================= | 84% 652 MB
|======================================================= | 84% 652 MB
|======================================================= | 84% 652 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 655 MB
|======================================================= | 84% 655 MB
|======================================================= | 84% 655 MB
|======================================================= | 84% 655 MB
|======================================================= | 84% 655 MB
|======================================================= | 84% 655 MB
|======================================================= | 85% 655 MB
|======================================================= | 85% 655 MB
|======================================================= | 85% 655 MB
|======================================================= | 85% 655 MB
|======================================================= | 85% 655 MB
|======================================================= | 85% 655 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 663 MB
|======================================================= | 85% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 664 MB
|======================================================= | 86% 664 MB
|======================================================= | 86% 664 MB
|======================================================= | 86% 664 MB
|======================================================== | 86% 664 MB
|======================================================== | 86% 664 MB
|======================================================== | 86% 664 MB
|======================================================== | 86% 664 MB
|======================================================== | 86% 664 MB
|======================================================== | 86% 664 MB
|======================================================== | 86% 664 MB
|======================================================== | 86% 664 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 670 MB
|======================================================== | 86% 670 MB
|======================================================== | 86% 670 MB
|======================================================== | 86% 670 MB
|======================================================== | 86% 670 MB
|======================================================== | 86% 670 MB
|======================================================== | 86% 670 MB
|======================================================== | 86% 670 MB
|======================================================== | 86% 670 MB
|======================================================== | 86% 670 MB
|======================================================== | 87% 670 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 676 MB
|======================================================== | 87% 676 MB
|======================================================== | 87% 676 MB
|========================================================= | 87% 676 MB
|========================================================= | 87% 676 MB
|========================================================= | 87% 676 MB
|========================================================= | 87% 676 MB
|========================================================= | 87% 676 MB
|========================================================= | 87% 676 MB
|========================================================= | 87% 676 MB
|========================================================= | 87% 676 MB
|========================================================= | 87% 676 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 678 MB
|========================================================= | 87% 678 MB
|========================================================= | 87% 678 MB
|========================================================= | 87% 678 MB
|========================================================= | 87% 678 MB
|========================================================= | 87% 678 MB
|========================================================= | 87% 678 MB
|========================================================= | 88% 678 MB
|========================================================= | 88% 678 MB
|========================================================= | 88% 678 MB
|========================================================= | 88% 678 MB
|========================================================= | 88% 678 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 686 MB
|========================================================= | 88% 686 MB
|========================================================= | 88% 686 MB
|========================================================= | 89% 686 MB
|========================================================= | 89% 686 MB
|========================================================= | 89% 686 MB
|========================================================= | 89% 686 MB
|========================================================= | 89% 686 MB
|========================================================= | 89% 686 MB
|========================================================= | 89% 686 MB
|========================================================= | 89% 686 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 701 MB
|=========================================================== | 90% 701 MB
|=========================================================== | 90% 701 MB
|=========================================================== | 90% 701 MB
|=========================================================== | 90% 701 MB
|=========================================================== | 90% 701 MB
|=========================================================== | 90% 701 MB
|=========================================================== | 90% 701 MB
|=========================================================== | 90% 701 MB
|=========================================================== | 91% 701 MB
|=========================================================== | 91% 701 MB
|=========================================================== | 91% 701 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 709 MB
|=========================================================== | 91% 709 MB
|=========================================================== | 91% 709 MB
|=========================================================== | 91% 709 MB
|=========================================================== | 91% 709 MB
|=========================================================== | 92% 709 MB
|=========================================================== | 92% 709 MB
|=========================================================== | 92% 709 MB
|=========================================================== | 92% 709 MB
|=========================================================== | 92% 709 MB
|=========================================================== | 92% 709 MB
|=========================================================== | 92% 709 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 711 MB
|=========================================================== | 92% 711 MB
|=========================================================== | 92% 711 MB
|=========================================================== | 92% 711 MB
|=========================================================== | 92% 711 MB
|=========================================================== | 92% 711 MB
|=========================================================== | 92% 711 MB
|=========================================================== | 92% 711 MB
|=========================================================== | 92% 711 MB
|============================================================ | 92% 711 MB
|============================================================ | 92% 711 MB
|============================================================ | 92% 711 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 717 MB
|============================================================ | 92% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 723 MB
|============================================================ | 93% 723 MB
|============================================================ | 93% 723 MB
|============================================================ | 93% 723 MB
|============================================================ | 93% 723 MB
|============================================================ | 93% 723 MB
|============================================================ | 93% 723 MB
|============================================================= | 93% 723 MB
|============================================================= | 93% 723 MB
|============================================================= | 93% 723 MB
|============================================================= | 93% 723 MB
|============================================================= | 93% 724 MB
|============================================================= | 93% 724 MB
|============================================================= | 93% 724 MB
|============================================================= | 93% 724 MB
|============================================================= | 93% 724 MB
|============================================================= | 93% 724 MB
|============================================================= | 93% 724 MB
|============================================================= | 93% 724 MB
|============================================================= | 93% 724 MB
|============================================================= | 93% 724 MB
|============================================================= | 94% 724 MB
|============================================================= | 94% 724 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 732 MB
|============================================================= | 94% 732 MB
|============================================================= | 94% 732 MB
|============================================================= | 94% 732 MB
|============================================================= | 94% 732 MB
|============================================================= | 94% 732 MB
|============================================================= | 94% 732 MB
|============================================================= | 95% 732 MB
|============================================================= | 95% 732 MB
|============================================================= | 95% 732 MB
|============================================================= | 95% 732 MB
|============================================================= | 95% 732 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 735 MB
|============================================================= | 95% 735 MB
|============================================================= | 95% 735 MB
|============================================================= | 95% 735 MB
|============================================================= | 95% 735 MB
|============================================================= | 95% 735 MB
|============================================================== | 95% 735 MB
|============================================================== | 95% 735 MB
|============================================================== | 95% 735 MB
|============================================================== | 95% 735 MB
|============================================================== | 95% 735 MB
|============================================================== | 95% 735 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 740 MB
|============================================================== | 95% 740 MB
|============================================================== | 95% 740 MB
|============================================================== | 96% 740 MB
|============================================================== | 96% 740 MB
|============================================================== | 96% 740 MB
|============================================================== | 96% 740 MB
|============================================================== | 96% 740 MB
|============================================================== | 96% 740 MB
|============================================================== | 96% 740 MB
|============================================================== | 96% 740 MB
|============================================================== | 96% 740 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 747 MB
|============================================================== | 96% 747 MB
|============================================================== | 96% 747 MB
|============================================================== | 96% 747 MB
|============================================================== | 96% 747 MB
|=============================================================== | 96% 747 MB
|=============================================================== | 96% 747 MB
|=============================================================== | 96% 747 MB
|=============================================================== | 96% 747 MB
|=============================================================== | 96% 747 MB
|=============================================================== | 96% 747 MB
|=============================================================== | 96% 747 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 755 MB
|=============================================================== | 97% 755 MB
|=============================================================== | 97% 755 MB
|=============================================================== | 97% 755 MB
|=============================================================== | 97% 755 MB
|=============================================================== | 97% 755 MB
|=============================================================== | 97% 755 MB
|=============================================================== | 97% 755 MB
|=============================================================== | 98% 755 MB
|=============================================================== | 98% 755 MB
|=============================================================== | 98% 755 MB
|=============================================================== | 98% 755 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 759 MB
|=============================================================== | 98% 759 MB
|=============================================================== | 98% 759 MB
|================================================================| 98% 759 MB
|================================================================| 98% 759 MB
|================================================================| 98% 759 MB
|================================================================| 98% 759 MB
|================================================================| 98% 759 MB
|================================================================| 98% 759 MB
|================================================================| 98% 759 MB
|================================================================| 98% 759 MB
|================================================================| 98% 759 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 763 MB
|================================================================| 98% 763 MB
|================================================================| 98% 763 MB
|================================================================| 98% 763 MB
|================================================================| 98% 763 MB
|================================================================| 99% 763 MB
|================================================================| 99% 763 MB
|================================================================| 99% 763 MB
|================================================================| 99% 763 MB
|================================================================| 99% 763 MB
|================================================================| 99% 763 MB
|================================================================| 99% 763 MB
|================================================================| 99% 763 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 771 MB
|=================================================================| 100% 771 MB
+
+#+begin_src R :results output :session :exports both
+df4.mpi %>%
+ filter(Iteration == 3) %>%
+ filter(Value != "timste") %>%
+ filter(grepl("MPI_", Value)) %>%
+ group_by(Value) %>%
+ summarize(Mean = sum(End-Start), N=n())
+#+end_src
+
+#+RESULTS:
+#+begin_example
+# A tibble: 8 x 3
+ Value Mean N
+
+1 MPI_Allreduce 356.860608 304794
+2 MPI_Barrier 60.943087 1575
+3 MPI_Comm_rank 0.000148 315
+4 MPI_Comm_size 0.000251 315
+5 MPI_Irecv 2.528487 1023948
+6 MPI_Isend 12.640539 1023948
+7 MPI_Sendrecv 329.724298 47880
+8 MPI_Waitall 46.962233 92814
+#+end_example
+
+***** Plot
+
+#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 1700 :height 800 :session
+df4.mpi %>%
+ filter(Rank != 0) %>%
+ filter(Iteration == 7) %>%
+ filter(End-Start > 0.1) -> tx;
+tx %>%
+ group_by(Iteration) %>%
+ mutate(End = End - min(Start),
+ Start = Start - min(Start)) %>%
+ ungroup() %>%
+ ggplot() +
+ facet_wrap(~Iteration) +
+ geom_rect(aes(fill=Value,
+ xmin=Start,
+ xmax=End,
+ ymin=Rank,
+ ymax=Rank+0.9));
+#+end_src
+
+#+RESULTS:
+[[file:/tmp/babel-24763lPR/figure24763btH.png]]
+
+#+begin_src R :results output :session :exports both
+tx %>% group_by(Rank, Value, Iteration) %>% summarize(N=n())
+#+end_src
+
+#+RESULTS:
+#+begin_example
+# A tibble: 1,407 x 4
+# Groups: Rank, Value [?]
+ Rank Value Iteration N
+
+ 1 0 solver 150 4
+ 2 1 nsi_matrix 150 1
+ 3 1 solver 150 4
+ 4 2 nsi_matrix 150 1
+ 5 2 solver 150 4
+ 6 3 nsi_matrix 150 1
+ 7 3 solver 150 4
+ 8 4 nsi_matrix 150 1
+ 9 4 solver 150 4
+10 5 nsi_matrix 150 1
+# ... with 1,397 more rows
+#+end_example
+
+***** Calculate Comm / Comp for this prep trace
+
+#+begin_src R :results output :session :exports both
+df4.mpi %>%
+ group_by(Rank, Iteration, Platform, Nodes, NP, Partitioning, EID) %>%
+ filter(grepl("MPI_", Value)) %>%
+ summarize(N=n(), S=min(Start), E=max(End), Comm=sum(End-Start), Comp=(E-S)-Comm) -> df4.sum;
+#+end_src
+
+#+RESULTS:
+
+#+begin_src R :results output :session :exports both
+df4.sum
+#+end_src
+
+#+RESULTS:
+#+begin_example
+# A tibble: 630 x 12
+# Groups: Rank, Iteration, Platform, Nodes, NP, Partitioning [?]
+ Rank Iteration Platform Nodes NP Partitioning EID N S
+
+ 1 0 1 grisou 4 63 sfc prep-4 5454 129.7846
+ 2 0 2 grisou 4 63 sfc prep-4 4752 174.9951
+ 3 0 3 grisou 4 63 sfc prep-4 4873 219.0275
+ 4 0 4 grisou 4 63 sfc prep-4 4697 263.0684
+ 5 0 5 grisou 4 63 sfc prep-4 4660 306.4143
+ 6 0 6 grisou 4 63 sfc prep-4 4544 349.4879
+ 7 0 7 grisou 4 63 sfc prep-4 4448 391.9349
+ 8 0 8 grisou 4 63 sfc prep-4 4324 433.9185
+ 9 0 9 grisou 4 63 sfc prep-4 4263 475.6839
+10 0 10 grisou 4 63 sfc prep-4 3579 517.5147
+# ... with 620 more rows, and 3 more variables: E , Comm , Comp
+#+end_example
+
+#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 500 :height 400 :session
+library(ggplot2);
+library(tidyr);
+df4.sum %>%
+ filter(Rank != 0) %>%
+ mutate(Total = Comm+Comp) %>%
+ select(-N, -S, -E) %>%
+ gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID) %>%
+ ggplot(aes(x=Iteration, y=Value, group=Iteration)) +
+ theme_bw(base_size=12) +
+ ylim(0,NA) +
+ geom_boxplot() +
+ theme(legend.position="top") +
+ facet_wrap(~Variable)
+#+end_src
+
+#+RESULTS:
+[[file:/tmp/babel-24763lPR/figure24763CTO.png]]
+
+** 50-node grisou :EXP5:
+** 7-node grimoire with *Infiniband* :deprecated:EXP6:
+*** First attempt
+Experiment did not work because Infiniband was not chosen by MPI.
+The message tells me this:
+#+BEGIN_EXAMPLE
+No OpenFabrics connection schemes reported that they were able to be
+used on a specific port. As such, the openib BTL (OpenFabrics
+support) will be disabled for this port.
+
+ Local host: grimoire-4
+ Local device: mlx4_0
+ Local port: 1
+ CPCs attempted: udcm
+#+END_EXAMPLE
+*** Second attempt
+JobId 1227284 @ nancy.
+**** Convert to CSV
+I need on the $PATH:
+- akypuera's otf22paje (compiled using OTF2 libraries of ScoreP 3.0)
+- pajeng's =pj_dump=
+
+#+begin_src shell :results output
+export PATH=$PATH:/home/lschnorr/akypuera/b/
+export PATH=$PATH:/home/lschnorr/pajeng/b/
+
+convert() {
+ pushd $(dirname $otf2)
+ otf22paje traces.otf2 | pj_dump | grep ^State | cut -d, -f2,4,5,8 | sed -e "s/ //g" -e "s/MPIRank//" | gzip > traces.csv.gz
+ popd
+}
+
+EDIR=exp_6_grimoire_7
+# Files already converted (whose CSV size is not zero)
+EXISTINGFILE=$(tempfile)
+OTF2FILE=$(tempfile)
+find $EDIR -not -empty | grep csv$ | sed -e "s/.csv.gz$//" | sort > $EXISTINGFILE
+find $EDIR | grep otf2 | sed -e "s/.otf2$//" | sort > $OTF2FILE
+
+for otf2 in $(comm -3 $OTF2FILE $EXISTINGFILE | sed "s/$/.otf2/"); do
+ echo $otf2
+ convert $otf2
+done
+#+end_src
+**** Post-processing in R
+#+begin_src R :results output :session :exports both :tangle do-exp6.R :tangle-mode (identity #o755)
+#!/usr/bin/Rscript
+library(readr);
+library(dplyr);
+alya_scorep_trace_read <- function(filename)
+{
+ meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[3], "_"));
+ read_csv(filename,
+ col_names=c("Rank", "Start", "End", "Value"),
+ progress=TRUE) %>%
+ # Transform Value to factor
+ mutate(Value = as.factor(Value)) %>%
+ # Detect begin and end of iterations
+ mutate(Iteration = case_when(
+ grepl("timste", .$Value) ~ 1,
+ grepl("endste", .$Value) ~ -1,
+ TRUE ~ 0)) %>%
+ group_by(Rank) %>%
+ # Create a logical to detect observations within iterations
+ mutate(Iteration = as.logical(cumsum(Iteration))) %>%
+ # Get only observations that belongs to some iteration
+ filter(Iteration == TRUE) %>%
+ ungroup() %>%
+ # Create the iteration by cumsum
+ mutate(Iteration = case_when(
+ grepl("timste", .$Value) ~ 1,
+ TRUE ~ 0)) %>%
+ group_by(Rank) %>%
+ mutate(Iteration = as.integer(cumsum(Iteration))) %>%
+ ungroup() %>%
+ # Define metadata
+ mutate(EID = meta[2],
+ Platform = meta[3],
+ Nodes = meta[4],
+ NP = meta[5],
+ Partitioning = meta[6]);
+}
+
+alya_scorep_trace_iterations <- function(filename)
+{
+ alya_scorep_trace_read(filename) %>%
+ group_by(Rank, Iteration, Platform, Nodes, NP, Partitioning, EID) %>%
+ filter(grepl("MPI_", Value)) %>%
+ summarize(N=n(), S=min(Start), E=max(End), Comm=sum(End-Start), Comp=(E-S)-Comm);
+}
+
+args = commandArgs(trailingOnly=TRUE)
+print(args);
+df <- do.call("rbind", lapply(args, function(x) { alya_scorep_trace_iterations(x) }));
+write.csv(df, "exp6_iterations.csv");
+#+end_src
+**** Read in R :ATTACH:
+:PROPERTIES:
+:Attachments: exp6_iterations.csv.gz
+:ID: 4e48b93e-3322-4cc1-9ba9-3145d1298878
+:END:
+
+***** Reading
+#+begin_src R :results output :session :exports both
+library(readr);
+library(dplyr);
+df <- read_csv("data/4e/48b93e-3322-4cc1-9ba9-3145d1298878/exp6_iterations.csv.gz") %>%
+ select(-X1) %>%
+ rename(Start = S, End = E) %>%
+ mutate(Duration = End - Start) %>%
+ mutate(Platform = as.factor(Platform),
+ Partitioning = as.factor(Partitioning),
+ Nodes = as.factor(Nodes));
+df %>% summary;
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Parsed with column specification:
+cols(
+ X1 = col_integer(),
+ Rank = col_integer(),
+ Iteration = col_integer(),
+ Platform = col_character(),
+ Nodes = col_integer(),
+ NP = col_integer(),
+ Partitioning = col_character(),
+ EID = col_integer(),
+ N = col_integer(),
+ S = col_double(),
+ E = col_double(),
+ Comm = col_double(),
+ Comp = col_double()
+)
+Warning message:
+Missing column names filled in: 'X1' [1]
+ Rank Iteration Platform Nodes NP
+ Min. : 0.00 Min. : 1.0 grimoire:4180 7:4180 Min. : 97
+ 1st Qu.: 26.00 1st Qu.: 3.0 1st Qu.: 97
+ Median : 52.00 Median : 5.5 Median :112
+ Mean : 52.02 Mean : 5.5 Mean :105
+ 3rd Qu.: 78.00 3rd Qu.: 8.0 3rd Qu.:112
+ Max. :111.00 Max. :10.0 Max. :112
+ Partitioning EID N Start End
+ metis:2090 Min. :6 Min. : 3579 Min. :122.2 Min. :149.6
+ sfc :2090 1st Qu.:6 1st Qu.:16353 1st Qu.:176.8 1st Qu.:200.4
+ Median :6 Median :23759 Median :243.1 Median :267.8
+ Mean :6 Mean :27314 Mean :240.3 Mean :264.6
+ 3rd Qu.:6 3rd Qu.:36328 3rd Qu.:295.3 3rd Qu.:319.3
+ Max. :6 Max. :85081 Max. :372.1 Max. :392.9
+ Comm Comp Duration
+ Min. : 0.5519 Min. : 0.0178 Min. :18.05
+ 1st Qu.: 8.4911 1st Qu.:12.9418 1st Qu.:22.99
+ Median :10.0122 Median :14.4247 Median :24.01
+ Mean : 9.7437 Mean :14.5349 Mean :24.28
+ 3rd Qu.:11.0630 3rd Qu.:15.8340 3rd Qu.:26.13
+ Max. :28.4181 Max. :27.2230 Max. :28.45
+#+end_example
+***** Plot
+#+begin_src R :results output :session :exports both
+df %>% select(-Start, -End, -N, -Duration) %>% mutate(Total = Comm+Comp)
+#+end_src
+
+#+RESULTS:
+#+begin_example
+# A tibble: 4,180 × 10
+ Rank Iteration Platform Nodes NP Partitioning EID Comm Comp
+
+1 0 1 grimoire 7 97 metis 6 28.41812 0.028814
+2 0 2 grimoire 7 97 metis 6 27.58274 0.024577
+3 0 3 grimoire 7 97 metis 6 27.34581 0.024256
+4 0 4 grimoire 7 97 metis 6 27.04863 0.023692
+5 0 5 grimoire 7 97 metis 6 26.84089 0.023234
+6 0 6 grimoire 7 97 metis 6 26.69917 0.022638
+7 0 7 grimoire 7 97 metis 6 26.48485 0.022195
+8 0 8 grimoire 7 97 metis 6 26.27733 0.021815
+9 0 9 grimoire 7 97 metis 6 26.06916 0.021781
+10 0 10 grimoire 7 97 metis 6 21.13847 0.017997
+# ... with 4,170 more rows, and 1 more variables: Total
+#+end_example
+
+#+begin_src R :results output :session :exports both
+library(tidyr);
+df %>%
+ select(-Start, -End, -N, -Duration) %>%
+ mutate(Total = Comm+Comp) %>%
+ gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID);
+#+end_src
+
+#+RESULTS:
+#+begin_example
+# A tibble: 12,540 × 9
+ Rank Iteration Platform Nodes NP Partitioning EID Variable Value
+
+1 0 1 grimoire 7 97 metis 6 Comm 28.41812
+2 0 2 grimoire 7 97 metis 6 Comm 27.58274
+3 0 3 grimoire 7 97 metis 6 Comm 27.34581
+4 0 4 grimoire 7 97 metis 6 Comm 27.04863
+5 0 5 grimoire 7 97 metis 6 Comm 26.84089
+6 0 6 grimoire 7 97 metis 6 Comm 26.69917
+7 0 7 grimoire 7 97 metis 6 Comm 26.48485
+8 0 8 grimoire 7 97 metis 6 Comm 26.27733
+9 0 9 grimoire 7 97 metis 6 Comm 26.06916
+10 0 10 grimoire 7 97 metis 6 Comm 21.13847
+# ... with 12,530 more rows
+#+end_example
+
+#+begin_src R :results output :session :exports both
+df %>%
+ select(-Start, -End, -N, -Duration) %>%
+ mutate(Total = Comm+Comp) %>%
+ gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID) %>%
+ group_by (Iteration, Platform, Nodes, NP, Partitioning, EID, Variable) %>%
+ summarize(Mean = mean(Value), SE = 3*sd(Value)/sqrt(n()), N=n()) %>%
+ # to check if it worked
+ filter(NP == 97, Partitioning == "metis")
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Source: local data frame [30 x 10]
+Groups: Iteration, Platform, Nodes, NP, Partitioning, EID [10]
+
+ Iteration Platform Nodes NP Partitioning EID Variable Mean
+
+1 1 grimoire 7 97 metis 6 Comm 11.77403
+2 1 grimoire 7 97 metis 6 Comp 16.53532
+3 1 grimoire 7 97 metis 6 Total 28.30936
+4 2 grimoire 7 97 metis 6 Comm 11.32483
+5 2 grimoire 7 97 metis 6 Comp 16.14359
+6 2 grimoire 7 97 metis 6 Total 27.46842
+7 3 grimoire 7 97 metis 6 Comm 11.09919
+8 3 grimoire 7 97 metis 6 Comp 16.13198
+9 3 grimoire 7 97 metis 6 Total 27.23118
+10 4 grimoire 7 97 metis 6 Comm 11.01637
+# ... with 20 more rows, and 2 more variables: SE , N
+#+end_example
+
+
+
+#+begin_src R :results output graphics :file img/exp_6-97np-metis-sfc.pdf :exports both :width 6 :height 3 :session
+library(ggplot2);
+library(tidyr);
+df %>%
+ select(-Start, -End, -N, -Duration) %>%
+ mutate(Total = Comm+Comp, Iteration = as.integer(Iteration)) %>%
+ gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID) %>%
+ group_by (Iteration, Platform, Nodes, NP, Partitioning, EID, Variable) %>%
+ summarize(Mean = mean(Value), SE = 3*sd(Value)/sqrt(n()), N=n()) %>%
+ ggplot(aes(x=Iteration, y=Mean, ymin=Mean-SE, ymax=Mean+SE,
+ color=Partitioning)) +
+ theme_bw(base_size=14) +
+ ylim(0,NA) +
+ xlim(0,10) +
+ scale_x_continuous(breaks=seq(1:10)) +
+ xlab("Iteration [count]") +
+ ylab("Time [seconds]") +
+ geom_point() +
+ geom_errorbar(width=.3) +
+ geom_line () +
+ theme(legend.position=c(.78,.7),
+ panel.margin = unit(.08, "lines"),
+ legend.spacing = unit(.0, "lines"),
+ legend.margin=margin(b = -.4, unit='cm'),
+ plot.margin = unit(x = c(0, 0, 0, 0), units = "mm")
+ ) +
+ facet_grid(NP~Variable)
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_6-97np-metis-sfc.pdf]]
+***** With Arnaud
+#+begin_src R :results output graphics :file img/exp_6-97np_112np-metis-sfc-LB.pdf :exports both :width 6 :height 3 :session
+library(tidyr);
+library(ggplot2);
+df %>%
+ select(-Start, -End, -N, -Duration) %>%
+ mutate(Total = Comm+Comp) %>%
+ gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID) %>%
+ filter(Variable == "Comp", Rank != 0, Iteration != 11) %>%
+ ggplot(aes(x=as.factor(Iteration), y=Value, color=as.factor(Rank))) +
+ theme_bw(base_size=14) +
+ ylim(0,30) +
+ ylab("Computation Time [seconds]") +
+ xlab("Iteration [count]") +
+# ggtitle("Each color is an MPI rank.") +
+ geom_point(alpha=.3) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning))) +
+ theme(legend.position="none",
+ panel.margin = unit(.08, "lines"),
+ legend.spacing = unit(.0, "lines"),
+ legend.margin=margin(b = -.4, unit='cm'),
+ plot.margin = unit(x = c(0, 0, 0, 0), units = "mm")
+ ) +
+ facet_grid(NP~Partitioning)
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_6-97np_112np-metis-sfc-LB.pdf]]
+
+#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 600 :height 400 :session
+df %>%
+ select(-Start, -End, -N, -Duration) %>%
+ mutate(Total = Comm+Comp) %>%
+ gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID) %>%
+ filter(Variable == "Comp", Rank != 0) %>%
+ filter(Iteration == 10) %>%
+ ggplot(aes(x=Value, fill=Partitioning)) + theme_bw(base_size=12) + geom_histogram() + facet_grid(Partitioning~NP);
+#+end_src
+
+#+RESULTS:
+[[file:/tmp/babel-1875bQK/figure1875FAD.png]]
+
+
+Status:
+- Load balance among processes is not equal (considering only computation)
+ - Iteration time is dominated by the slowest rank (numbers)
+ - Such ranks also have the higher number of points as shown in
+ fensap.par.log so we could have guessed in advance that the load
+ balance would be a problem.
+ - Actually, the load balancing is bad for both SFC and METIS
+ although it is slightly better (the max load is lower) for SFC.
+ - Communications are, as expected, more important for SFC than for
+ METIS but when the communication network is good (e.g., ib) this
+ becomes a problem only at large scale (probably above 512
+ nodes as Ricard pointed out). However, when using a commodity
+ network (e.g, eth0), communications are problematic at smaller
+ scale (around above 100). Therefore, no need to go at high scale
+ to study the load balancing problem. It is already very bad at
+ medium scale.
+Questions:
+- Do you know how to control the partitioning (of Metis and SFC),
+ e.g., to disable communication cost, so we have a "perfect" load
+ balance.
+- Do you know how to get the mapping between the average of mesh
+ points coordinates and ranks? This could help us understanding why
+ Metis and SFC have difficulties computing a good load balance. A
+ question we are asking ourselves is whether the heavy loaded
+ processes in both METIS and SFC correspond to the same region of the
+ mesh or not. (ref the fig)
+** 2-node and 4-node grimoire with Infiniband :EXP8:
+*** v1 :ATTACH:
+:PROPERTIES:
+:Attachments: exp_8-v1_grimoire_2.csv.gz
+:ID: faffa111-90b6-4e60-97f0-39475468994c
+:END:
+
+#+begin_src R :results output :session :exports both
+library(readr);
+library(dplyr);
+df <- read_csv("data/fa/ffa111-90b6-4e60-97f0-39475468994c/exp_8-v1_grimoire_2.csv.gz") %>%
+ select(-X1) %>%
+ rename(Start = S, End = E) %>%
+ mutate(Duration = End - Start) %>%
+ mutate(Platform = as.factor(Platform),
+ Partitioning = as.factor(Partitioning),
+ Nodes = as.factor(Nodes));
+df %>% summary;
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Parsed with column specification:
+cols(
+ X1 = col_integer(),
+ Rank = col_integer(),
+ Iteration = col_integer(),
+ Platform = col_character(),
+ Nodes = col_integer(),
+ NP = col_integer(),
+ Partitioning = col_character(),
+ EID = col_character(),
+ Infiniband = col_logical(),
+ N = col_integer(),
+ S = col_double(),
+ E = col_double(),
+ Comm = col_double(),
+ Comp = col_double()
+)
+Warning message:
+Missing column names filled in: 'X1' [1]
+ Rank Iteration Platform Nodes NP
+ Min. : 0.00 Min. : 1.0 grimoire:640 2:640 Min. :32
+ 1st Qu.: 7.75 1st Qu.: 3.0 1st Qu.:32
+ Median :15.50 Median : 5.5 Median :32
+ Mean :15.50 Mean : 5.5 Mean :32
+ 3rd Qu.:23.25 3rd Qu.: 8.0 3rd Qu.:32
+ Max. :31.00 Max. :10.0 Max. :32
+ Partitioning EID Infiniband N
+ metis:640 Length:640 Mode :logical Min. : 3579
+ Class :character FALSE:320 1st Qu.:11028
+ Mode :character TRUE :320 Median :13216
+ NA's :0 Mean :13343
+ 3rd Qu.:15539
+ Max. :22601
+ Start End Comm Comp
+ Min. :127.2 Min. :213.0 Min. : 0.2747 Min. : 0.02016
+ 1st Qu.:298.2 1st Qu.:382.1 1st Qu.:28.2750 1st Qu.:44.78627
+ Median :507.0 Median :588.7 Median :33.1652 Median :47.69400
+ Mean :504.3 Mean :585.0 Mean :33.0316 Mean :47.60811
+ 3rd Qu.:711.2 3rd Qu.:791.1 3rd Qu.:36.7584 3rd Qu.:52.60334
+ Max. :876.5 Max. :941.3 Max. :86.2371 Max. :84.82805
+ Duration
+ Min. :64.25
+ 1st Qu.:80.60
+ Median :82.08
+ Mean :80.64
+ 3rd Qu.:83.44
+ Max. :86.27
+#+end_example
+
+#+begin_src R :results output graphics :file img/exp_8-v1_total.png :exports both :width 500 :height 400 :session
+library(tidyr);
+library(ggplot2);
+df %>%
+ select(-Start, -End, -N, -Duration) %>%
+ mutate(Total = Comm+Comp) %>%
+ gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID, -Infiniband) %>%
+ filter(Variable == "Total", Rank != 0, Partitioning == "metis", NP == 32) %>%
+ ggplot(aes(x=as.factor(Iteration), y=Value, color=Infiniband)) +
+ theme_bw(base_size=12) +
+ geom_point(alpha=.3) +
+ ylim(0,NA) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
+ theme(legend.position="top") +
+ facet_grid(Partitioning~NP)
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_8-v1_total.png]]
+
+They look identical. I think both used Infiniband.
+
+I'll adapt the run alya script, run again to get v2 below.
+*** v2 :ATTACH:
+:PROPERTIES:
+:Attachments: exp_8-v2_grimoire_2.csv.gz
+:ID: 6475f529-6f29-4c6a-8d25-49c879403b6b
+:END:
+
+#+begin_src R :results output :session :exports both
+library(readr);
+library(dplyr);
+df <- read_csv("data/64/75f529-6f29-4c6a-8d25-49c879403b6b/exp_8-v2_grimoire_2.csv.gz") %>%
+ select(-X1) %>%
+ rename(Start = S, End = E) %>%
+ mutate(Duration = End - Start) %>%
+ mutate(Platform = as.factor(Platform),
+ Partitioning = as.factor(Partitioning),
+ Nodes = as.factor(Nodes));
+df %>% summary;
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Parsed with column specification:
+cols(
+ X1 = col_integer(),
+ Rank = col_integer(),
+ Iteration = col_integer(),
+ Platform = col_character(),
+ Nodes = col_integer(),
+ NP = col_integer(),
+ Partitioning = col_character(),
+ EID = col_character(),
+ Infiniband = col_logical(),
+ N = col_integer(),
+ S = col_double(),
+ E = col_double(),
+ Comm = col_double(),
+ Comp = col_double()
+)
+Warning message:
+Missing column names filled in: 'X1' [1]
+ Rank Iteration Platform Nodes NP
+ Min. : 0.00 Min. : 1.0 grimoire:640 2:640 Min. :32
+ 1st Qu.: 7.75 1st Qu.: 3.0 1st Qu.:32
+ Median :15.50 Median : 5.5 Median :32
+ Mean :15.50 Mean : 5.5 Mean :32
+ 3rd Qu.:23.25 3rd Qu.: 8.0 3rd Qu.:32
+ Max. :31.00 Max. :10.0 Max. :32
+ Partitioning EID Infiniband N
+ metis:640 Length:640 Mode :logical Min. : 3579
+ Class :character FALSE:320 1st Qu.:11028
+ Mode :character TRUE :320 Median :13216
+ NA's :0 Mean :13343
+ 3rd Qu.:15539
+ Max. :22601
+ Start End Comm Comp
+ Min. :126.5 Min. :213.5 Min. : 0.3249 Min. : 0.02036
+ 1st Qu.:298.8 1st Qu.:383.4 1st Qu.:28.6923 1st Qu.:44.80856
+ Median :507.5 Median :589.7 Median :33.4850 Median :47.64884
+ Mean :505.2 Mean :586.0 Mean :33.2070 Mean :47.57824
+ 3rd Qu.:710.9 3rd Qu.:790.4 3rd Qu.:36.8898 3rd Qu.:52.68086
+ Max. :882.3 Max. :946.9 Max. :86.9266 Max. :85.24896
+ Duration
+ Min. :63.92
+ 1st Qu.:80.31
+ Median :82.00
+ Mean :80.79
+ 3rd Qu.:83.64
+ Max. :86.96
+#+end_example
+
+#+begin_src R :results output graphics :file img/exp_8-v2_total.png :exports both :width 500 :height 400 :session
+library(tidyr);
+library(ggplot2);
+df %>%
+ select(-Start, -End, -N, -Duration) %>%
+ mutate(Total = Comm+Comp) %>%
+ gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID, -Infiniband) %>%
+ filter(Variable == "Total", Iteration != 10, Rank != 0, Partitioning == "metis", NP == 32) %>%
+ ggplot(aes(x=as.factor(Iteration), y=Value, color=as.factor(Infiniband))) +
+ theme_bw(base_size=12) +
+ geom_point(alpha=.3) +
+ ylim(0,NA) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
+ theme(legend.position="top") +
+ facet_grid(Partitioning~NP)
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_8-v2_total.png]]
+
+Well, it it still very similar. But now at least when Infiniband is
+TRUE, the total makespan of each iteration is smaller.
+
+*** v3 :ATTACH:
+:PROPERTIES:
+:Attachments: exp_8-v3_grimoire_4.csv.gz
+:ID: bd355764-1704-4485-bebd-5e714e76c2be
+:END:
+
+#+begin_src R :results output :session :exports both
+library(readr);
+library(dplyr);
+df <- read_csv("data/bd/355764-1704-4485-bebd-5e714e76c2be/exp_8-v3_grimoire_4.csv.gz") %>%
+ select(-X1) %>%
+ rename(Start = S, End = E) %>%
+ mutate(Duration = End - Start) %>%
+ mutate(Platform = as.factor(Platform),
+ Partitioning = as.factor(Partitioning),
+ Nodes = as.factor(Nodes));
+df %>% summary;
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Parsed with column specification:
+cols(
+ X1 = col_integer(),
+ Rank = col_integer(),
+ Iteration = col_integer(),
+ Platform = col_character(),
+ Nodes = col_integer(),
+ NP = col_integer(),
+ Partitioning = col_character(),
+ EID = col_character(),
+ Infiniband = col_logical(),
+ N = col_integer(),
+ S = col_double(),
+ E = col_double(),
+ Comm = col_double(),
+ Comp = col_double()
+)
+Warning message:
+Missing column names filled in: 'X1' [1]
+ Rank Iteration Platform Nodes NP
+ Min. : 0.00 Min. : 1.0 grimoire:1280 4:1280 Min. :64
+ 1st Qu.:15.75 1st Qu.: 3.0 1st Qu.:64
+ Median :31.50 Median : 5.5 Median :64
+ Mean :31.50 Mean : 5.5 Mean :64
+ 3rd Qu.:47.25 3rd Qu.: 8.0 3rd Qu.:64
+ Max. :63.00 Max. :10.0 Max. :64
+ Partitioning EID Infiniband N
+ metis:1280 Length:1280 Mode :logical Min. : 3579
+ Class :character FALSE:640 1st Qu.:11750
+ Mode :character TRUE :640 Median :14600
+ NA's :0 Mean :15362
+ 3rd Qu.:18626
+ Max. :28849
+ Start End Comm Comp
+ Min. :261.0 Min. :302.7 Min. : 0.3612 Min. : 0.03617
+ 1st Qu.:343.8 1st Qu.:384.1 1st Qu.:13.3089 1st Qu.:21.57136
+ Median :443.8 Median :483.0 Median :15.2729 Median :23.21151
+ Mean :442.5 Mean :481.2 Mean :15.2131 Mean :23.46578
+ 3rd Qu.:541.2 3rd Qu.:579.3 3rd Qu.:17.4194 3rd Qu.:25.69657
+ Max. :621.9 Max. :653.0 Max. :41.7033 Max. :40.66069
+ Duration
+ Min. :30.86
+ 1st Qu.:38.42
+ Median :39.29
+ Mean :38.68
+ 3rd Qu.:39.86
+ Max. :41.76
+#+end_example
+
+#+begin_src R :results output graphics :file img/exp_8-v3_total.png :exports both :width 500 :height 400 :session
+library(tidyr);
+library(ggplot2);
+df %>%
+ select(-Start, -End, -N, -Duration) %>%
+ mutate(Total = Comm+Comp) %>%
+ gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID, -Infiniband) %>%
+ filter(Variable == "Total", Iteration != 10, Rank != 0, Partitioning == "metis", NP == 64) %>%
+ ggplot(aes(x=as.factor(Iteration), y=Value, color=Infiniband)) +
+ theme_bw(base_size=12) +
+ geom_point(alpha=.3) +
+ ylim(0,NA) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
+ theme(legend.position="top") +
+ facet_grid(Partitioning~NP)
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_8-v3_total.png]]
+
+Same as in v3. I think Infiniband set to TRUE is already working.
+
+*** v4 :ATTACH:
+:PROPERTIES:
+:Attachments: exp_8-v4_grimoire_4.csv.gz
+:ID: 96a5786d-68c5-4d6c-9a54-6947b6fd5021
+:END:
+#+begin_src R :results output :session :exports both
+library(readr);
+library(dplyr);
+df <- read_csv("data/96/a5786d-68c5-4d6c-9a54-6947b6fd5021/exp_8-v4_grimoire_4.csv.gz") %>%
+ select(-X1) %>%
+ rename(Start = S, End = E) %>%
+ mutate(Duration = End - Start) %>%
+ mutate(Platform = as.factor(Platform),
+ Partitioning = as.factor(Partitioning),
+ Nodes = as.factor(Nodes));
+df %>% summary;
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Parsed with column specification:
+cols(
+ X1 = col_integer(),
+ Rank = col_integer(),
+ Iteration = col_integer(),
+ Platform = col_character(),
+ Nodes = col_integer(),
+ NP = col_integer(),
+ Partitioning = col_character(),
+ EID = col_character(),
+ Infiniband = col_logical(),
+ N = col_integer(),
+ S = col_double(),
+ E = col_double(),
+ Comm = col_double(),
+ Comp = col_double()
+)
+Warning message:
+Missing column names filled in: 'X1' [1]
+ Rank Iteration Platform Nodes NP
+ Min. : 0.00 Min. : 1.0 grimoire:2400 4:2400 Min. :60
+ 1st Qu.:14.75 1st Qu.: 3.0 1st Qu.:60
+ Median :29.50 Median : 5.5 Median :60
+ Mean :29.50 Mean : 5.5 Mean :60
+ 3rd Qu.:44.25 3rd Qu.: 8.0 3rd Qu.:60
+ Max. :59.00 Max. :10.0 Max. :60
+ Partitioning EID Infiniband N
+ metis:1200 Length:2400 Mode :logical Min. : 3579
+ sfc :1200 Class :character FALSE:1200 1st Qu.:13229
+ Mode :character TRUE :1200 Median :22332
+ NA's :0 Mean :25072
+ 3rd Qu.:33946
+ Max. :69461
+ Start End Comm Comp
+ Min. :238.8 Min. :278.3 Min. : 0.5323 Min. : 0.03779
+ 1st Qu.:339.3 1st Qu.:380.7 1st Qu.:10.4865 1st Qu.:23.37386
+ Median :431.8 Median :471.4 Median :13.5453 Median :25.22149
+ Mean :431.4 Mean :470.1 Mean :13.5252 Mean :25.13902
+ 3rd Qu.:523.3 3rd Qu.:562.9 3rd Qu.:16.4835 3rd Qu.:27.24979
+ Max. :643.5 Max. :676.9 Max. :44.0683 Max. :42.73652
+ Duration
+ Min. :28.50
+ 1st Qu.:36.52
+ Median :38.28
+ Mean :38.66
+ 3rd Qu.:41.78
+ Max. :44.13
+#+end_example
+
+#+begin_src R :results output graphics :file img/exp_8-v4_total.png :exports both :width 500 :height 400 :session
+library(tidyr);
+library(ggplot2);
+df %>%
+ select(-Start, -End, -N, -Duration) %>%
+ mutate(Total = Comm+Comp) %>%
+ gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID, -Infiniband) %>%
+ filter(Variable == "Total", Iteration != 10, Rank != 0, NP == 60) %>%
+ ggplot(aes(x=as.factor(Iteration), y=Value, color=Infiniband)) +
+ theme_bw(base_size=12) +
+ geom_point(alpha=.3) +
+ ylim(0,NA) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
+ theme(legend.position="top") +
+ facet_grid(NP~Partitioning)
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_8-v4_total.png]]
+
+Looks like we have proof that Infiniband can be desactivated.
+** 6-node grimoire infiniband check :EXP9:ATTACH:
+:PROPERTIES:
+:Attachments: exp_9-v1_grimoire_6.csv.gz
+:ID: e71e949a-1221-4b20-90b5-f8b9c6585810
+:END:
+
+#+begin_src R :results output :session :exports both
+library(readr);
+library(dplyr);
+df <- read_csv("data/e7/1e949a-1221-4b20-90b5-f8b9c6585810/exp_9-v1_grimoire_6.csv.gz") %>%
+ select(-X1) %>%
+ rename(Start = S, End = E) %>%
+ mutate(Duration = End - Start) %>%
+ mutate(Platform = as.factor(Platform),
+ Partitioning = as.factor(Partitioning),
+ Nodes = as.factor(Nodes));
+df %>% summary;
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Parsed with column specification:
+cols(
+ X1 = col_integer(),
+ Rank = col_integer(),
+ Iteration = col_integer(),
+ Platform = col_character(),
+ Nodes = col_integer(),
+ NP = col_integer(),
+ Partitioning = col_character(),
+ EID = col_character(),
+ Infiniband = col_logical(),
+ N = col_integer(),
+ S = col_double(),
+ E = col_double(),
+ Comm = col_double(),
+ Comp = col_double()
+)
+Warning message:
+Missing column names filled in: 'X1' [1]
+ Rank Iteration Platform Nodes NP
+ Min. : 0.00 Min. : 1.0 grimoire:3840 6:3840 Min. :96
+ 1st Qu.:23.75 1st Qu.: 3.0 1st Qu.:96
+ Median :47.50 Median : 5.5 Median :96
+ Mean :47.50 Mean : 5.5 Mean :96
+ 3rd Qu.:71.25 3rd Qu.: 8.0 3rd Qu.:96
+ Max. :95.00 Max. :10.0 Max. :96
+ Partitioning EID Infiniband N
+ metis:1920 Length:3840 Mode :logical Min. : 3579
+ sfc :1920 Class :character FALSE:1920 1st Qu.:16320
+ Mode :character TRUE :1920 Median :23759
+ NA's :0 Mean :27008
+ 3rd Qu.:36328
+ Max. :81957
+ Start End Comm Comp
+ Min. :123.4 Min. :150.7 Min. : 0.3444 Min. : 0.01754
+ 1st Qu.:190.8 1st Qu.:217.7 1st Qu.: 8.4054 1st Qu.:14.47885
+ Median :251.6 Median :277.3 Median : 9.7694 Median :15.58878
+ Mean :251.8 Mean :277.3 Mean : 9.6905 Mean :15.79769
+ 3rd Qu.:312.4 3rd Qu.:337.4 3rd Qu.:11.4077 3rd Qu.:17.16624
+ Max. :384.8 Max. :405.8 Max. :27.9786 Max. :27.03608
+ Duration
+ Min. :19.22
+ 1st Qu.:24.94
+ Median :26.06
+ Mean :25.49
+ 3rd Qu.:26.76
+ Max. :28.22
+#+end_example
+
+#+begin_src R :results output graphics :file img/exp_9-v1_total.png :exports both :width 500 :height 400 :session
+library(tidyr);
+library(ggplot2);
+df %>%
+ select(-Start, -End, -N, -Duration) %>%
+ mutate(Total = Comm+Comp) %>%
+ gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID, -Infiniband) %>%
+ filter(Variable == "Total", Iteration != 10, Rank != 0) %>%
+ ggplot(aes(x=as.factor(Iteration), y=Value, color=Infiniband)) +
+ theme_bw(base_size=12) +
+ geom_point(alpha=.3) +
+ ylim(0,NA) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
+ theme(legend.position="top") +
+ facet_grid(NP~Partitioning)
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_9-v1_total.png]]
+** 8-node grimoire infiniband-only after Alya modifications :EXP10:
+*** Read :ATTACH:
+:PROPERTIES:
+:Attachments: exp_10-v2_grimoire_8.csv.gz
+:ID: 15c9204a-2bde-4ee3-b166-242729fbc30d
+:END:
+
+#+begin_src R :results output :session :exports both
+library(readr);
+library(dplyr);
+df <- read_csv("data/15/c9204a-2bde-4ee3-b166-242729fbc30d/exp_10-v2_grimoire_8.csv.gz") %>%
+ select(-X1) %>%
+ rename(Start = S, End = E) %>%
+ mutate(Duration = End - Start) %>%
+ mutate(Platform = as.factor(Platform),
+ Partitioning = as.factor(Partitioning),
+ Nodes = as.factor(Nodes));
+df %>% summary;
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Parsed with column specification:
+cols(
+ X1 = col_integer(),
+ Rank = col_integer(),
+ Iteration = col_integer(),
+ Platform = col_character(),
+ Nodes = col_integer(),
+ NP = col_integer(),
+ Partitioning = col_character(),
+ EID = col_character(),
+ Infiniband = col_logical(),
+ N = col_integer(),
+ S = col_double(),
+ E = col_double(),
+ Comm = col_double(),
+ Comp = col_double()
+)
+Warning message:
+Missing column names filled in: 'X1' [1]
+ Rank Iteration Platform Nodes NP
+ Min. : 0.00 Min. : 1.0 grimoire:2560 8:2560 Min. :128
+ 1st Qu.: 31.75 1st Qu.: 3.0 1st Qu.:128
+ Median : 63.50 Median : 5.5 Median :128
+ Mean : 63.50 Mean : 5.5 Mean :128
+ 3rd Qu.: 95.25 3rd Qu.: 8.0 3rd Qu.:128
+ Max. :127.00 Max. :10.0 Max. :128
+ Partitioning EID Infiniband N Start
+ metis:1280 Length:2560 Mode:logical Min. : 3579 Min. :121.9
+ sfc :1280 Class :character TRUE:2560 1st Qu.:17937 1st Qu.:167.1
+ Mode :character NA's:0 Median :25074 Median :210.0
+ Mean :28662 Mean :210.3
+ 3rd Qu.:37422 3rd Qu.:254.4
+ Max. :91329 Max. :313.9
+ End Comm Comp Duration
+ Min. :139.5 Min. : 0.4235 Min. : 0.01789 Min. :13.30
+ 1st Qu.:186.4 1st Qu.: 3.8991 1st Qu.:10.76531 1st Qu.:16.52
+ Median :228.6 Median : 6.6020 Median :11.80869 Median :17.29
+ Mean :228.5 Mean : 6.3043 Mean :11.90795 Mean :18.21
+ 3rd Qu.:272.9 3rd Qu.: 8.5957 3rd Qu.:13.19428 3rd Qu.:20.24
+ Max. :330.2 Max. :21.5603 Max. :20.72990 Max. :21.59
+#+end_example
+*** Plot Comp
+#+begin_src R :results output graphics :file img/exp_10-v2_comp.png :exports both :width 500 :height 400 :session
+library(tidyr);
+library(ggplot2);
+df %>%
+ select(-Start, -End, -N, -Duration) %>%
+ mutate(Total = Comm+Comp) %>%
+ gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID, -Infiniband) %>%
+ filter(Variable == "Comp", Iteration != 10, Rank != 0) %>%
+ ggplot(aes(x=as.factor(Iteration), y=Value, color=Infiniband)) +
+ theme_bw(base_size=12) +
+ geom_point(alpha=.3) +
+ ylim(0,NA) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
+ theme(legend.position="top") +
+ facet_grid(NP~Partitioning)
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_10-v2_comp.png]]
+*** Plot mean comp (per rank, all iterations)
+#+begin_src R :results output graphics :file img/exp_10-v2_comp_mean.png :exports both :width 600 :height 400 :session
+library(tidyr);
+library(ggplot2);
+df %>%
+ select(-Start, -End, -N, -Duration) %>%
+ mutate(Total = Comm+Comp) %>%
+ gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID, -Infiniband) %>%
+ filter(Variable == "Comp", Iteration != 10, Rank != 0) %>%
+# group_by(Rank, Platform, Nodes, NP, Partitioning, EID, Infiniband) %>%
+# summarize(MeanComp = mean(Value), SE=3*sd(Value)/sqrt(n()), N=n()) %>%
+ ggplot(aes(x=Rank, y=Value, color=Partitioning)) +
+ theme_bw(base_size=12) +
+ geom_point(alpha=.3) +
+ ylim(0,NA) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
+ theme(legend.position="top") +
+ facet_grid(NP~Partitioning)
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_10-v2_comp_mean.png]]
+*** Partition analysis
+**** Process log files
+#+begin_src shell :results output
+EDIR=exp_10-v2_grimoire_8
+for file in $(find $EDIR | grep results | grep log$); do
+ OUTPUT=$(dirname $file)/$(basename $file .log).csv
+ cat $file | tr -s " " | sed "s/^ //" | tr ' ' ',' | cut -d"," -f1,3,5,7,9,11,13 | uniq > $OUTPUT
+ cat $file | tr -s " " | sed "s/^ //" | tr ' ' ',' | cut -d"," -f2,4,6,8,10,12,14 >> $OUTPUT
+ head $OUTPUT
+ tail $OUTPUT
+ echo
+done
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Rank,NPOIN,NELEM,NELEW,NBOUN,NNEIG,NBBOU
+1,14976,72536,72796,2384,17,14976
+2,14641,72392,72575,448,12,14641
+3,22264,72647,167221,4446,16,22264
+4,28129,72464,251684,7187,11,28129
+5,23239,72488,178813,4341,25,23239
+6,26093,72525,209615,5640,14,26093
+7,22996,72391,183401,4422,10,22996
+8,21190,72393,160643,3505,7,21190
+9,25178,72612,201462,5195,10,25178
+118,24285,72483,188018,6181,14,24285
+119,26512,72604,229879,6447,15,26512
+120,24865,72374,200264,5134,10,24865
+121,22912,72446,173477,4266,15,22912
+122,14466,72460,73072,776,11,14466
+123,14046,72589,72589,944,8,14046
+124,15427,72509,73961,2155,15,15427
+125,21776,72610,153930,4907,13,21776
+126,21662,72595,161294,4243,12,21662
+127,18484,72603,111590,4500,17,18484
+
+Rank,NPOIN,NELEM,NELEW,NBOUN,NNEIG,NBBOU
+1,22405,85371,172276,3477,4,22405
+2,22923,86049,179769,3742,5,22923
+3,24316,99541,175416,3029,5,24316
+4,22907,90905,170125,3170,4,22907
+5,22636,86964,173194,3448,5,22636
+6,22774,88416,171711,3333,6,22774
+7,23564,91625,177865,3452,7,23564
+8,22211,83394,173364,3612,5,22211
+9,23427,89980,179685,3588,4,23427
+118,19344,51553,177808,5044,6,19344
+119,19050,54006,174756,4829,3,19050
+120,18742,53558,169698,4652,6,18742
+121,19158,51974,179487,5103,4,19158
+122,19820,53397,179497,5028,8,19820
+123,19357,55432,172417,4679,6,19357
+124,18855,51717,176727,5006,3,18855
+125,19473,52093,179803,5107,6,19473
+126,18948,49074,179769,5225,4,18948
+127,19037,50759,172104,4863,6,19037
+
+#+end_example
+**** Load them in R
+#+begin_src R :results output :session :exports both
+library(readr);
+library(dplyr);
+library(tidyr);
+
+read_npoin <- function(filename)
+{
+ meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[2], "_"));
+ meta <- gsub(".dir", "", meta);
+ read_csv(filename) %>%
+ gather(Variable, Value, -Rank) %>%
+ mutate(EID = meta[1],
+ Platform = meta[2],
+ Nodes = meta[3],
+ NP = meta[4],
+ Partitioning = meta[5],
+ Infiniband = as.logical(meta[6]));
+
+}
+files <- list.files("exp_10-v2_grimoire_8", pattern="results_NPOIN_NELEM_NELEW_NBOUN.csv", recursive=TRUE, full.names=TRUE);
+files;
+df <- do.call("rbind", lapply(files, function(x) { read_npoin(x) }))
+df %>% filter(Rank == 111, Variable == "NELEW")
+#+end_src
+
+#+RESULTS:
+#+begin_example
+[1] "exp_10-v2_grimoire_8/10-v2_grimoire_8_128_metis_true.dir/results_NPOIN_NELEM_NELEW_NBOUN.csv"
+[2] "exp_10-v2_grimoire_8/10-v2_grimoire_8_128_sfc_true.dir/results_NPOIN_NELEM_NELEW_NBOUN.csv"
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ NPOIN = col_integer(),
+ NELEM = col_integer(),
+ NELEW = col_integer(),
+ NBOUN = col_integer(),
+ NNEIG = col_integer(),
+ NBBOU = col_integer()
+)
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ NPOIN = col_integer(),
+ NELEM = col_integer(),
+ NELEW = col_integer(),
+ NBOUN = col_integer(),
+ NNEIG = col_integer(),
+ NBBOU = col_integer()
+)
+# A tibble: 2 × 9
+ Rank Variable Value EID Platform Nodes NP Partitioning Infiniband
+
+1 111 NELEW 172833 10-v2 grimoire 8 128 metis TRUE
+2 111 NELEW 258786 10-v2 grimoire 8 128 sfc TRUE
+#+end_example
+
+#+begin_src R :results output graphics :file img/exp_10-v2_partition.png :exports both :width 600 :height 800 :session
+df %>%
+ ggplot(aes(x=Rank, y=Value, color=Partitioning)) +
+ theme_bw(base_size=12) +
+ geom_point() +
+ ylim(0,NA) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
+ theme(legend.position="top") +
+ facet_grid(Variable~Partitioning, scales="free_y")
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_10-v2_partition.png]]
+** 8-node grimoire infiniband-only before Alya modifs :EXP11:
+*** Read :ATTACH:
+:PROPERTIES:
+:Attachments: exp_11-v1_grimoire_8.csv.gz
+:ID: 11326040-b8e2-41f3-8479-c378f70864d9
+:END:
+
+#+begin_src R :results output :session :exports both
+library(readr);
+library(dplyr);
+df <- read_csv("data/11/326040-b8e2-41f3-8479-c378f70864d9/exp_11-v1_grimoire_8.csv.gz") %>%
+ select(-X1) %>%
+ rename(Start = S, End = E) %>%
+ mutate(Duration = End - Start) %>%
+ mutate(Platform = as.factor(Platform),
+ Partitioning = as.factor(Partitioning),
+ Nodes = as.factor(Nodes));
+df %>% summary;
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Parsed with column specification:
+cols(
+ X1 = col_integer(),
+ Rank = col_integer(),
+ Iteration = col_integer(),
+ Platform = col_character(),
+ Nodes = col_integer(),
+ NP = col_integer(),
+ Partitioning = col_character(),
+ EID = col_character(),
+ Infiniband = col_logical(),
+ N = col_integer(),
+ S = col_double(),
+ E = col_double(),
+ Comm = col_double(),
+ Comp = col_double()
+)
+Warning message:
+Missing column names filled in: 'X1' [1]
+ Rank Iteration Platform Nodes NP
+ Min. : 0.00 Min. : 1.0 grimoire:2560 8:2560 Min. :128
+ 1st Qu.: 31.75 1st Qu.: 3.0 1st Qu.:128
+ Median : 63.50 Median : 5.5 Median :128
+ Mean : 63.50 Mean : 5.5 Mean :128
+ 3rd Qu.: 95.25 3rd Qu.: 8.0 3rd Qu.:128
+ Max. :127.00 Max. :10.0 Max. :128
+ Partitioning EID Infiniband N Start
+ metis:1280 Length:2560 Mode:logical Min. : 3579 Min. :121.7
+ sfc :1280 Class :character TRUE:2560 1st Qu.:17937 1st Qu.:168.6
+ Mode :character NA's:0 Median :24815 Median :217.9
+ Mean :28185 Mean :217.4
+ 3rd Qu.:37093 3rd Qu.:265.8
+ Max. :81957 Max. :313.0
+ End Comm Comp Duration
+ Min. :142.9 Min. : 0.3322 Min. : 0.02086 Min. :15.47
+ 1st Qu.:189.2 1st Qu.: 7.0911 1st Qu.:10.84112 1st Qu.:19.75
+ Median :237.8 Median : 8.1973 Median :11.59094 Median :20.00
+ Mean :237.1 Mean : 7.9373 Mean :11.81608 Mean :19.75
+ 3rd Qu.:285.6 3rd Qu.: 9.0958 3rd Qu.:12.73405 3rd Qu.:20.39
+ Max. :328.8 Max. :21.1947 Max. :20.53365 Max. :21.22
+#+end_example
+*** Plot Comp
+#+begin_src R :results output graphics :file img/exp_11-v1_comp.png :exports both :width 500 :height 400 :session
+library(tidyr);
+library(ggplot2);
+df %>%
+ select(-Start, -End, -N, -Duration) %>%
+ mutate(Total = Comm+Comp) %>%
+ gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID, -Infiniband) %>%
+ filter(Variable == "Comp", Iteration != 10, Rank != 0) %>%
+ ggplot(aes(x=as.factor(Iteration), y=Value, color=Infiniband)) +
+ theme_bw(base_size=12) +
+ geom_point(alpha=.3) +
+ ylim(0,NA) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
+ theme(legend.position="top") +
+ facet_grid(NP~Partitioning)
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_11-v1_comp.png]]
+*** Partition analysis
+**** Process log files
+#+begin_src shell :results output
+EDIR=exp_11-v1_grimoire_8
+for file in $(find $EDIR | grep results | grep log$); do
+ OUTPUT=$(dirname $file)/$(basename $file .log).csv
+ cat $file | tr -s " " | sed "s/^ //" | tr ' ' ',' | cut -d"," -f1,3,5,7,9,11,13 | uniq > $OUTPUT
+ cat $file | tr -s " " | sed "s/^ //" | tr ' ' ',' | cut -d"," -f2,4,6,8,10,12,14 >> $OUTPUT
+ head $OUTPUT
+ tail $OUTPUT
+ echo
+done
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Rank,NPOIN,NELEM,NELEW,NBOUN,NNEIG,NBBOU
+1,32182,174168,174611,2973,16,32182
+2,20919,53529,174758,5381,15,20919
+3,19727,49972,175172,5020,11,19727
+4,21918,69049,174054,4277,23,21918
+5,21683,64345,174280,4439,14,21683
+6,22205,64122,175002,4582,18,22205
+7,21361,63432,174587,4436,11,21361
+8,22162,75699,174459,3914,8,22162
+9,21389,66339,174799,4332,8,21389
+118,19736,47749,174249,5084,13,19736
+119,27122,103080,174535,7852,12,27122
+120,21785,59025,175145,5304,13,21785
+121,20722,56163,175018,4869,12,20722
+122,21269,59300,174615,4657,10,21269
+123,22403,70519,174270,4135,13,22403
+124,30100,145205,174682,2874,9,30100
+125,31603,142203,174120,5271,15,31603
+126,22843,73894,175386,4418,11,22843
+127,26719,97390,174439,6359,13,26719
+
+Rank,NPOIN,NELEM,NELEW,NBOUN,NNEIG,NBBOU
+1,22405,85371,172276,3477,4,22405
+2,22923,86049,179769,3742,5,22923
+3,24316,99541,175416,3029,5,24316
+4,22907,90905,170125,3170,4,22907
+5,22636,86964,173194,3448,5,22636
+6,22774,88416,171711,3333,6,22774
+7,23564,91625,177865,3452,7,23564
+8,22211,83394,173364,3612,5,22211
+9,23427,89980,179685,3588,4,23427
+118,19344,51553,177808,5044,6,19344
+119,19050,54006,174756,4829,3,19050
+120,18742,53558,169698,4652,6,18742
+121,19158,51974,179487,5103,4,19158
+122,19820,53397,179497,5028,8,19820
+123,19357,55432,172417,4679,6,19357
+124,18855,51717,176727,5006,3,18855
+125,19473,52093,179803,5107,6,19473
+126,18948,49074,179769,5225,4,18948
+127,19037,50759,172104,4863,6,19037
+
+#+end_example
+**** Load them in R
+#+begin_src R :results output :session :exports both
+library(readr);
+library(dplyr);
+library(tidyr);
+
+read_npoin <- function(filename)
+{
+ meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[2], "_"));
+ meta <- gsub(".dir", "", meta);
+ read_csv(filename) %>%
+ gather(Variable, Value, -Rank) %>%
+ mutate(EID = meta[1],
+ Platform = meta[2],
+ Nodes = meta[3],
+ NP = meta[4],
+ Partitioning = meta[5],
+ Infiniband = as.logical(meta[6]));
+
+}
+files <- list.files("exp_11-v1_grimoire_8", pattern="results_NPOIN_NELEM_NELEW_NBOUN.csv", recursive=TRUE, full.names=TRUE);
+files;
+df <- do.call("rbind", lapply(files, function(x) { read_npoin(x) }))
+df %>% filter(Rank == 111, Variable == "NELEW")
+#+end_src
+
+#+RESULTS:
+#+begin_example
+[1] "exp_11-v1_grimoire_8/11-v1_grimoire_8_128_metis_true.dir/results_NPOIN_NELEM_NELEW_NBOUN.csv"
+[2] "exp_11-v1_grimoire_8/11-v1_grimoire_8_128_sfc_true.dir/results_NPOIN_NELEM_NELEW_NBOUN.csv"
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ NPOIN = col_integer(),
+ NELEM = col_integer(),
+ NELEW = col_integer(),
+ NBOUN = col_integer(),
+ NNEIG = col_integer(),
+ NBBOU = col_integer()
+)
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ NPOIN = col_integer(),
+ NELEM = col_integer(),
+ NELEW = col_integer(),
+ NBOUN = col_integer(),
+ NNEIG = col_integer(),
+ NBBOU = col_integer()
+)
+# A tibble: 2 × 9
+ Rank Variable Value EID Platform Nodes NP Partitioning Infiniband
+
+1 111 NELEW 172833 11-v1 grimoire 8 128 metis TRUE
+2 111 NELEW 173994 11-v1 grimoire 8 128 sfc TRUE
+#+end_example
+**** Plot
+#+begin_src R :results output graphics :file img/exp_11-v1_partition.png :exports both :width 600 :height 800 :session
+df %>%
+ ggplot(aes(x=Rank, y=Value, color=Partitioning)) +
+ theme_bw(base_size=12) +
+ geom_point() +
+ ylim(0,NA) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
+ theme(legend.position="top") +
+ facet_grid(Variable~Partitioning, scales="free_y")
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_11-v1_partition.png]]
+** 6-node grimoire tracing only application functions :EXP12:
+:PROPERTIES:
+:Attachments: scorep_12-v1_grimoire_6_48_metis_true_traces.csv.gz scorep_12-v1_grimoire_6_48_sfc_true_traces.csv.gz scorep_12-v1_grimoire_6_96_metis_true_traces.csv.gz scorep_12-v1_grimoire_6_96_sfc_true_traces.csv.gz
+:ID: 58ddccd9-95c8-4841-9c07-336a0209e264
+:END:
+
+*** Which files
+#+name: exp_12_traces
+#+begin_src shell :results output org
+find data/ | grep scorep_12-v1
+#+end_src
+
+#+RESULTS: exp_12_traces
+#+BEGIN_SRC org
+data/58/ddccd9-95c8-4841-9c07-336a0209e264/scorep_12-v1_grimoire_6_96_metis_true_traces.csv.gz
+data/58/ddccd9-95c8-4841-9c07-336a0209e264/scorep_12-v1_grimoire_6_48_metis_true_traces.csv.gz
+data/58/ddccd9-95c8-4841-9c07-336a0209e264/scorep_12-v1_grimoire_6_96_sfc_true_traces.csv.gz
+data/58/ddccd9-95c8-4841-9c07-336a0209e264/scorep_12-v1_grimoire_6_48_sfc_true_traces.csv.gz
+#+END_SRC
+*** Read in R :ATTACH:
+
+- =nsi_matrix=
+- =solver=
+- =nsi_inisol= (hardly seen)
+
+#+begin_src R :results output :session :exports both :var files=exp_12_traces
+library(readr);
+library(dplyr);
+alya_scorep_trace_read <- function(filename)
+{
+ meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[4], "_"));
+ read_csv(filename,
+ col_names=c("Rank", "Start", "End", "Value"),
+ progress=FALSE) %>%
+ # Transform Value to factor
+ mutate(Value = as.factor(Value)) %>%
+ # Detect begin and end of iterations
+ mutate(Iteration = case_when(
+ (.$Value == "timste") ~ 1,
+ (.$Value == "endste") ~ -1,
+ TRUE ~ 0)) %>%
+ group_by(Rank) %>%
+ # Create a logical to detect observations within iterations
+ mutate(Iteration = as.logical(cumsum(Iteration))) %>%
+ # Get only observations that belongs to some iteration
+ filter(Iteration == TRUE) %>%
+ ungroup() %>%
+ # Create the iteration by cumsum
+ mutate(Iteration = case_when(
+ (.$Value == "timste") ~ 1,
+ TRUE ~ 0)) %>%
+ group_by(Rank) %>%
+ mutate(Iteration = cumsum(Iteration)) %>%
+ mutate(EID = meta[2],
+ Platform = meta[3],
+ Nodes = meta[4],
+ NP = meta[5],
+ Partitioning = meta[6]);
+}
+
+files <- strsplit(files, "\n")[[1]]
+df <- do.call("rbind", lapply(files, function(x) { alya_scorep_trace_read(x) }));
+
+#
+# Keep only states that are interesting
+#
+
+df <- df %>% filter(Value %in% c("nsi_matrix", "solver", "nsi_inisol"));
+#+end_src
+
+#+RESULTS:
+#+begin_example
+
+Attaching package: ‘dplyr’
+
+The following objects are masked from ‘package:stats’:
+
+ filter, lag
+
+The following objects are masked from ‘package:base’:
+
+ intersect, setdiff, setequal, union
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ Start = col_double(),
+ End = col_double(),
+ Value = col_character()
+)
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ Start = col_double(),
+ End = col_double(),
+ Value = col_character()
+)
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ Start = col_double(),
+ End = col_double(),
+ Value = col_character()
+)
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ Start = col_double(),
+ End = col_double(),
+ Value = col_character()
+)
+#+end_example
+
+#+begin_src R :results output :session :exports both
+df %>%
+ group_by(Rank, Iteration, EID, Platform, Nodes, NP, Partitioning, Value) %>%
+ summarize(NewComp = sum(End - Start));
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Source: local data frame [86,400 x 9]
+Groups: Rank, Iteration, EID, Platform, Nodes, NP, Partitioning [?]
+
+ Rank Iteration EID Platform Nodes NP Partitioning Value NewComp
+
+1 0 1 12-v1 grimoire 6 48 metis nsi_inisol 0.000013
+2 0 1 12-v1 grimoire 6 48 metis nsi_matrix 0.000075
+3 0 1 12-v1 grimoire 6 48 metis solver 42.482770
+4 0 1 12-v1 grimoire 6 48 sfc nsi_inisol 0.000012
+5 0 1 12-v1 grimoire 6 48 sfc nsi_matrix 0.000079
+6 0 1 12-v1 grimoire 6 48 sfc solver 35.042236
+7 0 1 12-v1 grimoire 6 96 metis nsi_inisol 0.000014
+8 0 1 12-v1 grimoire 6 96 metis nsi_matrix 0.000076
+9 0 1 12-v1 grimoire 6 96 metis solver 21.981336
+10 0 1 12-v1 grimoire 6 96 sfc nsi_inisol 0.000012
+# ... with 86,390 more rows
+#+end_example
+*** Plots
+#+begin_src R :results output graphics :file img/alya_nastin_solver_3_funs.png :exports both :width 600 :height 400 :session
+library(ggplot2);
+df %>%
+ group_by(Rank, Iteration, EID, Platform, Nodes, NP, Partitioning) %>%
+ summarize(NewComp = sum(End - Start)) %>%
+ ggplot(aes(x=Iteration, y=NewComp, color=as.factor(Rank))) +
+ theme_bw(base_size=12) +
+ geom_point(alpha=.3) +
+ ylim(0,NA) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning))) +
+ theme(legend.position="none") +
+ facet_grid(NP~Partitioning);
+#+end_src
+
+#+RESULTS:
+[[file:img/alya_nastin_solver_3_funs.png]]
+
+#+begin_src R :results output graphics :file img/alya_nastin_solver_solver.png :exports both :width 600 :height 400 :session
+df %>%
+ filter(Value == "solver") %>%
+ group_by(Rank, Iteration, EID, Platform, Nodes, NP, Value, Partitioning) %>%
+ summarize(NewComp = sum(End - Start)) %>%
+ ggplot(aes(x=Iteration, y=NewComp, color=as.factor(Rank))) +
+ theme_bw(base_size=12) +
+ geom_point(alpha=.3) +
+ ylim(0,NA) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning))) +
+ theme(legend.position="none") +
+ facet_grid(NP~Partitioning);
+#+end_src
+
+#+RESULTS:
+[[file:img/alya_nastin_solver_solver.png]]
+
+#+begin_src R :results output graphics :file img/alya_nastin_solver_nsi_matrix.png :exports both :width 600 :height 400 :session
+df %>%
+ filter(Value == "nsi_matrix") %>%
+ group_by(Rank, Iteration, EID, Platform, Nodes, NP, Value, Partitioning) %>%
+ summarize(NewComp = sum(End - Start)) %>%
+ ggplot(aes(x=Iteration, y=NewComp, color=as.factor(Rank))) +
+ theme_bw(base_size=12) +
+ geom_point(alpha=.3) +
+ ylim(0,NA) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning))) +
+ theme(legend.position="none") +
+ facet_grid(NP~Partitioning);#, scales="free_y")
+#+end_src
+
+#+RESULTS:
+[[file:img/alya_nastin_solver_nsi_matrix.png]]
+
+
+#+begin_src R :results output graphics :file img/alya_nastin_solver_nsi_inisol.png :exports both :width 600 :height 400 :session
+df %>%
+ filter(Value == "nsi_inisol") %>%
+ group_by(Rank, Iteration, EID, Platform, Nodes, NP, Value, Partitioning) %>%
+ summarize(NewComp = sum(End - Start)) %>%
+ ggplot(aes(x=Iteration, y=NewComp, color=as.factor(Rank))) +
+ theme_bw(base_size=12) +
+ geom_point(alpha=.3) +
+ ylim(0,NA) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning))) +
+ theme(legend.position="none") +
+ facet_grid(NP~Partitioning);#, scales="free_y")
+#+end_src
+
+#+RESULTS:
+[[file:img/alya_nastin_solver_nsi_inisol.png]]
+** 12-node grisou (computing instrumentations, Alya modifs yes/no) :EXP13:ATTACH:
+:PROPERTIES:
+:ID: 0717e0e2-99b8-44c7-acfc-4d4ad685a2c5
+:Attachments: exp_13-v1_grisou_12
+:END:
+
+*** Which files
+#+name: exp_13_traces
+#+begin_src shell :results output
+cd data/07/17e0e2-99b8-44c7-acfc-4d4ad685a2c5/
+find exp_13-v1_grisou_12/| grep csv.gz
+#+end_src
+
+#+RESULTS: exp_13_traces
+: exp_13-v1_grisou_12/13-v1_grisou_12_96_sfc_true_Alya.x.modif.dir/scorep_13-v1_grisou_12_96_sfc_true_Alya.x.modif/traces.csv.gz
+: exp_13-v1_grisou_12/13-v1_grisou_12_96_metis_true_Alya.x.modif.dir/scorep_13-v1_grisou_12_96_metis_true_Alya.x.modif/traces.csv.gz
+: exp_13-v1_grisou_12/13-v1_grisou_12_192_metis_true_Alya.x.dir/scorep_13-v1_grisou_12_192_metis_true_Alya.x/traces.csv.gz
+: exp_13-v1_grisou_12/13-v1_grisou_12_192_sfc_true_Alya.x.dir/scorep_13-v1_grisou_12_192_sfc_true_Alya.x/traces.csv.gz
+: exp_13-v1_grisou_12/13-v1_grisou_12_192_metis_true_Alya.x.modif.dir/scorep_13-v1_grisou_12_192_metis_true_Alya.x.modif/traces.csv.gz
+: exp_13-v1_grisou_12/13-v1_grisou_12_96_metis_true_Alya.x.dir/scorep_13-v1_grisou_12_96_metis_true_Alya.x/traces.csv.gz
+: exp_13-v1_grisou_12/13-v1_grisou_12_96_sfc_true_Alya.x.dir/scorep_13-v1_grisou_12_96_sfc_true_Alya.x/traces.csv.gz
+: exp_13-v1_grisou_12/13-v1_grisou_12_192_sfc_true_Alya.x.modif.dir/scorep_13-v1_grisou_12_192_sfc_true_Alya.x.modif/traces.csv.gz
+
+#+begin_src R :results output :session :exports both :var files=exp_13_traces
+files <- strsplit(files, "\n")[[1]]
+filename <- files[[1]]
+filename;
+meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[3], "_"));
+EID = meta[2]
+Platform = meta[3]
+Nodes = meta[4]
+NP = meta[5]
+Partitioning = meta[6]
+Infiniband = meta[7]
+Alya = meta[8]
+EID;
+Platform;
+Nodes;
+NP;
+Partitioning;
+Infiniband;
+Alya;
+#+end_src
+
+#+RESULTS:
+: [1] "exp_13-v1_grisou_12/13-v1_grisou_12_96_sfc_true_Alya.x.modif.dir/scorep_13-v1_grisou_12_96_sfc_true_Alya.x.modif/traces.csv.gz"
+: [1] "13-v1"
+: [1] "grisou"
+: [1] "12"
+: [1] "96"
+: [1] "sfc"
+: [1] "true"
+: [1] "Alya.x.modif"
+*** Read in R to =dft13=
+#+begin_src R :results output :session :exports both :var files=exp_13_traces
+library(readr);
+library(dplyr);
+alya_scorep_trace_read <- function(filename, basedir = ".")
+{
+ meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[3], "_"));
+ fullpath <- paste0(basedir, "/", filename);
+ read_csv(fullpath,
+ col_names=c("Rank", "Start", "End", "Value"),
+ progress=FALSE) %>%
+ # Transform Value to factor
+ mutate(Value = as.factor(Value)) %>%
+ # Detect begin and end of iterations
+ mutate(Iteration = case_when(
+ (.$Value == "timste") ~ 1,
+ (.$Value == "endste") ~ -1,
+ TRUE ~ 0)) %>%
+ group_by(Rank) %>%
+ # Create a logical to detect observations within iterations
+ mutate(Iteration = as.logical(cumsum(Iteration))) %>%
+ # Get only observations that belongs to some iteration
+ filter(Iteration == TRUE) %>%
+ ungroup() %>%
+ # Create the iteration by cumsum
+ mutate(Iteration = case_when(
+ (.$Value == "timste") ~ 1,
+ TRUE ~ 0)) %>%
+ group_by(Rank) %>%
+ mutate(Iteration = cumsum(Iteration)) %>%
+ mutate(EID = as.factor(meta[2]),
+ Platform = as.factor(meta[3]),
+ Nodes = as.factor(meta[4]),
+ NP = as.factor(meta[5]),
+ Partitioning = as.factor(meta[6]),
+ Infiniband = as.logical(meta[7]),
+ Alya = as.factor(meta[8]));
+}
+
+files <- strsplit(files, "\n")[[1]]
+dft13 <- do.call("rbind", lapply(files, function(x) { alya_scorep_trace_read(x, basedir="data/07/17e0e2-99b8-44c7-acfc-4d4ad685a2c5") }));
+
+#
+# Keep only states that are interesting
+#
+
+dft13 <- dft13 %>% filter(Value %in% c("nsi_matrix", "solver", "nsi_inisol"));
+#+end_src
+
+#+RESULTS:
+#+begin_example
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ Start = col_double(),
+ End = col_double(),
+ Value = col_character()
+)
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ Start = col_double(),
+ End = col_double(),
+ Value = col_character()
+)
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ Start = col_double(),
+ End = col_double(),
+ Value = col_character()
+)
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ Start = col_double(),
+ End = col_double(),
+ Value = col_character()
+)
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ Start = col_double(),
+ End = col_double(),
+ Value = col_character()
+)
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ Start = col_double(),
+ End = col_double(),
+ Value = col_character()
+)
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ Start = col_double(),
+ End = col_double(),
+ Value = col_character()
+)
+Parsed with column specification:
+cols(
+ Rank = col_integer(),
+ Start = col_double(),
+ End = col_double(),
+ Value = col_character()
+)
+Warning messages:
+1: In bind_rows_(x, .id) : Unequal factor levels: coercing to character
+2: In bind_rows_(x, .id) : Unequal factor levels: coercing to character
+3: In bind_rows_(x, .id) : Unequal factor levels: coercing to character
+#+end_example
+
+#+begin_src R :results output :session :exports both
+dft13 %>% .$NP %>% unique
+dft13 %>% .$Alya %>% unique
+dft13;
+dft13 %>% mutate(Partitioning=as.factor(Partitioning), NP=as.factor(NP), Alya=as.factor(Alya)) %>% summary;
+#+end_src
+
+#+RESULTS:
+#+begin_example
+[1] "96" "192"
+[1] "Alya.x.modif" "Alya.x"
+Source: local data frame [2,695,680 x 12]
+Groups: Rank [192]
+
+ Rank Start End Value Iteration EID Platform Nodes NP
+
+1 95 0.333969 0.333970 nsi_inisol 1 13-v1 grisou 12 96
+2 95 0.335719 1.417964 nsi_matrix 1 13-v1 grisou 12 96
+3 95 1.420657 1.978257 solver 1 13-v1 grisou 12 96
+4 95 1.982556 2.796641 solver 1 13-v1 grisou 12 96
+5 95 2.799346 2.991894 solver 1 13-v1 grisou 12 96
+6 95 3.010278 3.864155 solver 1 13-v1 grisou 12 96
+7 95 4.066358 4.066361 nsi_inisol 1 13-v1 grisou 12 96
+8 95 4.066362 4.119009 solver 1 13-v1 grisou 12 96
+9 95 4.888364 5.940038 nsi_matrix 1 13-v1 grisou 12 96
+10 95 5.942733 6.901603 solver 1 13-v1 grisou 12 96
+# ... with 2,695,670 more rows, and 3 more variables: Partitioning ,
+# Infiniband , Alya
+ Rank Start End Value
+ Min. : 0.00 Min. : 0.1715 Min. : 0.1715 solver :1843200
+ 1st Qu.: 35.75 1st Qu.: 270.0145 1st Qu.: 270.2660 nsi_inisol: 483840
+ Median : 71.50 Median : 529.9820 Median : 530.3756 nsi_matrix: 368640
+ Mean : 79.50 Mean : 572.4097 Mean : 572.7664 doiter : 0
+ 3rd Qu.:119.25 3rd Qu.: 785.8916 3rd Qu.: 786.1151 endste : 0
+ Max. :191.00 Max. :1721.8589 Max. :1721.8598 nastin : 0
+ (Other) : 0
+ Iteration EID Platform Nodes NP
+ Min. : 1.00 13-v1:2695680 grisou:2695680 12:2695680 192:1797120
+ 1st Qu.: 21.00 96 : 898560
+ Median : 47.00
+ Mean : 47.81
+ 3rd Qu.: 74.00
+ Max. :100.00
+
+ Partitioning Infiniband Alya
+ metis:1347840 Mode:logical Alya.x :1347840
+ sfc :1347840 TRUE:2695680 Alya.x.modif:1347840
+ NA's:0
+#+end_example
+*** Plot only "computing time"
+**** nastin solver 3 functions =nsi_matrix=, =solver=, =nsi_inisol=
+#+begin_src R :results output graphics :file img/exp_13_alya_nastin_solver_3_funs.png :exports both :width 1200 :height 400 :session
+library(ggplot2);
+dft13 %>%
+ filter(Rank != 0) %>%
+ filter(NP == 192, Partitioning == "sfc") %>%
+ group_by(Rank, Iteration, EID, Platform, Nodes, NP, Partitioning, Infiniband, Alya) %>%
+ summarize(NewComp = sum(End - Start)) %>%
+ ggplot(aes(x=Iteration, y=NewComp, color=as.factor(Rank))) +
+ theme_bw(base_size=12) +
+ geom_point() +
+ ylim(0,NA) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning))) +
+ theme(legend.position="none") +
+ facet_grid(Partitioning~Alya);
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_13_alya_nastin_solver_3_funs.png]]
+**** nastin solver only =nsi_matrix=
+#+begin_src R :results output graphics :file img/exp_13_alya_nastin_nsi_matrix.png :exports both :width 1200 :height 400 :session
+library(ggplot2);
+dft13 %>%
+ filter(Value == "nsi_matrix") %>%
+ filter(Rank != 0) %>%
+ filter(NP == 192, Partitioning == "sfc") %>%
+ group_by(Rank, Iteration, EID, Platform, Nodes, NP, Partitioning, Infiniband, Alya) %>%
+ summarize(NewComp = sum(End - Start)) %>%
+ ggplot(aes(x=Iteration, y=NewComp, color=as.factor(Rank))) +
+ theme_bw(base_size=12) +
+ geom_point() +
+ ylim(0,NA) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning))) +
+ theme(legend.position="none") +
+ facet_grid(Partitioning~Alya);
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_13_alya_nastin_nsi_matrix.png]]
+
+**** nastin solver only =solver=
+#+begin_src R :results output graphics :file img/exp_13_alya_nastin_solver.png :exports both :width 1200 :height 400 :session
+library(ggplot2);
+dft13 %>%
+ filter(Value == "solver") %>%
+ filter(Rank != 0) %>%
+ filter(NP == 192, Partitioning == "sfc") %>%
+ group_by(Rank, Iteration, EID, Platform, Nodes, NP, Partitioning, Infiniband, Alya) %>%
+ summarize(NewComp = sum(End - Start)) %>%
+ ggplot(aes(x=Iteration, y=NewComp, color=as.factor(Rank))) +
+ theme_bw(base_size=12) +
+ geom_point() +
+ ylim(0,NA) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning))) +
+ theme(legend.position="none") +
+ facet_grid(Partitioning~Alya);
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_13_alya_nastin_solver.png]]
+
+**** nastin solver only =nsi_inisol=
+#+begin_src R :results output graphics :file img/exp_13_alya_nastin_nsi_inisol.png :exports both :width 1200 :height 400 :session
+library(ggplot2);
+dft13 %>%
+ filter(Value == "nsi_inisol") %>%
+ filter(Rank != 0) %>%
+ filter(NP == 192, Partitioning == "sfc") %>%
+ group_by(Rank, Iteration, EID, Platform, Nodes, NP, Partitioning, Infiniband, Alya) %>%
+ summarize(NewComp = sum(End - Start)) %>%
+ ggplot(aes(x=Iteration, y=NewComp, color=as.factor(Rank))) +
+ theme_bw(base_size=12) +
+ geom_point() +
+ ylim(0,NA) +
+ geom_line (aes(group=as.factor(Rank):as.factor(Partitioning))) +
+ theme(legend.position="none") +
+ facet_grid(Partitioning~Alya);
+#+end_src
+
+#+RESULTS:
+[[file:img/exp_13_alya_nastin_nsi_inisol.png]]
+**** Amount of calls to these functions along iterations
+#+begin_src R :results output :session :exports both
+dft13 %>%
+ filter(Value == "nsi_matrix") %>%
+ filter(Rank != 0) %>%
+ filter(NP==192, Rank == 95, Iteration %in% c(1,100), Alya == "Alya.x") %>%
+ mutate(Duration = End - Start) %>%
+ select(-EID, -Platform, -Nodes, -Infiniband, -NP, -Partitioning) %>%
+ group_by(Iteration) %>%
+ mutate(N=n()) %>%
+ ungroup() %>%
+ arrange(Start, Iteration) %>%
+ as.data.frame()
+#+end_src
+
+#+RESULTS:
+#+begin_example
+ Rank Start End Value Iteration Alya Duration N
+1 95 0.283716 0.856840 nsi_matrix 1 Alya.x 0.573124 10
+2 95 0.378482 0.927219 nsi_matrix 1 Alya.x 0.548737 10
+3 95 3.298924 3.875023 nsi_matrix 1 Alya.x 0.576099 10
+4 95 3.320866 3.865816 nsi_matrix 1 Alya.x 0.544950 10
+5 95 6.500311 7.052999 nsi_matrix 1 Alya.x 0.552688 10
+6 95 6.595464 7.164909 nsi_matrix 1 Alya.x 0.569445 10
+7 95 9.646956 10.194573 nsi_matrix 1 Alya.x 0.547617 10
+8 95 9.782018 10.349733 nsi_matrix 1 Alya.x 0.567715 10
+9 95 12.637448 13.184027 nsi_matrix 1 Alya.x 0.546579 10
+10 95 12.929592 13.499010 nsi_matrix 1 Alya.x 0.569418 10
+11 95 902.438214 902.987738 nsi_matrix 100 Alya.x 0.549524 6
+12 95 905.091038 905.636659 nsi_matrix 100 Alya.x 0.545621 6
+13 95 907.747868 908.293808 nsi_matrix 100 Alya.x 0.545940 6
+14 95 914.429194 914.996252 nsi_matrix 100 Alya.x 0.567058 6
+15 95 917.061845 917.634066 nsi_matrix 100 Alya.x 0.572221 6
+16 95 919.813262 920.384819 nsi_matrix 100 Alya.x 0.571557 6
+#+end_example
+
+There are, for =nsi_matrix= in Rank 95:
+- 15 calls in iteration 1
+- but only 9 in iteration 100.
+
+#+begin_src R :results output :session :exports both
+dft13 %>%
+ filter(Value == "solver") %>%
+ filter(Rank != 0) %>%
+ filter(NP == 192, Partitioning == "sfc") %>%
+ filter(Rank == 3, Iteration %in% c(1,100), Alya == "Alya.x") %>%
+ mutate(Duration = End-Start) %>%
+ select(-EID, -Platform, -Nodes, -Infiniband, -NP, -Partitioning) %>%
+ group_by(Iteration) %>%
+ mutate(N=n()) %>%
+ ungroup() %>%
+ arrange(Start, Iteration) %>%
+ as.data.frame()
+#+end_src
+
+#+RESULTS:
+#+begin_example
+ Rank Start End Value Iteration Alya Duration N
+1 3 0.806091 1.397790 solver 1 Alya.x 0.591699 25
+2 3 1.400163 1.883410 solver 1 Alya.x 0.483247 25
+3 3 1.884954 1.987991 solver 1 Alya.x 0.103037 25
+4 3 1.995055 2.475562 solver 1 Alya.x 0.480507 25
+5 3 2.558144 2.693178 solver 1 Alya.x 0.135034 25
+6 3 3.821168 4.586247 solver 1 Alya.x 0.765079 25
+7 3 4.588758 5.062848 solver 1 Alya.x 0.474090 25
+8 3 5.064446 5.387481 solver 1 Alya.x 0.323035 25
+9 3 5.394513 5.799146 solver 1 Alya.x 0.404633 25
+10 3 5.881718 6.016954 solver 1 Alya.x 0.135236 25
+11 3 7.130104 7.898844 solver 1 Alya.x 0.768740 25
+12 3 7.901355 8.296631 solver 1 Alya.x 0.395276 25
+13 3 8.298181 8.584412 solver 1 Alya.x 0.286231 25
+14 3 8.591484 9.002578 solver 1 Alya.x 0.411094 25
+15 3 9.085122 9.221985 solver 1 Alya.x 0.136863 25
+16 3 10.303988 11.044705 solver 1 Alya.x 0.740717 25
+17 3 11.047239 11.510650 solver 1 Alya.x 0.463411 25
+18 3 11.512203 11.736265 solver 1 Alya.x 0.224062 25
+19 3 11.743266 12.150266 solver 1 Alya.x 0.407000 25
+20 3 12.232800 12.369987 solver 1 Alya.x 0.137187 25
+21 3 13.452176 14.151580 solver 1 Alya.x 0.699404 25
+22 3 14.154113 14.507091 solver 1 Alya.x 0.352978 25
+23 3 14.508630 14.696163 solver 1 Alya.x 0.187533 25
+24 3 14.703222 15.079027 solver 1 Alya.x 0.375805 25
+25 3 15.161596 15.297411 solver 1 Alya.x 0.135815 25
+26 3 914.954351 915.583598 solver 100 Alya.x 0.629247 15
+27 3 915.586138 915.877785 solver 100 Alya.x 0.291647 15
+28 3 915.879321 916.002794 solver 100 Alya.x 0.123473 15
+29 3 916.009884 916.294803 solver 100 Alya.x 0.284919 15
+30 3 916.381085 916.513058 solver 100 Alya.x 0.131973 15
+31 3 917.587713 918.332705 solver 100 Alya.x 0.744992 15
+32 3 918.335221 918.630677 solver 100 Alya.x 0.295456 15
+33 3 918.632221 918.755400 solver 100 Alya.x 0.123179 15
+34 3 918.762429 919.045410 solver 100 Alya.x 0.282981 15
+35 3 919.131495 919.265010 solver 100 Alya.x 0.133515 15
+36 3 920.334900 921.001416 solver 100 Alya.x 0.666516 15
+37 3 921.003959 921.290349 solver 100 Alya.x 0.286390 15
+38 3 921.291913 921.425185 solver 100 Alya.x 0.133272 15
+39 3 921.432295 921.728660 solver 100 Alya.x 0.296365 15
+40 3 921.815039 921.947326 solver 100 Alya.x 0.132287 15
+#+end_example
+
+There are, for =solver= in Rank 95:
+- 25 calls in iteration 1
+- but only 15 in iteration 100.
+
+Let's check for other ranks and kernels:
+
+#+begin_src R :results output :session :exports both
+dft13 %>%
+ filter(Value == "solver") %>%
+ filter(Rank != 0) %>%
+ filter(NP == 192, Partitioning == "sfc") %>%
+ filter(Iteration %in% c(1,100), Alya == "Alya.x") %>%
+ mutate(Duration = End-Start) %>%
+ select(-EID, -Platform, -Nodes, -Infiniband, -NP, -Partitioning) %>%
+ group_by(Rank, Value, Iteration) %>%
+ mutate(N=n()) %>%
+ ungroup() %>%
+ select(Rank, Iteration, N, Value) %>%
+ unique %>%
+ sample_n(20) %>%
+ arrange(Iteration, Value, N)
+#+end_src
+
+#+RESULTS:
+#+begin_example
+# A tibble: 20 × 4
+ Rank Iteration N Value
+
+1 163 1 25 solver
+2 36 1 25 solver
+3 129 1 25 solver
+4 66 1 25 solver
+5 89 1 25 solver
+6 185 1 25 solver
+7 97 1 25 solver
+8 152 1 25 solver
+9 114 1 25 solver
+10 159 1 25 solver
+11 94 1 25 solver
+12 57 1 25 solver
+13 134 1 25 solver
+14 46 100 15 solver
+15 101 100 15 solver
+16 159 100 15 solver
+17 147 100 15 solver
+18 153 100 15 solver
+19 124 100 15 solver
+20 174 100 15 solver
+#+end_example
+
+#+begin_src R :results output :session :exports both
+dft13 %>%
+ filter(Value == "nsi_matrix") %>%
+ filter(Rank != 0) %>%
+ filter(NP == 192, Partitioning == "sfc") %>%
+ filter(Iteration %in% c(1,100), Alya == "Alya.x") %>%
+ mutate(Duration = End-Start) %>%
+ select(-EID, -Platform, -Nodes, -Infiniband, -NP, -Partitioning) %>%
+ group_by(Rank, Value, Iteration) %>%
+ mutate(N=n()) %>%
+ ungroup() %>%
+ select(Rank, Iteration, N, Value) %>%
+ unique %>%
+ sample_n(20) %>%
+ arrange(Iteration, Value, N)
+#+end_src
+
+#+RESULTS:
+#+begin_example
+# A tibble: 20 × 4
+ Rank Iteration N Value
+