diff --git a/module2/ressources/video_examples/README.org b/module2/ressources/video_examples/README.org
deleted file mode 100755
index e82c575468ac8eeb867b16293c2626f0cac19b2a..0000000000000000000000000000000000000000
--- a/module2/ressources/video_examples/README.org
+++ /dev/null
@@ -1,62 +0,0 @@
-# -*- coding: utf-8 -*-
-# -*- mode: org -*-
-
-#+TITLE: Org document examples
-#+AUTHOR: Arnaud Legrand
-#+STARTUP: overview indent inlineimages logdrawer
-#+LANGUAGE: en
-
-In the MOOC video, I quickly demo how org-mode can be used in various
-contexts. Here are the (sometimes trimmed) corresponding
-org-files. These documents depend on many other external data files
-and are not meant to lead to reproducible documents but it will give
-you an idea of how it can be organized:
-
-1. [[file:journal.org][journal.org]]: an excerpt (I've only left a few code samples and links
- to some resources on R, Stats, ...) from my own journal. This is a
- personal document where everything (meeting notes, hacking, random
- thoughts, ...) goes by default. Entries are created with the =C-c c=
- shortcut.
-2. [[file:labbook_single.org][labbook_single.org]]: this is an excerpt from the laboratory notebook
- [[https://cornebize.net/][Tom Cornebize]] wrote during his Master thesis internship under my
- supervision. This a personal labbook. I consider this notebook to be
- excellent and was the ideal level of details for us to communicate
- without any ambiguity and for him to move forward with confidence.
-3. [[file:paper.org][paper.org]]: this is an ongoing paper based on the previous labbook of
- Tom Cornebize. As such it is not reproducible as there are hardcoded
- paths and uncleaned dependencies but writing it from the labbook was
- super easy as we just had to cut and paste the parts we
- needed. What may be interesting is the organization and the org
- tricks to export to the right LaTeX style. As you may notice, in
- the end of the document, there is a commented section with emacs
- commands that are automatically executed when opening the file. It
- is an effective way to depend less on the =.emacs/init.el= which is
- generally customized by everyone.
-4. [[file:labbook_several.org][labbook_several.org]]: this is a labbook for a specific project shared
- by several persons. As a consequence it starts with information
- about installation, common scripts, has section with notes about all
- our meetings, a section with information about experiments and an
- other one about analysis. Entries could have been labeled by who
- wrote them but there were only a few of us and this information was
- available in git so we did not bother. In such labbook, it is common
- to find annotations indicating that such experiment was =:FLAWED:= as
- it had some issues.
-5. [[file:technical_report.org][technical_report.org]]: this is a short technical document I wrote
- after a colleague sent me a PDF describing an experiment he was
- conducting and asked me about how reproducible I felt it was. It
- turned out I had to cut and paste the C code from the PDF, then
- remove all the line numbers and fix syntax, etc. Obviously I got
- quite different performance results but writing everything in
- org-mode made it very easy to generate both HTML and PDF and to
- explicitly explain how the measurements were done.
-
-Here are a few links to other kind of examples:
-- Slides: all my slides for a series of lectures is available here:
- https://github.com/alegrand/SMPE. Here is a [[https://raw.githubusercontent.com/alegrand/SMPE/master/lectures/lecture_central_limit_theorem.org][typical source]] and the
- [[https://raw.githubusercontent.com/alegrand/SMPE/master/lectures/lecture_central_limit_theorem.pdf][resulting PDF]]
-- Lucas Schnorr, a colleague, maintains:
- - a set of templates for various computer science
- journals/conferences: [[https://github.com/schnorr/ieeeorg][IEEE]], [[https://github.com/schnorr/wileyorg][Wiley]], [[https://github.com/schnorr/acmorg][ACM]], [[https://github.com/schnorr/llncsorg][LNCS]]
- - his lecture on programming languages for undergrads:
- https://github.com/schnorr/mlp/tree/master/conteudo
-
diff --git a/module2/ressources/video_examples/journal.org b/module2/ressources/video_examples/journal.org
deleted file mode 100755
index 70223877aafd5db908324763d3ec1f063bb266fa..0000000000000000000000000000000000000000
--- a/module2/ressources/video_examples/journal.org
+++ /dev/null
@@ -1,2136 +0,0 @@
-# -*- coding: utf-8 -*-
-#+TITLE: Blog
-#+AUTHOR: Arnaud Legrand
-#+HTML_HEAD:
-#+STARTUP: overview indent inlineimages logdrawer
-#+LANGUAGE: en
-#+TAGS: Seminar(s)
-#+TAGS: SG(s) WP1(1) WP2(2) WP3(3) WP4(4) WP5(5) WP6(6) WP7(7) WP8(8) WP0(0) Argonne(A)
-#+TAGS: POLARIS(P) LIG(L) INRIA (I) HOME(H) Europe(E)
-#+TAGS: twitter(t)
-#+TAGS: Workload(w) BOINC(b) Blog noexport(n) Stats(S)
-#+TAGS: BULL(B)
-#+TAGS: autotuning(a)
-#+TAGS: Epistemology(E) Vulgarization(V) Teaching(T)
-#+TAGS: R(R) Python(p) OrgMode(O) HACSPECIS(h)
-#+PROPERTY: header-args :eval never-export
-#+EXPORT_SELECT_TAGS: Blog
-#+OPTIONS: H:3 num:t toc:t \n:nil @:t ::t |:t ^:t -:t f:t *:t <:t
-#+OPTIONS: TeX:t LaTeX:nil skip:nil d:nil todo:t pri:nil tags:not-in-toc
-#+LATEX_HEADER: %\usepackage{palatino,a4wide,eurosym,graphicx}\usepackage[francais]{babel}
-#+INFOJS_OPT: view:nil toc:nil ltoc:t mouse:underline buttons:0 path:http://orgmode.org/org-info.js
-#+EXPORT_SELECT_TAGS: export
-#+EXPORT_EXCLUDE_TAGS: noexport
-#+EPRESENT_FRAME_LEVEL: 2
-#+COLUMNS: %25ITEM %TODO %3PRIORITY %TAGS
-#+SEQ_TODO: TODO(t!) STARTED(s!) WAITING(w!) APPT(a!) | DONE(d!) CANCELLED(c!) DEFERRED(f!) DELEGATED(D!)
-
-
-* 2011
-** 2011-02 février
-*** 2011-02-08 mardi :R:
-**** Pour apprendre:
- - Pour les débutants:
- http://wiki.stdout.org/rcookbook/
- http://www.r-bloggers.com/
- http://rstudio.org/ but emacs is just great too once ess is installed
- - Indispensables:
- + http://had.co.nz/ggplot2/
- + http://plyr.had.co.nz/ et la démonstration par l'exemple
- http://plyr.had.co.nz/09-user/
- - Une intro pas trop mal faite:
- - http://bioconnector.github.io/workshops/lessons/intro-r-lifesci/01-intro-r/
- - Encore plus interactif: http://swirlstats.com/
- - Plus avancé:
- http://cran.r-project.org/doc/contrib/Paradis-rdebuts_fr.pdf
- - Pour ceux qui veulent aller plus loin et coder
- http://zoonek2.free.fr/UNIX/48_R/all.html
- - Bien plus avancé pour les fans de sémantique et de ruses de
- fou par Hadlay Wickam:
- http://adv-r.had.co.nz/Computing-on-the-language.html
- - Un [[http://ww2.coastal.edu/kingw/statistics/R-tutorials/dataframes.html][excellent tutorial on data frames]] (=attach=, =with=, =rownames=,
- =dimnames=, notions of scope...)
-**** R 101 :Blog:
-[[file:public_html/blog/2012/09/12/R101.org][Moved to the blog]]
-[[file:~/Work/SimGrid/infra-songs/slides/140422-compas-R101/R101.org][Compas tutorial]]
-**** R tricks
-***** Reshaping
-http://www.statmethods.net/management/reshape.html
-#+begin_src R :results output :session :exports both
-# example of melt function
-library(reshape)
-mdata <- melt(mydata, id=c("id","time"))
-#+end_src
-***** somme d'éléments avec fenêtre glissante
-#+begin_src R
-filter(x, rep(1,4))
-#+end_src
-***** sorting a data frame
-#+BEGIN_SRC R
- dd[with(dd, order(-z, b)), ]
-#+END_SRC
-***** Capture output
-#+begin_src R
-sink("myfile.txt", append=TRUE, split=TRUE)
-#+end_src
-When redirecting output, use the cat() function to annotate the
-output.
-***** Batch processing
-#+begin_src sh
-R CMD BATCH [options] my_script.R [outfile]
-#+end_src
-***** Convenient commands
- - describe
- - structure
- - ddply
- - cbind/rbind
-***** Labels and Factors
- http://stackoverflow.com/questions/12075037/ggplot-legends-change-labels-order-and-title
-#+begin_src R
-dtt$model <- factor(dtt$model, levels=c("mb", "ma", "mc"), labels=c("MBB", "MAA", "MCC"))
-#+end_src
- here is another way of reordering factors:
-#+begin_src R
-dtt$model <- relevel(dtt$model, ref="MBB").
-#+end_src
-This puts the factor given by ref at the beginning.
-***** "parallel" Prefix
-#+BEGIN_SRC
-cumsum
-#+END_SRC
-***** knitr preembule
-check out "Tools for making a paper" in R-bloggers:
-#+BEGIN_SRC
-<>=
-opts_knit$set(stop_on_error=2L)
-@
-<>=
-suppressMessages(require(memisc))
-@
-#+END_SRC
-***** Annotate in facet_wrap/facet_grid
-http://www.ansci.wisc.edu/morota/R/ggplot2/ggplot2.html
-***** Interactive plotting
- http://rstudio.org/docs/advanced/manipulate
- [[file:~/Work/SimGrid/infra-songs/WP4/R/Sweep3D_analysis/analyze.Rnw]]
-#+begin_src R
-GC <- function(df,start,end) {
- ggplot(
- df[(df$Start>=start & df$Start<=end)|(df$End>=start &
- df$End<=end)|
- (df$Start<=start & df$End>=end) ,],
- aes(xmin=Start,xmax=End, ymin=ResourceId, ymax=ResourceId+1,
- fill=Value))+
- theme_bw()+geom_rect()+coord_cartesian(xlim = c(start, end))
-}
-GC(df_tau,1.1,1.2)
-animate(GC(df_tau, start, end),start=slider...)
-#+end_src
-***** scoping issue with ggplot: mixing external variables with column names
- There is a magical function designed for this: here()
-#+BEGIN_EXAMPLE
-ddply(df_native, c("ResourceId"), here(transform),
- Chunk = compute_chunk(Start,End,Duration,min_time_pure))
-#+END_EXAMPLE
- Here, min_time_pure is an external variable, not a column name.
-***** speeding things up with parallel plyR
-#+BEGIN_SRC R
- library(doMP)
- library(plyr)
- perf_win <- ddply(df_win,c("host_id"), summarize,
- astro_avg=sum(et_avg*astro_win),
- astro_var=sum(et_var*astro_win),
- seti_avg=sum(et_avg*seti_win),
- seti_var=sum(et_var*seti_win),
- .parallel=TRUE, .progress = "text")
-#+END_SRC
-
-***** dessin de graphe avec courbes de bezier
- https://gist.github.com/dsparks/4331058
-***** Side effect in local functions
-http://my.safaribooksonline.com/book/programming/r/9781449377502/9dot-functions/id3440389
-***** Non-standard evaluation
-http://adv-r.had.co.nz/Computing-on-the-language.html
-***** Arrays of functions in for loops
-http://stackoverflow.com/questions/26064649/enclosing-variables-within-for-loop
-**** R weblinks/statistiques r-cran :WP8:
- http://en.wikibooks.org/wiki/R_Programming/Graphics
-
- http://freecode.com/articles/creating-charts-and-graphs-with-gnu-r
-
- http://www.statmethods.net/graphs/density.html
-
- http://addictedtor.free.fr/graphiques/RGraphGallery.php?graph=78
- (scatterplot + histogram)
-
- http://cran.r-project.org/doc/contrib/Ricci-distributions-en.pdf
- (comparaison et fitting de distributions)
-
- http://www.sr.bham.ac.uk/~ajrs/R/r-gallery.html
- http://www.som.yale.edu/faculty/pks4/files/teaching/handouts/r2_tstat_explained.pdf
- about t-values
-
- http://www.statmethods.net/stats/anova.html
-
- http://www.stat.wisc.edu/courses/st850-lindstro/handouts/blocking.pdf
- (blocking dans un anova en R)
-
- http://www-rocq.inria.fr/axis/modulad/archives/numero-34/Goupy-34/goupy-34.pdf
- (tutorial on DOE in French)
-
- file:/home/alegrand/Work/Documents/Enseignements/M2R_Mesures_Analyse_Eval_Perf_06/Intro_Statistics/doesimp2excerpt--chap3.pdf
- file:/home/alegrand/Work/Documents/Enseignements/M2R_Mesures_Analyse_Eval_Perf_06/Intro_Statistics/doeprimer.pdf
- DOE
-
- http://cran.r-project.org/doc/contrib/Faraway-PRA.pdf
- (gros bouquin de R sur l'ANOVA)
- http://pages.cs.wisc.edu/~cyffka/R_regression-and-anova.pdf
-
- https://marvelig.liglab.fr/doku.php/thematiques/methodologie/accueil
- Documents de Nadine Mandran + pointeurs vers cours de stat
-
- http://pbil.univ-lyon1.fr/R/pdf/bsa.pdf
- http://grasland.script.univ-paris-diderot.fr/go303/ch5/doc_ch5.htm
- Document sur l'analyse de données spatialisées
-
- http://nsaunders.wordpress.com/2010/08/20/a-brief-introduction-to-apply-in-r/
- Utilisation de Apply
-
- http://zoonek2.free.fr/UNIX/48_R/all.html
- Un utilisateur de R qui a noté tout un tas de choses utiles et
- d'exemples, nottamment de programmation.
-
- http://sharpstatistics.co.uk/r/ggplot/
- http://rug.mnhn.fr/semin-r/PDF/INED-SFdS-MNHN_Sueur_280411.pdf
- Tutorial ggplot2
-
- https://catalyst.uw.edu/workspace/tbranch/24589/155528
- Cours sur la visu en R à la Tufte
-
-***** Linear regression and heteroscedasticity :ATTACH:
-:PROPERTIES:
-:Attachments: ModeleLineaireRegrDegerine.pdf Regression101R.pdf GLSHeteroskedasticity.pdf week2_ht.pdf
-:ID: b3ced951-cda8-40ce-b281-cc71b55f1da9
-:END:
-- http://ljk.imag.fr/membres/Anatoli.Iouditski/cours/MLDESS.pdf (see
- attachment) cours en français sur la régression linéaire, vision
- proba.
-- http://smat.epfl.ch/courses/Regression/Slides/week2_ht.pdf slides
- sur la régression linéaire et le lien avec maximum likelihood
-- http://www.r-tutor.com/elementary-statistics/simple-linear-regression/confidence-interval-linear-regression
- #+begin_src R :results output :session :exports both
- predict(eruption.lm, newdata, interval="confidence")
- #+end_src
-- http://www.princeton.edu/~otorres/Regression101R.pdf (std error et
- confidence interval on parameters estimates + heteroscedasticity)
-- http://www.econ.uiuc.edu/~wsosa/econ471/GLSHeteroskedasticity.pdf
- Comment gérer l'hétéroscedastisité.
-***** Time series :ATTACH:
-:PROPERTIES:
-:Attachments: SCBio.pdf
-:ID: 40d5498d-e8b3-4c73-8722-7d0056667c15
-:END:
-http://ljk.imag.fr/membres/Serge.Degerine/Enseignement/SCBio.pdf
-
-**** Quantile Regression and Bootstrap :Stats:ATTACH:
-:PROPERTIES:
-:Attachments: mcgill-r.pdf st-m-app-bootstrap.pdf stnews70.pdf
-:ID: 8e3038dc-fa3e-4a7d-a4b1-216513e4359f
-:END:
-http://freakonometrics.hypotheses.org/date/2012/04 (open data and
-ecological falacies (Simpson's paradox)).
-
-http://freakonometrics.hypotheses.org/2396
-(Talk-on-quantiles-at-the-R-Montreal-group)
-
-http://www.cscu.cornell.edu/news/statnews/stnews70.pdf
-**** Reproducible research :WP8:
-Andrew Davison tutorial, which is full of interesting references:
-http://rrcns.readthedocs.org/en/latest/index.html
-***** org-mode
-Une autre approche, uniquement en org.
-http://orgmode.org/worg/org-contrib/babel/how-to-use-Org-Babel-for-R.html
-***** R/Sweave/knitr
-http://users.stat.umn.edu/~geyer//Sweave/
-Sweave, exemple minimaux, emacs.
-
-http://www.bepress.com/cgi/viewcontent.cgi?article=1001&context=bioconductor
-Un article sur reproducible research et sweave
-
-http://cran.r-project.org/web/packages/pgfSweave/vignettes/pgfSweave.pdf
-Pgfsweave, un paquet latex qui améliore le look et la vitesse de
-sweave. Le paquet est mort ceci dit et mes premiers essais n'étaient
-pas concluants car tout convertir en pgf, c'est un peu bourrin.
-
-http://yihui.name/knitr/
-knitr, le dernier, bien à la mode, stable et très prometteur
-
-
-http://www.stat.uiowa.edu/~rlenth/StatWeave/OLD/SRC-talk.pdf
-StatWeave. Permet aussi de mettre du Maple.
-***** Ipython notebook
-https://osf.io/h9gsd/
-sympa, facile à mettre en place
-***** ActivePapers
-- http://www.activepapers.org/
-- https://bitbucket.org/khinsen/active_papers_py/wiki/Tutorial
-***** Elsevier approach
-http://www.elsevier.com/physical-sciences/computer-science/executable-papers
-https://collage.elsevier.com/manual/
-http://is.ieis.tue.nl/staff/pvgorp/research/?page=SCP11
-***** Research Gate
-https://www.researchgate.net/publicliterature.OpenReviewInfo.html
-***** Conference or general discussions
-http://reproducibleresearch.net/index.php/Main_Page
-http://wiki.stodden.net/Main_Page
-
-
-- [[http://www.eecg.toronto.edu/~enright/wddd/][Workshop on Duplicating, Deconstructing and Debunking (WDDD)]] ([[http://cag.engr.uconn.edu/isca2014/workshop_tutorial.html][2014
- edition]])
-- http://evaluate2010.inf.usi.ch
-- [[http://www.stodden.net/AMP2011/][Reproducible Research: Tools and Strategies for Scientific Computing]]
-- [[http://wssspe.researchcomputing.org.uk/][Working towards Sustainable Software for Science: Practice and
- Experiences]]
-- [[http://hunoldscience.net/conf/reppar14/pc.html][REPPAR'14: 1st International Workshop on Reproducibility in Parallel
- Computing]]
-- [[https://www.xsede.org/web/reproducibility][Reproducibility@XSEDE: An XSEDE14 Workshop]]
-- [[http://www.occamportal.org/reproduce][Reproduce/HPCA 2014]]
-- [[http://www.ctuning.org/cm/wiki/index.php?title%3DEvents:TRUST2014][TRUST 2014]]
-- http://vee2014.cs.technion.ac.il/docs/VEE14-present602.pdf
-
- http://www-958.ibm.com/software/data/cognos/manyeyes/visualizations
- http://www.myexperiment.org/
- http://wiki.galaxyproject.org/
- http://www.runmycode.org/CompanionSite/
-
- http://evaluate.inf.usi.ch/
-
- github ?
- workflow ?
- vistrails ?
- sumatra
- vcr
-***** Politics
-http://michaelnielsen.org/blog/how-you-can-help-the-federal-research-public-access-act-frpaa-become-law/7
-
-http://en.wikipedia.org/wiki/Federal_Research_Public_Access_Act
-
-http://michaelnielsen.org/blog/on-elsevier/
-
-**** General discussions about scientific practice :WP8:
-http://michaelnielsen.org/blog/three-myths-about-scientific-peer-review*/
-http://michaelnielsen.org/blog/some-garbage-in-gold-out/
-http://www.ploscompbiol.org/article/info%3Adoi%2F10.1371%2Fjournal.pcbi.1003285
-**** Coursera
-- https://www.coursera.org/course/compdata
-- https://class.coursera.org/exdata-002/lecture
-- https://class.coursera.org/repdata-002
-**** ggplot2 cool examples
-http://felixfan.github.io/rstudy/2014/02/28/ggplot2-cheatsheet/
-http://blog.revolutionanalytics.com/graphics/
-http://grrrraphics.blogspot.com.br/2012/05/ever-wanted-to-see-at-glance.html
-http://www.ancienteco.com/2012/03/basic-introduction-to-ggplot2.html
-http://sape.inf.usi.ch/quick-reference/ggplot2
-http://www.r-bloggers.com/overplotting-solution-for-black-and-white-graphics/
-http://stats.stackexchange.com/questions/12029/is-it-possible-to-create-parallel-sets-plot-using-r
-http://novyden.blogspot.fr/2013/09/how-to-expand-color-palette-with-ggplot.html
-https://gastonsanchez.wordpress.com/2012/08/27/scatterplot-matrices-with-ggplot/
-**** Visualisations
-http://www.visual-literacy.org/periodic_table/periodic_table.html
-**** Design of Experiments (DoE)
-- Montgommery book
-- http://www.cs.wayne.edu/~hzhang/courses/7290/Lectures/4%20-%20Introduction%20to%20Experimental%20Design.pdf
-- http://www.obgyn.cam.ac.uk/cam-only/statsbook/stexdes.html#3g
-- http://mescal.imag.fr/membres/arnaud.legrand/teaching/2011/EP_czitrom.pdf
-- http://www.basic.northwestern.edu/statguidefiles/oneway_anova_ass_viol.html
-- http://techdigest.jhuapl.edu/TD/td2703/telford.pdf
-*** 2011-02-15 mardi
-**** Réunion CIGRI
-***** Présents
- - Olivier Richard, Mcf UJF/MESCAL, gestions de ressources,
- initiateur de OAR et Cigri, G5K
- - Bruno Bzeznik, Ingénieur CIMENT (admin, gestion clusters) et
- MESCAL (dev OAR, outils pour CIMENT).
- - Chislain Charrier, Ingénieur INRIA G5K à Rennes depuis quelques
- mois. Mission: s'occuper des campagnes d'expérimentations.
- - Philippe Leprouster, Ingénieur CDD UJF MESCAL pour bosser sur
- l'optimisation d'OAR
- - Bernard Boutherin, responsable info au LPSC, noeud Tier3 de la
- grille EGI (600 coeurs de calcul, 700 To de stoquage,
- précurseur autour du free-cooling, installation à moins de 60
- kW depuis 2008).
- - Catherine Biscarat, IR CNRS qui va s'occuper de la liaison
- CIGRI/LPSC.
- - Pierre Neyron, IR CNRS MESCAL/MOAIS, responsable de digitalis.
-***** Point de Bruno sur l'état actuel de Cigri
- Site web: https://ciment-grid.ujf-grenoble.fr
- Logiciel principalement déployé dans CIMENT. Exploite
- actuellement 3000 cores sur une vingtaine de machines.
- R2D2 et fostino sont les plus grosses et gérées par un seul
- serveur OAR.
-
- Resources très faiblement utilisées (en général un ou deux
- utilisateurs à un instant donnée). Besoin d'accompagner les
- utilisateurs qui n'ont pas forcément conscience que CIGRI est
- adapté à leurs besoins. Les utilisateur qui utilisent CIGRI
- actuellement sont de gros consommateurs de ressources.
-
- Collaboration CIGRI/LPSC initiée par un projet autour du
- stoquage. Bruno a du coup équipé CIGRI de noeuds de stoquage et
- a déployé Irods.
-***** Plus d'infos sur : http://wiki-oar.imag.fr/index.php/CiGri-ng
-
-Entered on [2011-02-15 mar. 09:41]
-
- [[file:~/Liste.org]]
-* 2012
-** 2013-02 février
-**** 2013-02-11 lundi
-***** Reproducible research links :WP8:R:
- http://wiki.stodden.net/ICERM_Reproducibility_in_Computational_and_Experimental_Mathematics:_Readings_and_References
- http://www.rpubs.com/
- An interesting article with a dissenting opinion on reproducible research:
- http://cogprints.org/8675/
-
- Entered on [2013-02-11 lun. 09:52]
-***** Audio StarPU :WP4:
- Lionel, Samuel, Paul, Luka, Brice.
-****** Séquentialisation des comms
- - Idées: faire des mesures automatiques
- - Deux implems' (Sam & Paul), pas équivalentes, l'une modélisant
- plus les communications synchrone et l'autre les
- asynchrones. À creuser.
-****** Petites macros pour mesurer/injecter le temps
- - Temps injecté dans la version initiale de Sam = temps moyen
- observé par StarPU .
- - Une fois les problème de communication réglés (virer le
- slow-start, séquentialiser ce qui doit l'être), les dernières
- différences viennent de la variabilité vraie vie / simu
- (surtout sur CPU).
- - Objectif: insérer variabilité. C'est le même problème que pour
- SMPI. Dans la version actuelle, on regarde le temps pris lors
- de la simulation et on le réinjecte, d'où une très mauvaise portabili
- - Idée: identifier les blocs, capturer les temps et utiliser en
- simu un tirage à partir du profil capturé. C'est assez
- "nouveau" car FSuter capturait une trace de niveau MPI donc
- sans info sur quel bloc de code => pas d'information sur la
- source de la variabilité.
- - On commence par une approche basique: à la compilation, on
- identifie un bloc par FILE,LINE, avec éventuellement une
- extension via une annotation manuelle (c'est le cas pour
- StarPU qui lance les calculs toujours au même endroit).
- - Niveau workflow, première exécution pour avoir les timings,
- puis R, puis réinsertion ds SG.
- - La capture est pas compliquée et comme il y a le même besoin
- pour SMPI, on factorise pour éviter les divergences. Ce code
- est donc dans SG. Paul et Luka ont fait ça la semaine dernière
- et Paul l'a utilisé dans *PU, reste à tester pour confirmer
- - Luka essaie maintenant de mettre ça ds SMPI, c'est plus
- difficile de savoir où mettre les benchmarks. L'ideal serait
- de regarder dans la pile, c'est un peu compliqué donc on reste
- sur notre approche simple pour l'instant et on raffinera plus
- tard si c'est vraiment nécessaire. L'avantage escompté, c'est
- sur les plates-formes Mt Blanc par exemples, on peut exécuter
- une fois et utiliser ensuite les timings pour faire des tests
- de scalabilité sur une vraie machine de brute qui va vite.
-****** Objectifs des uns et des autres
- - Lionel & Paul à Bdx: objectif = proposer des modèles, support
- - Sam: objectif = bricoler DES ordonnanceurs, lancer vite sur
- différentes et évaluer l'impact de tailles de blocs ou de la
- taille d'une fenêtre glissante. C'est donc clairement un outil
- de développement pour tester des choses et il faut donc que
- l'outil soit un minimum stable. Rien de grave mais il faut
- bien en être conscient en terme de développement. Il sera
- important de propager les informations du genre "attention on
- a corrigé un truc, ça risque d'invalider les expériences
- précédentes".
- - Arnaud rappelle que d'un point de vue développement, c'est
- comme pour SMPI, il faut être conscient qu'il y a trois types
- de tâches toutes aussi importantes les unes que les autres
- (i.e., quand on en néglige une on s'en mord toujours les
- doigts à un moment ou à un autre):
- + Exploration: le plus fun, de petites expériences pour voir
- si ça marche. Pour moi, la démarche de Sam et ses
- expériences faites rentrent dans cette catégorie.
- + Ingénierie: écriture de code, petites fonctions
- techniques. Dans le contexte de starPU, typiquement, il
- s'agit du travail initial de Sam mais aussi du codage de la
- séquentialisation des communications ou bien des macros de
- capture de traces.
- + Consolidation: moins drôle, mais il faut le faire pour
- vérifier que tout le monde peut fairer ses mesures,
- réutiliser, et qu'on puisse avancer en toute
- confiance. Souvent, sur une nouvelle machine, de nouveaux
- phénomènes apparaissent et ce n'est qu'avec des outils
- d'exploration systématique et automatiques qu'on s'en sort.
-
- Il faut donc mettre en place dès le début des outils de capture
- d'information et d'analyse.
-****** Roadmap
- - Séquentialisation / Parallélisation: Paul à Bordeaux s'en
- occupe. Il met en place le code qui crache une matrice
- d'interférences et met en place dans StarPU/SG le code qui
- l'exploite.
- - Infrastructure de mesure / collecte de traces: Luka et Arnaud à
- Grenoble s'en occupent. Réflexion sur un workflow qui va bien
- pour garder de bonnes traces et pouvoir facilement tester de
- nouvelles machines.
- - Paul vient le 14 Mars à Grenoble et on en profitera pour faire
- le point.
-****** Divers
-******* Campagne d'expériences pour valider le modèle / invalider les précédents. Comment faire ?
- Il est difficile (impossible ?) de dire qu'un modèle est
- valide. Il est plus raisonnable de montrer à quel point on a
- essayer de l'invalider, ce qui permet à chacun d'évaluer à quel
- point il fait confiance aux capacités d'extrapolation et
- d'explication du modèle.
-
- On peut donc montrer l'impact des améliorations successives du
- modèle, soit sur le temps final soit sur des choses plus fines
- de la trace. Il faut tester sur des cas de plus en plus
- complexes, d'où la nécessité d'avoir une méthode un peu
- automatique pour comparer des résultats. À titre d'illustration
- voici le genre de choses que Martin a raconté à l'éval d'Héméra.
-
- http://mescal.imag.fr/membres/arnaud.legrand/uss_simgrid/130211-HEMERA-eval.pdf
-
- + Mesure 0: makespan. C'est ce qui nous intéresse mais c'est
- généralement très pauvre et on peut arriver par hasard à de
- bons résultats ou avoir de mauvais résultats juste parce
- qu'un paramètre a été mal mesuré. Même si c'est uniquement,
- cette mesure là qui nous intéresse au final, il est
- indispensable de comparer pour des mesures plus fines car
- c'est ce qui permet de mettre une certaine confiance dans les
- capacités d'extrapolation de l'outil.
- + Mesure 1: comparaison visuelle de gantt chartt, peut être
- joli, facile et instructif avec R, mais difficilement
- quantifiable.
- + Mesure 2: Regarder les distributions de temps passés dans
- différents états. On peut faire ça en partie pour StarPU mais
- uniquement pour les temps de calcul, pas pour les temps de
- communication. En effet, on a peu de maîtrises sur les temps
- de comms, on ne sait pas vraiment à quel moment la
- communication s'est terminée ni quand elle a commencé.
- + Mesure 3: Comparer les schedules... C'est difficile car déjà
- quand c'est stable, une métrique n'est pas évidente à définir
- mais quand en plus c'est variable d'une fois sur
- l'autre... Idéalement, il faudrait comparer plutôt la
- distribution des schedules plutôt que les schedules
- individuels... C'est difficile mais passionnant. Ce qui est
- super, c'est qu'on a l'outil qui permet de générer les
- traces.
-******* Problème de main() entre *PU et SG
- Passer systématiquement par le XML a résolu le problème de
- l'initialisation et du lancement de simgrid. Ça reste gênant
- parce qu'il faut recompiler l'appli Sam. aimerait pouvoir
- changer l'appli en changeant le LD_LIBRARY_PATH, avec 2
- versions de libstarpu.so => Mais alors comment passer les
- arguments à SG ?
-
- C'est l'appli qui donne argc,argv dans l'appel à *pu_init
- L'appli risque de pas aimer les options de SG.
-
- *pu passe par des variables d'environnements pour éviter ces soucis
-
- Vision d'arnaud : --platform=toto.xml mangé par *pu, reste des
- args mangés par SG Comment faire pour mettre les stats de temps
- d'exécution adaptés au fichier XML Ptetre qu'on peut identifier
- les GPUs par modèle plutôt que par numéro de GPU Possibilité de
- vérifier les hostname (STARPU_HOSTNAME)
-******* Comment faire pour que ça marche pour exécuter en local.
- On peut pondre le .xml, tout classer par hostname. Tout est dans
- .starpu/sampling, pour les différents hostnames et codelets, les
- traces de perfs.
-
- Du coup ce serait *pu qui génèrerait automatiquement le .xml ?
- ça coûte pas cher. Mais les valeurs qu'on met dedans, comment
- on les obtient ?
- + BP/latence du bus sont mesuréesde toutes façons au départ
- + Pour la matrice d'interférence de Paul, il faudrait aussi la
- mettre dans le XML.
- Du coup, il serait peut-être plus naturel que ce soit le script
- de Paul qui ponde le .XML qui s'occupe de l'étalonnage
-
- Entered on [2013-02-11 lun. 09:53]
-** 2013-05 mai
-*** 2013-05-21 mardi
-**** BIS Workshop
-
-#+BEGIN_SRC sh :results output raw :exports both
-for i in gnome gnome-desktop-environment ifupdown iproute iproute-dev isc-dhcp-client libatm1 network-manager network-manager-gnome ; do
- dir=`apt-cache showsrc $i | grep Directory | sed 's/.*: //'`
- version=`apt-cache showsrc $i | grep ^Version | sed 's/.*: *//g'`
- echo "wget http://http.us.debian.org/debian/$dir/$i""_$version"_amd64.deb
-done
-#+END_SRC
-
-Entered on [2013-05-21 mar. 08:46]
-**** Discussions avec Anne-Cécile à propos des timeouts TCP MPI :WP4:
- I have some related news. I had the chance to chat with
- Anne-Cecile and talked her about our timeout problem. After
- digging a little, she was able to point me to related work:
- - Understanding TCP Incast Throughput Collapse in Datacenter
- Networks
- - Safe and Effective Fine-grained TCP Retransmissions for
- Datacenter Communication
- - On the properties of an adaptive TCP minimum rto
- - http://www.hjp.at/doc/rfc/rfc2988.txt
-
- So the problem has a name (incast) and is linked to the following
- TCP parameter:
- | OS | TCP RTOmin |
- |---------+-------------|
- | Linux | 200ms |
- | BSD | 200ms |
- | Solaris | 400ms |
-
- I haven't read the articles so I don't know the details. All I
- can say so far is I don't know how to change this parameter
- without recompiling the kernel...
-
- #+BEGIN_SRC sh :results output raw :exports both
- cd /usr/src/linux-headers-3.2.0-4-common
- cg 'define *TCP_RTO_MIN' '*'
- cg 'define *HZ' '*'
- #+END_SRC
-
- Basically, this value is good enough for wide area where RTT is
- large but in our SAN setting, it's rather bad. Looking further,
- http://comments.gmane.org/gmane.linux.network/162986, I learnt
- that although this parameter cannot be modified through sysctl,
- it could be overriden per route with iproute.
-
-
- #+BEGIN_SRC sh :results output text :exports both
- for i in ip ip-address ipcontroller iplogger ip-netns iproxy iptables-save ipython ip6tables ip-addrlabel ipcrm ip-maddress ip-ntable ip-rule iptables-xml ipython2.6 ip6tables-apply ipc ipcs ip-monitor ipptool iptables ip-tunnel ipython2.7 ip6tables-restore ipcluster ipengine ip-mroute ipptoolfile iptables-apply ipv6 ip6tables-save ipcmk ip-link ip-neighbour ip-route iptables-restore ip-xfrm ; do
- man -T --troff-device=ascii $i | grep -i rto
- done
- #+END_SRC
-
- #+RESULTS:
-
-Entered on [2013-05-21 mar. 12:37]
-**** Play with R xkcd :R:
- #+begin_src R :results output :session :exports both
- # install.packages('xkcd') # did not work so I made it manually
- library(extrafont)
- download.file("http://simonsoftware.se/other/xkcd.ttf", dest="xkcd.ttf")
- system("mkdir ~/.fonts")
- system("cp xkcd.tff -t ~/.fonts")
- font_import()
- loadfonts()
- #+END_SRC
-
- #+BEGIN_SRC R :results graphics :file /tmp/plot.png :exports results :width 600 :height 200 :session
- library(xkcd)
- theme_xkcd <- theme(
- panel.background = element_rect(fill="white"),
- axis.ticks = element_line(colour=NA),
- panel.grid = element_line(colour="white"),
- axis.text.y = element_text(colour=NA),
- axis.text.x = element_text(colour="black"),
- text = element_text(size=16, family="xkcd")
- )
- ggplot(data.frame(x=c(0, 10)), aes(x)) + theme_xkcd +
- stat_function(fun=sin,position="jitter", color="red", size=2) +
- stat_function(fun=cos,position="jitter", color="white", size=3) +
- stat_function(fun=cos,position="jitter", color="blue", size=2) +
- geom_text(family="xkcd", x=4, y=0.7, label="A SIN AND COS CURVE")+
- xkcdaxis(c(0, 10),c(-1,1))
- #+END_SRC
-
- #+RESULTS:
- [[file:/tmp/plot.png]]
-
-Entered on [2013-05-21 mar. 15:49]
-* 2015
-** 2015-01 janvier
-*** 2015-01-07 mercredi
-**** Helping Martin with R :Teaching:R:
-
-#+tblname: daily
-| Date | exos_java | traces_java | exos_python | traces_python | exos_scala | traces_scala |
-|------------+-----------+-------------+-------------+---------------+------------+--------------|
-| 2014.9.2 | 6 | 1 | 0 | 0 | 0 | 0 |
-| 2014.9.3 | 5 | 1 | 0 | 0 | 0 | 0 |
-| 2014.9.4 | 8 | 2 | 0 | 0 | 0 | 0 |
-| 2014.9.8 | 7 | 4 | 0 | 0 | 1290 | 86 |
-| 2014.9.9 | 0 | 0 | 3 | 1 | 1615 | 86 |
-| 2014.9.10 | 0 | 0 | 1 | 1 | 163 | 16 |
-| 2014.9.11 | 3 | 2 | 0 | 0 | 999 | 63 |
-| 2014.9.12 | 67 | 4 | 2 | 2 | 1149 | 67 |
-| 2014.9.13 | 20 | 3 | 1 | 1 | 132 | 14 |
-| 2014.9.14 | 7 | 1 | 0 | 0 | 170 | 12 |
-| 2014.9.15 | 9 | 2 | 0 | 0 | 1112 | 73 |
-| 2014.9.16 | 16 | 2 | 0 | 0 | 768 | 60 |
-| 2014.9.17 | 36 | 3 | 0 | 0 | 274 | 40 |
-| 2014.9.18 | 1 | 1 | 22 | 2 | 20 | 2 |
-| 2014.9.19 | 1 | 1 | 18 | 2 | 10 | 2 |
-| 2014.9.20 | 0 | 0 | 12 | 1 | 61 | 6 |
-| 2014.9.21 | 0 | 0 | 6 | 2 | 36 | 6 |
-| 2014.9.22 | 3 | 2 | 11 | 2 | 420 | 50 |
-| 2014.9.23 | 1 | 1 | 0 | 0 | 218 | 31 |
-| 2014.9.24 | 0 | 0 | 12 | 2 | 39 | 4 |
-| 2014.9.25 | 0 | 0 | 1 | 1 | 220 | 30 |
-| 2014.9.26 | 0 | 0 | 19 | 2 | 28 | 5 |
-| 2014.9.27 | 0 | 0 | 10 | 1 | 17 | 4 |
-| 2014.9.28 | 0 | 0 | 12 | 2 | 37 | 6 |
-| 2014.9.29 | 26 | 3 | 8 | 1 | 509 | 81 |
-| 2014.9.30 | 9 | 2 | 16 | 2 | 243 | 36 |
-| 2014.10.1 | 1 | 1 | 26 | 14 | 99 | 16 |
-| 2014.10.2 | 1 | 1 | 1 | 1 | 325 | 38 |
-| 2014.10.3 | 26 | 15 | 52 | 16 | 22 | 4 |
-| 2014.10.4 | 25 | 1 | 4 | 3 | 36 | 9 |
-| 2014.10.5 | 10 | 1 | 2 | 1 | 49 | 9 |
-| 2014.10.6 | 5 | 2 | 39 | 22 | 192 | 37 |
-| 2014.10.7 | 24 | 4 | 17 | 7 | 143 | 25 |
-| 2014.10.8 | 50 | 3 | 0 | 0 | 77 | 14 |
-| 2014.10.9 | 24 | 2 | 11 | 3 | 48 | 9 |
-| 2014.10.10 | 35 | 4 | 7 | 2 | 0 | 0 |
-| 2014.10.11 | 0 | 0 | 9 | 3 | 3 | 1 |
-| 2014.10.12 | 20 | 6 | 7 | 3 | 10 | 1 |
-| 2014.10.13 | 32 | 4 | 18 | 4 | 0 | 0 |
-| 2014.10.14 | 44 | 1 | 41 | 3 | 8 | 1 |
-| 2014.10.15 | 5 | 3 | 64 | 10 | 6 | 2 |
-| 2014.10.16 | 27 | 2 | 24 | 5 | 1 | 1 |
-| 2014.10.17 | 43 | 3 | 14 | 4 | 0 | 0 |
-| 2014.10.18 | 84 | 2 | 57 | 8 | 0 | 0 |
-| 2014.10.19 | 10 | 2 | 86 | 11 | 0 | 0 |
-| 2014.10.20 | 0 | 0 | 94 | 11 | 0 | 0 |
-| 2014.10.21 | 15 | 1 | 67 | 8 | 10 | 2 |
-| 2014.10.22 | 20 | 5 | 76 | 15 | 1 | 1 |
-| 2014.10.23 | 33 | 3 | 12 | 5 | 0 | 0 |
-| 2014.10.24 | 29 | 2 | 58 | 11 | 1 | 1 |
-| 2014.10.25 | 33 | 8 | 38 | 8 | 1 | 1 |
-| 2014.10.26 | 13 | 6 | 39 | 8 | 34 | 3 |
-| 2014.10.27 | 13 | 4 | 49 | 12 | 15 | 1 |
-| 2014.10.28 | 4 | 2 | 44 | 8 | 3 | 1 |
-| 2014.10.29 | 0 | 0 | 28 | 9 | 13 | 2 |
-| 2014.10.30 | 4 | 3 | 49 | 8 | 0 | 0 |
-| 2014.10.31 | 3 | 2 | 58 | 14 | 7 | 1 |
-| 2014.11.1 | 0 | 0 | 71 | 9 | 7 | 2 |
-| 2014.11.2 | 23 | 2 | 57 | 6 | 0 | 0 |
-| 2014.11.3 | 10 | 1 | 18 | 5 | 0 | 0 |
-| 2014.11.4 | 19 | 1 | 49 | 10 | 3 | 1 |
-| 2014.11.5 | 29 | 2 | 28 | 9 | 0 | 0 |
-| 2014.11.6 | 86 | 3 | 142 | 19 | 0 | 0 |
-| 2014.11.7 | 38 | 2 | 4 | 2 | 0 | 0 |
-| 2014.11.8 | 0 | 0 | 18 | 4 | 6 | 1 |
-| 2014.11.9 | 25 | 2 | 39 | 10 | 0 | 0 |
-| 2014.11.10 | 16 | 1 | 17 | 3 | 0 | 0 |
-| 2014.11.11 | 0 | 0 | 70 | 16 | 1 | 1 |
-| 2014.11.12 | 0 | 0 | 4 | 3 | 0 | 0 |
-| 2014.11.13 | 0 | 0 | 168 | 20 | 1 | 1 |
-| 2014.11.14 | 0 | 0 | 18 | 2 | 0 | 0 |
-| 2014.11.15 | 0 | 0 | 5 | 2 | 8 | 1 |
-| 2014.11.16 | 16 | 2 | 16 | 4 | 4 | 1 |
-| 2014.11.17 | 0 | 0 | 8 | 3 | 0 | 0 |
-| 2014.11.18 | 4 | 1 | 7 | 3 | 0 | 0 |
-| 2014.11.19 | 17 | 2 | 4 | 1 | 0 | 0 |
-| 2014.11.20 | 0 | 0 | 102 | 13 | 0 | 0 |
-| 2014.11.21 | 7 | 1 | 31 | 3 | 1 | 1 |
-| 2014.11.22 | 1 | 1 | 17 | 4 | 0 | 0 |
-| 2014.11.23 | 4 | 1 | 25 | 6 | 0 | 0 |
-| 2014.11.24 | 0 | 0 | 2 | 1 | 3 | 1 |
-| 2014.11.25 | 4 | 1 | 0 | 0 | 7 | 2 |
-| 2014.11.26 | 0 | 0 | 4 | 1 | 0 | 0 |
-| 2014.11.27 | 0 | 0 | 1 | 1 | 6 | 1 |
-| 2014.11.28 | 0 | 0 | 6 | 3 | 1 | 1 |
-| 2014.11.29 | 1 | 1 | 29 | 4 | 13 | 3 |
-| 2014.11.30 | 3 | 1 | 57 | 10 | 15 | 2 |
-| 2014.12.1 | 8 | 1 | 15 | 4 | 7 | 3 |
-| 2014.12.2 | 8 | 3 | 17 | 5 | 0 | 0 |
-| 2014.12.3 | 3 | 1 | 6 | 2 | 0 | 0 |
-| 2014.12.4 | 4 | 3 | 1 | 1 | 1 | 1 |
-| 2014.12.5 | 0 | 0 | 17 | 2 | 5 | 2 |
-| 2014.12.6 | 0 | 0 | 6 | 2 | 3 | 1 |
-| 2014.12.7 | 0 | 0 | 7 | 3 | 0 | 0 |
-| 2014.12.8 | 11 | 3 | 0 | 0 | 0 | 0 |
-| 2014.12.9 | 7 | 1 | 0 | 0 | 0 | 0 |
-| 2014.12.10 | 27 | 2 | 0 | 0 | 0 | 0 |
-| 2014.12.11 | 0 | 0 | 0 | 0 | 1 | 1 |
-| 2014.12.13 | 17 | 3 | 0 | 0 | 0 | 0 |
-| 2014.12.14 | 3 | 1 | 10 | 1 | 0 | 0 |
-| 2014.12.15 | 25 | 3 | 1 | 1 | 9 | 2 |
-| 2014.12.16 | 34 | 3 | 10 | 4 | 0 | 0 |
-| 2014.12.17 | 11 | 2 | 3 | 2 | 1 | 1 |
-| 2014.12.18 | 3 | 1 | 8 | 1 | 0 | 0 |
-| 2014.12.19 | 7 | 1 | 1 | 1 | 9 | 1 |
-| 2014.12.20 | 96 | 3 | 11 | 4 | 0 | 0 |
-| 2014.12.21 | 1 | 1 | 17 | 4 | 12 | 3 |
-| 2014.12.23 | 0 | 0 | 21 | 5 | 1 | 1 |
-| 2014.12.24 | 5 | 1 | 11 | 4 | 0 | 0 |
-| 2014.12.25 | 14 | 2 | 8 | 2 | 0 | 0 |
-| 2014.12.26 | 0 | 0 | 13 | 4 | 0 | 0 |
-| 2014.12.27 | 0 | 0 | 9 | 3 | 0 | 0 |
-| 2014.12.28 | 0 | 0 | 24 | 4 | 0 | 0 |
-| 2014.12.29 | 0 | 0 | 21 | 7 | 0 | 0 |
-| 2014.12.30 | 0 | 0 | 34 | 6 | 0 | 0 |
-| 2014.12.31 | 1 | 1 | 47 | 5 | 0 | 0 |
-| 2015.1.1 | 0 | 0 | 33 | 4 | 0 | 0 |
-| 2015.1.2 | 0 | 0 | 29 | 7 | 0 | 0 |
-| 2015.1.3 | 0 | 0 | 25 | 4 | 12 | 1 |
-| 2015.1.4 | 0 | 0 | 14 | 5 | 0 | 0 |
-| 2015.1.5 | 12 | 2 | 0 | 0 | 0 | 0 |
-
-
-#+tblname: idle_periods_mt
-| Start | End |
-|------------+------------|
-| 2014.9.2 | 2014.9.20 |
-| 2014.12.18 | 2014.12.31 |
-
-
-#+begin_src R :exports both :results output graphics :var daily=daily :var idle=idle_periods_mt :file /tmp/daily.png :width 600 :height 600
- library(reshape2)
- library(ggplot2)
- require(gridExtra)
-
- daily$Date <- as.Date(daily$Date, "%Y.%m.%d")
-
- data_long <- melt(daily, id.vars=c("Date"))
-
- idle$Start <- as.Date(idle$Start, "%Y.%m.%d")
- idle$End <- as.Date(idle$End, "%Y.%m.%d")
-
- ymax1=200
-
- p1 <- ggplot() +
- geom_area(data=data_long[data_long$variable %in% c("exos_scala","exos_python","exos_java"),],
- aes(x=Date, y=value, color=variable,fill=variable)) +
- ggtitle("Daily activity (exercises)") +
- geom_rect(data=idle,aes(xmin=Start, xmax=End, ymin=0, ymax=ymax1),alpha=.1,fill="red",color="blue") +
- theme(legend.justification=c(1,0), legend.position=c(1,.6)) +
- coord_cartesian(ylim = c(0,ymax1)) +
- ylab("Exercises (#)")
-
- ymax2 = 40
-
- p2 <- ggplot() +
- geom_area(data=data_long[data_long$variable %in% c("traces_scala","traces_python","traces_java"),],
- aes(x=Date, y=value, color=variable,fill=variable)) +
- ggtitle("Daily activity (users)") +
- geom_rect(data=idle,aes(xmin=Start, xmax=End, ymin=0, ymax=ymax2),alpha=.1,fill="red",color="blue") +
- theme(legend.justification=c(1,0), legend.position=c(1,.6)) +
- coord_cartesian(ylim = c(0,ymax2)) + ### zoom with ggplot
- ylab("Active Traces (#)")
- grid.arrange(p1, p2)
-#+end_src
-
-#+RESULTS:
-[[file:/tmp/daily.png]]
-
-Entered on [2015-01-07 mer. 16:27]
-
- [[file:/tmp/plm-iticse.org::*Data%20Analysis][Data Analysis]]
-** 2015-07 juillet
-*** 2015-07-31 vendredi
-**** LOESS :WP8:Stats:R:
-An involved lecture:
- http://web.as.uky.edu/statistics/users/pbreheny/621/F10/notes/11-4.pdf
-
-A few R examples illustrating the influence of bandwidth:
-- http://research.stowers-institute.org/efg/R/Statistics/loess.htm
-- http://www.duclert.org/Aide-memoire-R/Statistiques/Local-polynomial-fitting.php
-
-Entered on [2015-07-31 ven. 09:35]
-**** Harald Servat's Phd defense
-Comments:
-- I really enjoyed the very *clear presentation* of the document, of the
- related work, etc.
-- I particularly enjoyed the fact that *hypothesis are clearly stated*,
- which I think is the sign of a true *scientific approach* in term of
- methodology. I also think that moving to *continous approximations* as
- you tried by using Kriging or segmented linear regressions is an
- excellent idea.
-- Last, *thanks for giving me the opportunity* to think more carefully
- about the mathematical foundations of such tools and how they could
- make sense or not. It actually raised a lots of questions.
-
-Questions:
-- You did not hesitate to use of *elaborate statistical tools*. Such
- tools rely on probabilistic model and on of the interesting feature
- is that they allow two things:
- - Hypothesis testing
- - Confidence interval calculation
- Do you think it would be worth building on such features
-- You demonstrated that your clustering methodology could be applied
- to many use cases. Can tell me if you can think about situations
- where it would not apply.
-- 11: segmented linear regression seems more meaningful than krigging
- here. Is it really the case ? Kriging does an interpolation and
- maybe the nuggeting is unable to smooth things enough. But maybe the
- different phases detected by segmented regression are not that
- meaningful either ?
-- Reuse of previous analysis to capture better traces with a lower
- overhead ?
-
-
-- 15: multi-dimensionnal segmented linear regression ?
-- 28: you are annoyed because time is a random variable too. There is
- uncertainty on it, which is why the classical technique (krigging or
- segmented regression) do not apply
-- Machine learning for pointing out to situations where correlation
- make sense or not.
-- 39: shouldn't the compiler have been able to do such kind of
- optimization ?
-- 49: L1-cache based sampling allowed to detect when MPI was receiving
- a message
-
-
-Entered on [2015-07-31 ven. 10:58]
-** 2015-12 décembre
-*** 2015-12-22 mardi
-**** Programmation avec Clément: hexagone magique :Teaching:Python:
-***** Génération de permutations toute simple
-#+begin_src python :results output :exports both
-N = 5
-A = range(1,N)
-
-def generate(tab,i):
- if i>=len(tab):
- print(tab)
- else:
- for j in range(i,len(tab)):
- tab[i],tab[j] = tab[j],tab[i]
- generate(tab,i+1)
- tab[i],tab[j] = tab[j],tab[i]
-
-generate(A,0)
-#+end_src
-
-#+RESULTS:
-#+begin_example
-[1, 2, 3, 4]
-[1, 2, 4, 3]
-[1, 3, 2, 4]
-[1, 3, 4, 2]
-[1, 4, 3, 2]
-[1, 4, 2, 3]
-[2, 1, 3, 4]
-[2, 1, 4, 3]
-[2, 3, 1, 4]
-[2, 3, 4, 1]
-[2, 4, 3, 1]
-[2, 4, 1, 3]
-[3, 2, 1, 4]
-[3, 2, 4, 1]
-[3, 1, 2, 4]
-[3, 1, 4, 2]
-[3, 4, 1, 2]
-[3, 4, 2, 1]
-[4, 2, 3, 1]
-[4, 2, 1, 3]
-[4, 3, 2, 1]
-[4, 3, 1, 2]
-[4, 1, 3, 2]
-[4, 1, 2, 3]
-#+end_example
-
-***** Exploration comme un bourrin
-On représente l'hexagone par un tableau numéroté comme ceci:
-#+BEGIN_EXAMPLE
- 0 1 2
- 3 4 5 6
-7 8 9 10 11
- 12 13 14 15
- 17 18 19
-#+END_EXAMPLE
-
-#+begin_src python :results output :exports both :tangle /tmp/test_bourrin.py
-def check(tab):
- start = 0
- for r in [3,4,5,4]:
- if sum(tab[start:(start+r)])!=38:
- return False
- start = start + r
- for t in [[2,6,11],[1,5,10,15],[0,4,9,14,18],[3,8,13,17]]:
- if sum([tab[i] for i in t])!=38:
- return False
- for t in [[7,3,0],[1,4,8,12],[2,5,9,13,16],[6,10,14,17]]:
- if sum([tab[i] for i in t])!=38:
- return False
- return True
-
-def generate(tab,i):
- if i>=len(tab):
- if check(tab):
- print(tab)
- else:
- for j in range(i,len(tab)):
- tab[i],tab[j] = tab[j],tab[i]
- generate(tab,i+1)
- tab[i],tab[j] = tab[j],tab[i]
-
-generate([3, 17, 18, 19, 7, 1, 11, 16, 2, 5, 6, 9, 12, 4, 8, 14, 10, 13, 15],0)
-#+end_src
-
-Bon, sauf que ça va être monstrueusement long en fait. Sur ma machine:
-#+begin_src sh :results output :exports both
-cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq
-#+end_src
-
-#+RESULTS:
-: 3300000
-
-Donc, dans l'hypothèse, ultra optimiste où je serais capable de
-vérifier une permutation par cycle d'horloge, il me faudrait:
-#+begin_src R :results output :session :exports both
-factorial(19)/3300000/24/3600/365
-#+end_src
-
-#+RESULTS:
-: [1] 1168.891
-
-Plus de 1000 ans. Bon, la loi de Moore finira par nous aider mais pas
-des masses. :)
-
-***** Génération de permutation avec coupe au plus tôt
-De façon à éliminer les branches au plus tôt, on représente l'hexagone
-par un tableau numéroté comme ceci:
-#+BEGIN_EXAMPLE
- 0 1 2
- 11 12 13 3
-10 17 19 14 4
- 9 16 15 5
- 8 7 6
-#+END_EXAMPLE
-
-Et on remarque qu'on n'a de choix de branchement que pour 0, 1, 3, 5,
-7, 9, et 12. Tous les autres sont induits par les précédents.
-
-#+begin_src python :results output :exports both :tangle /tmp/test_rapide.py
-def assign(tab, i, x):
- if x in tab[i:len(tab)]:
- for j in range(i,len(tab)):
- if(tab[j]==x):
- tab[i],tab[j] = tab[j],tab[i]
- generate(tab,i+1)
- tab[i],tab[j] = tab[j],tab[i]
- return
-
-def generate(tab,i):
- # print(i)
- if i>=len(tab):
- print(tab)
- else:
- if i in [0,1,3,5,7,9,12]:
- for j in range(i,len(tab)):
- tab[i],tab[j] = tab[j],tab[i]
- generate(tab,i+1)
- tab[i],tab[j] = tab[j],tab[i]
- elif i in [2,4,6,8,10]:
- x = 38 - (tab[i-1]+tab[i-2])
- assign(tab,i,x)
- elif i==11:
- x = 38 - (tab[i-1]+tab[0])
- assign(tab,i,x)
- elif i==13:
- x = 38 - (tab[11]+tab[12]+tab[3])
- assign(tab,i,x)
- elif i==14:
- x = 38 - (tab[1]+tab[13]+tab[5])
- assign(tab,i,x)
- elif i==15:
- x = 38 - (tab[3]+tab[14]+tab[7])
- assign(tab,i,x)
- elif i==16:
- x = 38 - (tab[5]+tab[15]+tab[9])
- assign(tab,i,x)
- elif i==17:
- x = 38 - (tab[7]+tab[16]+tab[11])
- if x+tab[9]+tab[12]+tab[1]!=38:
- return
- assign(tab,i,x)
- elif i==18:
- if tab[10]+tab[17]+tab[18]+tab[14]+tab[4]==38 and \
- tab[0]+tab[12]+tab[18]+tab[15]+tab[6]==38 and \
- tab[2]+tab[13]+tab[18]+tab[16]+tab[8]==38:
- generate(tab,i+1)
-
-generate(range(1,20),0)
-#+end_src
-
-
-Et maintenant, combien de temps pour avoir la solution ?
-#+begin_src sh :results output :exports both
-cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
-cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq
-time python /tmp/test_rapide.py 2>&1
-#+end_src
-
-#+RESULTS:
-#+begin_example
-performance
-3300000
-[3, 17, 18, 11, 9, 14, 15, 13, 10, 12, 16, 19, 7, 1, 6, 8, 4, 2, 5]
-[3, 19, 16, 12, 10, 13, 15, 14, 9, 11, 18, 17, 7, 2, 4, 8, 6, 1, 5]
-[9, 11, 18, 17, 3, 19, 16, 12, 10, 13, 15, 14, 6, 1, 7, 2, 4, 8, 5]
-[9, 14, 15, 13, 10, 12, 16, 19, 3, 17, 18, 11, 6, 8, 4, 2, 7, 1, 5]
-[10, 12, 16, 19, 3, 17, 18, 11, 9, 14, 15, 13, 4, 2, 7, 1, 6, 8, 5]
-[10, 13, 15, 14, 9, 11, 18, 17, 3, 19, 16, 12, 4, 8, 6, 1, 7, 2, 5]
-[15, 13, 10, 12, 16, 19, 3, 17, 18, 11, 9, 14, 8, 4, 2, 7, 1, 6, 5]
-[15, 14, 9, 11, 18, 17, 3, 19, 16, 12, 10, 13, 8, 6, 1, 7, 2, 4, 5]
-[16, 12, 10, 13, 15, 14, 9, 11, 18, 17, 3, 19, 2, 4, 8, 6, 1, 7, 5]
-[16, 19, 3, 17, 18, 11, 9, 14, 15, 13, 10, 12, 2, 7, 1, 6, 8, 4, 5]
-[18, 11, 9, 14, 15, 13, 10, 12, 16, 19, 3, 17, 1, 6, 8, 4, 2, 7, 5]
-[18, 17, 3, 19, 16, 12, 10, 13, 15, 14, 9, 11, 1, 7, 2, 4, 8, 6, 5]
-1.25user 0.00system 0:01.26elapsed 99%CPU (0avgtext+0avgdata 6668maxresident)k
-0inputs+0outputs (0major+873minor)pagefaults 0swaps
-#+end_example
-
-***** Solution par d'autres personnes
-Finalement, un peu de google nous donne ça:
-
-http://codegolf.stackexchange.com/questions/6304/code-solution-for-the-magic-hexagon
-
-La solution en C++ est fondamentalement la même mais sans utilisation
-de récursion, i.e. en inlinant les 9 boucles et avec une macro pour
-aléger l'écriture. Elle trouve les mêmes solutions que nous mais 60
-fois plus vite:
-
-#+begin_src cpp :results output :exports both :tangle /tmp/test_cpp.cpp
-#include
-#define LOOP(V) for(int V=1;V<20;V++){if(m&1<&1
-#+end_src
-
-#+RESULTS:
-#+begin_example
-3 17 18 19 7 1 11 16 2 5 6 9 12 4 8 14 10 13 15
-3 19 16 17 7 2 12 18 1 5 4 10 11 6 8 13 9 14 15
-9 11 18 14 6 1 17 15 8 5 7 3 13 4 2 19 10 12 16
-9 14 15 11 6 8 13 18 1 5 4 10 17 7 2 12 3 19 16
-10 12 16 13 4 2 19 15 8 5 7 3 14 6 1 17 9 11 18
-10 13 15 12 4 8 14 16 2 5 6 9 19 7 1 11 3 17 18
-15 13 10 14 8 4 12 9 6 5 2 16 11 1 7 19 18 17 3
-15 14 9 13 8 6 11 10 4 5 1 18 12 2 7 17 16 19 3
-16 12 10 19 2 4 13 3 7 5 8 15 17 1 6 14 18 11 9
-16 19 3 12 2 7 17 10 4 5 1 18 13 8 6 11 15 14 9
-18 11 9 17 1 6 14 3 7 5 8 15 19 2 4 13 16 12 10
-18 17 3 11 1 7 19 9 6 5 2 16 14 8 4 12 15 13 10
-0.02user 0.00system 0:00.02elapsed 100%CPU (0avgtext+0avgdata 1284maxresident)k
-0inputs+0outputs (0major+64minor)pagefaults 0swaps
-#+end_example
-
-Entered on [2015-12-22 mar. 17:59]
-*** 2015-12-23 mercredi
-**** Parrot :HOME:
-Appel le 23/12/15 vers 17:35. No de dossier: 516 440
-* 2016
-** 2016-02 février
-*** 2016-02-17 mercredi
-**** Programmation pendu avec Rémi :Teaching:Python:ATTACH:
-:PROPERTIES:
-:Attachments: lst.txt
-:ID: 0032c718-137f-464c-ab82-5cd3b378a222
-:END:
-***** Code
-#+begin_src python :results output :exports both :tangle /tmp/pendu.py
-from random import *
-from sys import stdin
-
-def valide(mot):
- for l in mot:
- ok = 0
- for lp in ["a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z","-"]:
- if(l==lp): ok=1
- if(ok==0): return 0
- return 1
-
-def lit_dictionnaire(nom_fichier):
- f = open(nom_fichier, 'r')
- L = []
- for line in f:
- mot = line.rstrip()
- if(valide(mot)):
- L.append(mot)
-
- # print "Il y a " + str(len(L)) + " mots dans mon dictionnaire."
- # print "Le premier est '" + L[0] + "'."
- # print "Le dixieme est '" + L[9] + "'."
- return L
-
-def choisit_mot(dico):
- return dico[int(len(dico)*random())]
-
-def trouve(lettre,lettres_autorisees):
- for l in lettres_autorisees:
- if(lettre==l):
- return 1
- return 0
-
-def enleve(l,L):
- for i in range(0,len(L)):
- if(l==L[i]):
- return(L[:i]+L[(i+1):])
-
-def lit_lettre(lettres_autorisees):
- print(lettres_autorisees)
- input = stdin.readline().rstrip()
- while (len(input)!=1 or (trouve(input,lettres_autorisees)!=1) ):
- print trouve(input,lettres_autorisees)
- print "Imbecile! Donne moi UNE lettre et qui soit autorisee!"
- input = stdin.readline().rstrip()
- return input
-
-def remplace(mot_joueur, l, mot):
- # print ">>> ("+mot_joueur+","+l+","+mot+")"
- a_trouve = 0
- for i in range(0,len(mot)):
- if(mot[i] == l):
- # print "Youpi!!! j'ai trouve: "+mot[i]
- # print " "+mot_joueur
- mot_joueur = mot_joueur[:i] + l + mot_joueur[(i+1):]
- a_trouve = 1
- # print " "+mot_joueur
- return (mot_joueur,a_trouve)
-
-
-def motif_ok(mot,motif):
- if(len(mot)!=len(motif)):
- return 0
- for i in range(0,len(motif)):
- if(motif[i]!="#"):
- if(mot[i]!=motif[i]):
- return 0;
- return 1
-
-def lettres_exclues_ok(mot,lettres_exclues):
- for l in mot:
- for le in lettres_exclues:
- if l==le:
- return 0
- return 1
-
-def filtre(dictionnaire,motif,lettres_exclues):
- nouveau_dico = []
- for mot in dictionnaire:
- if(motif_ok(mot,motif) and lettres_exclues_ok(mot, lettres_exclues)):
- nouveau_dico.append(mot)
- return nouveau_dico
-
-def conseille_stupide(mots_possibles, lettres_possibles):
- return lettres_possibles[0]
-
-def lettre_dans_mot(l,mot):
- for lm in mot:
- if(lm==l):
- return 1
- return 0
-
-def conseille(mots_possibles, lettres_possibles):
- nombre_mots = len(mots_possibles)
-
- if nombre_mots==1:
- for l in lettres_possibles:
- if(lettre_dans_mot(l,mots_possibles[0])):
- return l
- def compte(l,mots_possibles):
- num = 0
- for mot in mots_possibles:
- if lettre_dans_mot(l,mot)==1:
- num=num+1
- return num
-
- nombre_mots_avec_la_bonne_lettre = []
- score = []
- for l in lettres_possibles:
- num = compte(l,mots_possibles)
- nombre_mots_avec_la_bonne_lettre.append(num)
- score.append(abs(num-nombre_mots/2.0))
-
- score_min = score[0]+.1
- i_min = 0
- for i in range(0,len(lettres_possibles)):
- if(score[i]freq_max):
- freq_max = frequence_lettre[i]
- i_max = i
- # print lettres_possibles
- # print nombre_mots
- # print nombre_mots_avec_la_bonne_lettre
- # print score
- # print i_min
- return lettres_possibles[i_max]
-
-def jeu(dictionnaire,mot,mode):
- mot_joueur = "#" * len(mot)
- for i in range(0,len(mot)):
- if mot[i]=="-": mot_joueur = mot_joueur[:i] + "-" + mot_joueur[(i+1):]
-
- lettres_autorisees = ["a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z"]
- lettres_exclues = []
- max_erreur=18
- erreur=0
- mots_possibles=dictionnaire
-
- # print mot
-
- while(mot_joueur != mot):
- if mode=="interactif": print mot_joueur + " | Nombre d'erreurs autorisees restant : " + str(max_erreur-erreur)
- mots_possibles = filtre(mots_possibles,mot_joueur,lettres_exclues)
- if mode=="interactif": print "Il reste " + str(len(mots_possibles)) + " mot(s)"
- if mode=="interactif": lettre_conseillee = conseille(mots_possibles,lettres_autorisees)
- if mode=="frequence": lettre_conseillee = conseille_freq(mots_possibles,lettres_autorisees)
- if mode=="dichotomie": lettre_conseillee = conseille(mots_possibles,lettres_autorisees)
-
- if mode=="interactif": print " Conseil: " + lettre_conseillee
- if mode=="interactif": lettre = lit_lettre(lettres_autorisees)
- else: lettre=lettre_conseillee
- (mot_joueur,a_trouve) = remplace(mot_joueur, lettre, mot)
- lettres_autorisees = enleve(lettre,lettres_autorisees)
- if(a_trouve==0):
- erreur += 1
- lettres_exclues.append(lettre)
- if(erreur==max_erreur):
- if mode=="interactif": print "Tu as perdu!!!!"
- if mode=="interactif": print "C'etait : " + mot
- return erreur
- if mode=="interactif": print mot_joueur + " | Nombre d'erreurs autorisees restant : " + str(max_erreur-erreur)
- if mode=="interactif": print "Bravo!!!!"
- return erreur
-
-
-def main():
- mon_dico = lit_dictionnaire("/home/alegrand/Hacking/boggle/Words.txt");
- while(1):
- mot = choisit_mot(mon_dico);
- jeu(mon_dico,mot,"interactif")
-
-def main2():
- mon_dico = lit_dictionnaire("/home/alegrand/Hacking/boggle/Words.txt");
- while(1):
- mot = choisit_mot(mon_dico);
- freq = jeu(mon_dico,mot,"frequence")
- dicho = jeu(mon_dico,mot,"dichotomie")
- print mot + " , " + str(freq) + " , " + str(dicho)
-
-main2()
-### Quelques constructions equivalentes
-
-# i=0
-# while(i<10):
-# print i
-# i=i+1
-#
-# for i in range(0,10):
-# print i
-
-# for i in range(0,len(liste)):
-# print liste[i]
-#
-# for mot in liste:
-# print mot
-
-#+end_src
-
-J'exécute et j'arrête au boût de 3 minutes
-#+begin_src sh :results output :exports both
-python pendu.py > lst.txt
-#+end_src
-
-#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 400 :height 400 :session
-library(ggplot2)
-df=read.csv("data/00/32c718-137f-464c-ab82-5cd3b378a222/lst.txt",strip.white=T,header=F)
-names(df)=c("mot","freq","dicho")
-ggplot(data=df,aes(x=freq,y=dicho)) + geom_point(alpha=.3)
-#+end_src
-
-#+RESULTS:
-[[file:/tmp/babel-9398SP8/figure9398caH.png]]
-
-
-#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 400 :height 400 :session
-ggplot(data=df,aes(x=freq-dicho)) + geom_histogram()
-#+end_src
-
-#+RESULTS:
-[[file:/tmp/babel-9398SP8/figure9398QDg.png]]
-
-
-#+begin_src R :results output :session :exports both
-summary(df)
-#+end_src
-
-#+RESULTS:
-: mot freq dicho
-: bionique : 2 Min. :0.000 Min. : 0.000
-: crevassait: 2 1st Qu.:0.000 1st Qu.: 1.000
-: pin : 2 Median :1.000 Median : 3.000
-: primevere : 2 Mean :1.296 Mean : 3.087
-: terrien : 2 3rd Qu.:2.000 3rd Qu.: 4.000
-: abattement: 1 Max. :8.000 Max. :11.000
-: (Other) :836
-
-#+begin_src R :results output :session :exports both
-X=df$freq-df$dicho
-summary(X)
-mean(X)
-err = sd(X)/sqrt(length(X))
-mean(X) - 2*err
-mean(X) + 2*err
-#+end_src
-
-#+RESULTS:
-: Min. 1st Qu. Median Mean 3rd Qu. Max.
-: -10.000 -3.000 -1.000 -1.791 0.000 7.000
-: [1] 2.304793
-: [1] 0.07919362
-***** Links
-http://web.stanford.edu/class/cs106l/handouts/assignment-2-evil-hangman.pdf
-http://www.sharkfeeder.com/hangman/
-http://blog.wolfram.com/2010/08/13/25-best-hangman-words/
-Entered on [2016-02-17 mer. 10:14]
-*** 2016-02-25 jeudi
-**** Hacking screenkey :Python:
-#+begin_src sh :session foo :results output :exports both
-diff -u /usr/share/pyshared/Screenkey/listenkdb.py_old /usr/share/pyshared/Screenkey/listenkdb.py
-#+end_src
-
-#+RESULTS:
-#+begin_example
---- /usr/share/pyshared/Screenkey/listenkdb.py_old 2016-03-07 09:40:13.271193249 +0100
-+++ /usr/share/pyshared/Screenkey/listenkdb.py 2016-03-07 09:42:41.216862924 +0100
-@@ -230,7 +230,14 @@
- mod = mod + "Alt+"
- if self.cmd_keys['super']:
- mod = mod + "Super+"
--
-+
-+ if self.cmd_keys['shift']:
-+ if (len(key_shift)==1) and not(ord(key_normal) in range(97,123)) and not(ord(key_shift) in range(33,126)):
-+ mod = mod + "Shift+"
-65000):
-+ mod = mod + "Shift+"
-print "---------"
-print key, key_shift, keysym
- if self.cmd_keys['shift']:
- key = key_shift
- if self.cmd_keys['capslock'] \
-#+end_example
-
-Entered on [2016-02-25 jeu. 11:19]
-
-** 2016-07 juillet
-*** 2016-07-19 mardi
-**** [[http://rmarkdown.rstudio.com/flexdashboard/][flexdashboard: Easy interactive dashboards for R]] :WP7:WP8:R:twitter:
-
-Entered on [2016-07-19 mar. 09:04]
-**** Steps toward reproducible research (Karl Broman) :WP8:twitter:R:
-- https://github.com/kbroman/Talk_ReproRes (slides)
- - Why
- - I'm sorry but I think you haven't used the right data.
- - The results in Table 1 don’t seem to correspond to those in
- Figure 2.
- - In what order do I run these scripts?
- - Where did we get this data file?
- - Why did I omit those samples?
- - How did I make that figure?
- - Important points
- - Organize your data & code
- #+BEGIN_QUOTE
- Your closest collaborator is you six months ago,
- but you don’t reply to emails.
- (paraphrasing Mark Holder)
- #+END_QUOTE
- - Everything with a script
- If you do something once, you’ll do it 1000 times.
- - Automate the process as much as you can
- In addition to automating a complex process, it also documents
- the process, including the dependencies among data files and
- scripts.
- - Turn scripts into reproducible reports
- - Use version control (git/GitHub)
- #+BEGIN_QUOTE
- The most important tool is the mindset,
- when starting, that the end product
- will be reproducible.
- – Keith Baggerly
- #+END_QUOTE
-
-- [[http://kbroman.org/steps2rr/][initial steps toward reproducible research]] (lecture)
-- https://github.com/kbroman/Tools4RR (Materials for a one-credit
- course on reproducible research)
-
-Entered on [2016-07-19 mar. 09:06]
-**** [[http://michaellevy.name/blog/teaching-r-to-200-students-in-a-week/][Teaching R to 200 students in a week • Michael Levy]] :Teaching:R:
-Link from Michael Blum
-- Motivation precedes detail: “Here’s what you’re going to learn to do this week”
-- Live coding: shows that I make mistakes, builds in flexibility, forces to slow down
-- Live code piped to their browsers (dropbox? pad? floobits?)
-- In-class exercises instead of lectures
-- Stickies and good assistants
-- Daily feedback: Each day, I asked the students to fill out a quick
- survey: How well do you understand what was taught today, what’s
- working for you, and what could use a change?
-- Advanced exercises for experts.
-
-Entered on [2016-07-19 mar. 09:36]
- CEDEX 9
-* 2017
-** 2017-01 janvier
-*** 2017-01-09 lundi
-**** Ridge regression :Stats:R:
-http://web.as.uky.edu/statistics/users/pbreheny/764-F11/notes/9-1.pdf
-
-Ridge regression penalizes the size of the regression coefficients,
-which is convenient in the presence of multicollinearity
-
-- http://www.few.vu.nl/~wvanwie/Courses/HighdimensionalDataAnalysis/WNvanWieringen_HDDA_Lecture4_RidgeRegression_20162017.pdf
-- https://arxiv.org/pdf/1509.09169.pdf
-
-Entered on[2017-01-09 lun. 22:10]
-
-*** 2017-01-19 jeudi
-**** Groupe de lecture: [[file:public_html/readings/Jackson_Networks.pdf][Social and Economic Networks]] (session 1) :POLARIS:
-***** Chapter 1: Introduction
-- Example from the Medici graph
-- degree can be seen as a measure of the influence of the node but a
- probably more interesting notion is the betweenness which indicates
- the probability the path a pair of peole have to go through you to
- communicate:
-
- b(k) = 1/((n-1)(n-2)/2) \sum_{i\neq j} (number of shortest paths from i to j going through
- k)/(number ofr shortest paths from i to j)
-***** Chapter 2: Basic notions
-- notion of adjacency matrix with convenient properties.
-- deg(i) = \sum_j g_{i,j}
-- #triangles = tr(g^3)/6
-- Clustering: "how close are you from a clique"
- - local notion of connectivity
- - Cl_i(G) = (#of pairs j,k connected to i s.t. j and k are also
- connected )/(d(i).(d(i)-1)/2)
- = \sum_{j,k s.t. i\ne j\ne k} g_{i,j} g_{j,k} g_{k,i} / \sum_{j,k s.t. i\ne j\ne k} g_{i,j} g_{i,k}
- \approx (sum ith diag element of g^3) / (sum ith diag element of g^2)
- - Cl(G) = 1/N \sum_i Cl_i(G) is then the /average clustering/ of the graph
- if you average over nodes. If one consider a "star" of small
- "cliques", all the nodes but the one in the center will have a
- local clique of 1 (hence the average clustering tends to 1)
- whereas it is far of being a clique itself.
- - You could average over triples directly and get the /overall
- clustering/ by considering directly:
- \sum_{i\ne j\ne k} g_{i,j} g_{j,k} g_{k,i} / \sum_{i\ne j\ne k} g_{i,j} g_{i,k}
- This may be a better notion as the overall clustering would go to
- 0... But we could totally have the reverse...
-- Centrality:
- - degree centrality is the average degree
- - betweeness centrality is what we saw earlier
- - Eigen centrality. Relates to a notion of prestige P:
-
- P_i (g) = \sum_{j\ne i} g_{i,j} P_j(g)/\delta_g(j)
-
- Hence P = \bar{g}.P with \bar{g} being the normalized graph (that
- could be different if the graph is undirected or if one does not
- consider \delta to be the degree d). Note that the Degree satisfies
- such equation, hence you recove the degree centrality notion.
- - In general, one may consider the larger eigenvalue of \bar{g} and
- the Perron Frobenius tells that all components of the
- corresponding eigenvector are positive. One may thus normalize by
- the sum of the elements of the eigenvector P.
-- Another notion is distance or decay centrality:
- - \sum_j 1/d(i,j
-- We don't really know whether there is a link between
- Eigen-centrality and Betweenness centrality.
-**** MOOC, Jean-Marc Hasenfratz :Vulgarization:Teaching:WP8:
-Discussions sur l'opportunité de faire un MOOC Recherche reproductible
-sur FUN. On fait le tour des choses existantes:
-- Il y a des choses en stats, sur R
- https://www.fun-mooc.fr/cours/#filter/subject/mathematiques-et-statistiques?page=1&rpp=50
- mais rien sur les aspects recherche reproductible ou la
- programmation littérale en tant que tel.
-- Sur coursera, il y a
- https://www.coursera.org/specializations/jhu-data-science. Il me
- semble que les 5 premiers (intro/techno, R basis, data cleaning,
- EDA, litterate programming) valent le coup mais c'est trop long.
-- Lorena Barba's tutorial on RR ?
- https://barbagroup.github.io/essential_skills_RRC/
-- Si on reste sur l'idée que le "cahier de laboratoire" c'est la base,
- il faut décider de quelle techno on pousse: knitr, jupyter ou
- org-mode. D'où une discussion prévue avec Konrad et Christophe, ce
- qui permettrait en plus de donnée une dimension pluri-disciplinaire
- intéressante.
-
-Entered on [2017-01-19 jeu. 14:29]
-
-** 2017-12 décembre
-*** 2017-12-07 jeudi
-**** Refs Recherche reproductible intéressantes :WP8:
-[[https://www.jove.com/blog/2017/10/27/reproducibility-librarian-yes-that-should-be-your-next-job/][Reproducible research librarian]]
-
-- [[http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0038234][The Effects of FreeSurfer Version, Workstation Type, and Macintosh
- Operating System Version on Anatomical Volume and Cortical Thickness
- Measurements]]:
- - No differences were detected between repeated single runs nor
- between single runs and parallel runs on the same workstation and
- for the same FreeSurfer and OS version. For the same OS version,
- all Mac workstations produced identical results. However,
- differences were revealed between:
- - Mac and HP workstations
- - FreeSurfer versions v4.3.1, v4.5.0, and v5.0.0
- - OSX 10.5.8 and OSX 10.6.4/5
- - Focus sur l'importance des erreurs mais pas sur leur origine car
- tout est très closed-source.
-- [[https://genomebiology.biomedcentral.com/articles/10.1186/s13059-016-1044-7][Gene name errors are widespread in the scientific literature]]
-- [[https://genomebiology.biomedcentral.com/articles/10.1186/s13059-015-0850-7][Five selfish reasons to work reproducibly]]: style sympa
-- Jupyter extension with Reprozip: https://www.youtube.com/watch?v=Y8YmGVYHhS8
-- [[https://github.com/pantsbuild/pex][Pex]]: un environnement permettant de transformer un script python en
- un exécutable autonome (packing interne des libs).
-- [[http://o2r.info/][O2R]]: hyper préliminaire, aucun intérêt.
-
-Entered on [2017-12-07 jeu. 11:09]
-
- [[file:~/org/journal.org::*Autotuning%20context:][Autotuning context:]]
-**** [[file:~/Archives/Cours/maths/R/Verzani-SimpleR.pdf][simpleR – Using R for Introductory StatisticsR]] :Stats:R:
-
-Un document que je n'avais pas pris le temps de lire mais qui est pas
-mal fichu du tout. Points intéressants sur R:
-
-- Trouver des outliers en interagissant avec les plots:
- =identify(BUSH,BUCHANAN,n=2)=.
-- Using rlm or lqs for resistant regression. rlm minimise la somme
- d'une fraction des résidus et pas de l'intégralité.
-- pas mal d'exemples/exos sur le test (paired, avec même variance ou
- pas, Cox-Wilkinson, chi2, etc.)
-
-Entered on [2017-12-07 jeu. 12:28]
-
- [[file:~/org/journal.org::*Refs%20Recherche%20reproductible%20int%C3%A9ressantes][Refs Recherche reproductible intéressantes]]
-* 2018
-** 2018-10 octobre
-*** 2018-10-02 mardi
-**** Learning emacs lisp
-https://www.gnu.org/software/emacs/manual/html_mono/elisp.html
-
-file:///home/alegrand/tmp/Programming%20in%20Emacs%20Lisp%20-%20https:_www.gnu.org_software_emacs_manual_html_mono_eintr.html
-
-Also eval some emacs lisp using M-:
-***** Evaluation
-#+begin_src emacs-lisp
-(+ 2 2)
-#+end_src
-
-#+RESULTS:
-: 4
-
-#+begin_src emacs-lisp
-'(this is a quoted list) ;; a list
-#+end_src
-
-#+begin_src emacs-lisp :
-(this is a quoted list) ;; won't work as it will call "this" with args "is" "a" "quoted" "list"
-#+end_src
-***** Setting variables
-#+begin_src emacs-lisp
-(setq toto 2)
-toto
-#+end_src
-
-#+RESULTS:
-: 2
-
-
-#+begin_src emacs-lisp
-(setq toto 2)
-(set 'toto' 2)
-toto
-#+end_src
-
-#+RESULTS:
-: 2
-
-#+begin_src emacs-lisp
-(setq counter 0) ; Let's call this the initializer.
-(setq counter (+ counter 1)) ; This is the incrementer.
-counter ; This is the counter.
-#+end_src
-
-#+RESULTS:
-: 1
-
-#+begin_src emacs-lisp
-(let ((var1 2)
- (var2 3))
- (+ var1 var2))
-#+end_src
-
-#+RESULTS:
-: 5
-
-#+begin_src emacs-lisp
- (let ((zebra "stripes")
- (tiger "fierce"))
- (message "One kind of animal has %s and another is %s."
- zebra tiger))
-#+end_src
-
-#+RESULTS:
-: One kind of animal has stripes and another is fierce.
-
-***** Defining and calling function
-#+begin_src emacs-lisp
-(defun tutu() 2)
-(defun tutu() '(2))
-(tutu)
-#+end_src
-
-#+RESULTS:
-| 2 |
-
-#+begin_src emacs-lisp
-(functionp 'tutu)
-#+end_src
-
-#+RESULTS:
-: t
-
-***** Testing
-#+begin_src emacs-lisp
-(if (functionp 'tutu) (message "this is a function"))
-#+end_src
-
-#+RESULTS:
-: this is a function
-
-#+begin_src emacs-lisp
-(defun type-of-animal (characteristic)
- "Print message in echo area depending on CHARACTERISTIC.
- If the CHARACTERISTIC is the string \"fierce\",
- then warn of a tiger."
- (if (equal characteristic "fierce")
- (message "It is a tiger!")))
-(type-of-animal "fierce")
-;; (type-of-animal "striped")
-#+end_src
-
-#+RESULTS:
-: It is a tiger!
-
-#+begin_src emacs-lisp
-(if (> 4 5) ; if-part
- (message "4 falsely greater than 5!") ; then-part
- (message "4 is not greater than 5!")) ; else-part
-#+end_src
-
-#+RESULTS:
-: 4 is not greater than 5!
-
-***** Useful functions
-#+begin_src emacs-lisp
-;; describe-function
-;; describe-key
-;; list-matching-lines
-;; delete-window
-;; point-to-register
-;; eval-expression
-;; car, cdr, cons
-#+end_src
-***** Playing with babel templates (1)
-#+begin_src emacs-lisp
-;; (add-to-list 'org-structure-template-alist
-;; '("Y" "#+begin_src R\n?\n#+end_src"))
-;; (add-to-list 'org-structure-template-alist
-;; '("Y" '(tutu)))
-(setq tata "2")
-(add-to-list 'org-structure-template-alist
- '("Y" tata))
-#+end_src
-
-#+RESULTS:
-| Y | tata |
-| Y | toto |
-| Y | (quote (tutu)) |
-| Y | (tutu) |
-| Y | tutu |
-| Y | #+begin_src R |
-
-#+begin_src emacs-lisp
-(setq a (assoc "Y" org-structure-template-alist))
-a
-#+end_src
-
-#+RESULTS:
-| Y | tata |
-
-Unfortunately, when expending, the code checks whether the right value
-is a string (through th stringp function).
-
-[[file:~/Work/org-mode/lisp/org.el::(defun%20org-try-structure-completion%20()][This is where org-structure-template-alist is used in org-mode 9.0.5's code]].
-
-#+begin_src emacs-lisp
-(defun org-try-structure-completion ()
- "Try to complete a structure template before point.
-This looks for strings like \" @file Alya.f90
- !! @author Guillaume Houzeaux
- !! @brief Ayla main
-@@ -20,7 +21,13 @@
- use def_master, only : kfl_goblk
- use def_master, only : kfl_gocou
- use def_coupli, only : kfl_gozon
-+ use mod_parall, only : PAR_MY_WORLD_RANK
- implicit none
-+ INTEGER :: iter,ierror
-+ character*100 striter
-+ character*100 iterfile
-+ real :: tnow
-+ real, dimension(2) :: tarray
- !
- ! DLB should be disabled as we only wabnt to activate it for particular loops
- ! Master does not disble to lend its resources automatically
-@@ -39,6 +46,10 @@
-
- call Parall(22270_ip)
-
-+ write(iterfile,'(a,i4.4,a)') 'iterations-', PAR_MY_WORLD_RANK, '.csv'
-+ open(unit=2, file=iterfile)
-+
-+ iter = 1
- optimization: do while ( kfl_goopt == 1 )
-
- call Iniunk()
-@@ -49,6 +60,11 @@
- time: do while ( kfl_gotim == 1 )
-
- call Timste()
-+ call ETIME(tarray, tnow)
-+ write(2,*) tnow, tarray(1), tarray(2), PAR_MY_WORLD_RANK, iter
-+
-+ write(striter, '(a,i3.3)') 'iter',iter
-+ SCOREP_USER_REGION_BY_NAME_BEGIN(striter, SCOREP_USER_REGION_TYPE_PHASE)
-
- reset: do
- call Begste()
-@@ -77,6 +93,13 @@
-
- call Endste()
-
-+ SCOREP_USER_REGION_BY_NAME_END(striter)
-+ call ETIME(tarray, tnow)
-+ write(2,*) tnow, tarray(1), tarray(2), PAR_MY_WORLD_RANK, iter
-+ iter = iter + 1
-+
-+
-+
- call Filter(ITASK_ENDTIM)
- call Output(ITASK_ENDTIM)
-
-@@ -91,6 +114,9 @@
-
- end do optimization
-
-+ close(2)
-+
- call Turnof()
-
-+
- end program Alya
-Index: Sources/services/parall/par_prepro.f90
-===================================================================
---- Sources/services/parall/par_prepro.f90 (revision 9154)
-+++ Sources/services/parall/par_prepro.f90 (working copy)
-@@ -23,6 +23,7 @@
- use mod_redistribute, only : redistribute
- use mod_redistribute, only : gather_to_master
- use mod_parall, only : PAR_WORLD_SIZE
-+ use mod_parall, only : PAR_MY_WORLD_RANK
- use mod_parall, only : PAR_METIS4
- use mod_parall, only : PAR_SFC
- use mod_parall, only : PAR_ORIENTED_BIN
-@@ -39,6 +40,13 @@
- integer(ip) :: npoin_tmp, nelem_tmp, nboun_tmp,ndims
- real(rp) :: time1,time2,time3,time4,time5
- character(100) :: messa_integ
-+ ! LUCAS
-+ character*100 dfile
-+ integer(ip) ii
-+ ! END LUCAS
-+
-+
-+
- !
- ! Intermediate variables for partitioning
- !
-@@ -384,4 +392,19 @@
- end if
-
-
-+ ! LUCAS
-+ write(dfile,'(a,i4.4,a)') 'domain-', PAR_MY_WORLD_RANK, '.csv'
-+ open(unit=12345, file=dfile)
-+
-+ write(12345,*) "T1", PAR_MY_WORLD_RANK, nelem, npoin, nboun, npoi3-npoi2
-+ do ii = 1, nelem
-+ write(12345,*) "T2", PAR_MY_WORLD_RANK, ii, ltype(ii)
-+ end do
-+ do ii = 1, nboun
-+ write(12345,*) "T3", PAR_MY_WORLD_RANK, ii, ltypb(ii)
-+ end do
-+
-+ close(12345)
-+ ! END LUCAS
-+
- end subroutine par_prepro
-#+end_example
-
-**** Compilation
-
-In the =./ThirdParties/metis-4.0/=, compile Metis:
-
-#+begin_src shell :results output
-cd ./ThirdParties/metis-4.0/
-make
-#+end_src
-
-In the =./Executables/unix/= Alya directory, do.
-
-Copy =./configure.in/config_gfortran.in= to config.in.
-
-Make it look like this (in the begginning):
-
-#+BEGIN_EXAMPLE
-SCOREP = ~/install/scorep-4.0/bin/scorep
-MPIF90 = ~//spack-ALYA/opt/spack/linux-debian9-x86_64/gcc-6.3.0/openmpi-3.0.0-a7g33v4ulwtb4g2verliyelvtifybrq3/bin/mpif90
-MPICC = ~//spack-ALYA/opt/spack/linux-debian9-x86_64/gcc-6.3.0/openmpi-3.0.0-a7g33v4ulwtb4g2verliyelvtifybrq3/bin/mpicc
-F77 = $(SCOREP) --user --nocompiler --nopomp --noopenmp $(MPIF90) -cpp
-F90 = $(SCOREP) --user --nocompiler --nopomp --noopenmp $(MPIF90) -cpp
-FCOCC = $(SCOREP) --user --nocompiler --nopomp --noopenmp $(MPICC) -c
-#+END_EXAMPLE
-
-Then, configure the compilation
-
-#+begin_src shell :results output
-./configure nastin parall
-make -j 48
-#+end_src
-
-The compilation fails with
-
-#+BEGIN_EXAMPLE
-/gpfs/home/bsc21/bsc21835/alya-bsc-sfc/Sources/kernel/coupli/mod_commdom_alya.f90:489:2:
-
- CPLNG%sendrecv(1,6) = (current_task==ITASK_BEGSTE).and.( current_when==ITASK_BEFORE).and.& ! \
- 1
-Error: Unclassifiable statement at (1)
-#+END_EXAMPLE
-
-The solution is to comment those lines (and =make= again).
-
-The compilation fails with =par_prepro.f90=, just comment (or remove)
-the line that has =use mod_par_partit_sfc,=.
-
-*** R
-
-Installed in the image.
-
-** Jequi
-* Scripts
-** Get info from the platform
-*** Get machine information before doing the experiment
-#+begin_src sh :results output :tangle scripts/get_info.sh :tangle-mode (identity #o755)
-#!/bin/bash
-# Script for to get machine information before doing the experiment
-
-set +e # Don't fail fast since some information is maybe not available
-
-title="Experiment results"
-starpu_build=""
-inputfile=""
-host="$(hostname | sed 's/[0-9]*//g' | cut -d'.' -f1)"
-help_script()
-{
- cat << EOF
-Usage: $0 [options] outputfile.org
-
-Script for to get machine information before doing the experiment
-
-OPTIONS:
- -h Show this message
- -t Title of the output file
- -s Path to the StarPU installation
- -i Input file name if doing SimGrid simulation based on input
-EOF
-}
-# Parsing options
-while getopts "t:s:i:h" opt; do
- case $opt in
- t)
- title="$OPTARG"
- ;;
- s)
- starpu_build="$OPTARG"
- ;;
- i)
- inputfile="$OPTARG"
- ;;
- h)
- help_script
- exit 4
- ;;
- \?)
- echo "Invalid option: -$OPTARG"
- help_script
- exit 3
- ;;
- esac
-done
-
-shift $((OPTIND - 1))
-filedat=$1
-if [[ $# != 1 ]]; then
- echo 'ERROR!'
- help_script
- exit 2
-fi
-
-##################################################
-# Preambule of the output file
-echo "#+TITLE: $title" >> $filedat
-echo "#+DATE: $(eval date)" >> $filedat
-echo "#+AUTHOR: $(eval whoami)" >> $filedat
-echo "#+MACHINE: $(eval hostname)" >> $filedat
-echo "#+FILE: $(eval basename $filedat)" >> $filedat
-if [[ -n "$inputfile" ]];
-then
- echo "#+INPUTFILE: $inputfile" >> $filedat
-fi
-echo " " >> $filedat
-
-##################################################
-# Collecting metadata
-echo "* MACHINE INFO:" >> $filedat
-
-echo "** PEOPLE LOGGED WHEN EXPERIMENT STARTED:" >> $filedat
-who >> $filedat
-echo "############################################" >> $filedat
-
-echo "** ENVIRONMENT VARIABLES:" >> $filedat
-env >> $filedat
-echo "############################################" >> $filedat
-
-echo "** HOSTNAME:" >> $filedat
-hostname >> $filedat
-echo "############################################" >> $filedat
-
-if [[ -n $(command -v lstopo) ]];
-then
- echo "** MEMORY HIERARCHY:" >> $filedat
- lstopo --of console >> $filedat
- echo "############################################" >> $filedat
-fi
-
-if [[ -n "$starpu_build" ]];
-then
- echo "** STARPU MACHINE DISPLAY:" >> $filedat
- $starpu_build/bin/starpu_machine_display 1> tmp 2> /dev/null
- cat tmp >> $filedat
- rm -f tmp
- echo "############################################" >> $filedat
-fi
-
-if [ -f /proc/cpuinfo ];
-then
- echo "** CPU INFO:" >> $filedat
- cat /proc/cpuinfo >> $filedat
- echo "############################################" >> $filedat
-fi
-
-if [ -f /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor ];
-then
- echo "** CPU GOVERNOR:" >> $filedat
- ONLINECPUS=$(for CPU in $(find /sys/devices/system/cpu/ | grep cpu[0-9]*$); do [[ $(cat $CPU/online) -eq 1 ]] && echo $CPU; done | grep cpu[0-9]*$ | sed 's/.*cpu//')
- for PU in ${ONLINECPUS}; do
- echo -n "CPU frequency for cpu${PU}: " >> $filedat
- cat /sys/devices/system/cpu/cpu${PU}/cpufreq/scaling_governor >> $filedat
- done
- echo "############################################" >> $filedat
-fi
-
-if [ -f /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq ];
-then
- echo "** CPU FREQUENCY:" >> $filedat
- ONLINECPUS=$(for CPU in $(find /sys/devices/system/cpu/ | grep cpu[0-9]*$); do [[ $(cat $CPU/online) -eq 1 ]] && echo $CPU; done | grep cpu[0-9]*$ | sed 's/.*cpu//')
- for PU in ${ONLINECPUS}; do
- echo -n "CPU frequency for cpu${PU}: " >> $filedat
- cat /sys/devices/system/cpu/cpu${PU}/cpufreq/scaling_cur_freq >> $filedat
- done
- echo "############################################" >> $filedat
-fi
-
-if [ -f /usr/bin/cpufreq-info ];
-then
- echo "** CPUFREQ_INFO" >> $filedat
- cpufreq-info >> $filedat
- echo "############################################" >> $filedat
-fi
-
-if [ -f /usr/bin/lspci ];
-then
- echo "** LSPCI" >> $filedat
- lspci >> $filedat
- echo "############################################" >> $filedat
-fi
-
-if [ -f /usr/bin/ompi_info ];
-then
- echo "** OMPI_INFO" >> $filedat
- ompi_info --all >> $filedat
- echo "############################################" >> $filedat
-fi
-
-if [ -f /sbin/ifconfig ];
-then
- echo "** IFCONFIG" >> $filedat
- /sbin/ifconfig >> $filedat
- echo "############################################" >> $filedat
-fi
-
-if [[ -n $(command -v nvidia-smi) ]];
-then
- echo "** GPU INFO FROM NVIDIA-SMI:" >> $filedat
- nvidia-smi -q >> $filedat
- echo "############################################" >> $filedat
-fi
-
-if [ -f /proc/version ];
-then
- echo "** LINUX AND GCC VERSIONS:" >> $filedat
- cat /proc/version >> $filedat
- echo "############################################" >> $filedat
-fi
-
-if [[ -n $(command -v module) ]];
-then
- echo "** MODULES:" >> $filedat
- module list 2>> $filedat
- echo "############################################" >> $filedat
-fi
-#+end_src
-
-** Common Scripts
-*** Colors + error report
-#+begin_src sh :results output :tangle scripts/colors_error_report.sh :tangle-mode (identity #o755)
-#!/bin/bash
-set -euo pipefail
-
-# Error function
-function error_script {
- ERRORMSG=${1:-}
- echo "Error: $ERRORMSG"
- exit 1
-}
-
-# Check functions
-function check_var {
- VAR=${1:-}
- MSG=${2:-}
- if [[ -z "$VAR" ]]; then
- error_script $MSG
- fi
-}
-
-function check_bin {
- BIN=${1:-}
- MSG=${2:-}
- if [[ ! -x "$BIN" ]]; then
- error_script $MSG
- fi
-}
-
-function check_env {
- ENV=${1:-}
- MSG=${2:-}
- if [ -z ${ENV+x} ]; then
- error_script $MSG
- fi
-}
-
-function info {
- INFOMSG=${1:-}
- echo "Info: $INFOMSG"
-}
-#+end_src
-
-*** Create experimental directory (only functions)
-#+begin_src sh :results output :tangle scripts/create_experiment_dir.sh :tangle-mode (identity #o755)
-#!/bin/bash
-
-# fail often, so we are robust
-set -euo pipefail
-
-# source functions
-DIR=$(dirname $0)
-source $DIR/colors_error_report.sh
-
-# Function to cat a file within an example block
-function org_example_file {
- FILE=${1:-}
- if [ -e "$FILE" ]; then
- echo "#+BEGIN_EXAMPLE"
- cat $FILE
- echo "#+END_EXAMPLE"
- fi
-}
-
-# Function that returns the unique org filename given an experiment unique identifier
-function org_err_file {
- UNIQUE=${1:-}
- if [ -z "$UNIQUE" ]; then
- error_script "Error: Unique key not passed as parameter to org_err_file."
- fi
- echo $(basename $UNIQUE .org)_err.org
-}
-
-# Function that returns the unique org filename given an experiment unique identifier
-function org_file {
- UNIQUE=${1:-}
- if [ -z "$UNIQUE" ]; then
- error_script "Error: Unique key not passed as parameter to org_file."
- fi
- echo $(basename $UNIQUE .org).org
-}
-
-function org_info_file {
- UNIQUE=${1:-}
- if [ -z "$UNIQUE" ]; then
- error_script "Error: Unique key not passed as parameter to org_file."
- fi
- echo $(basename $UNIQUE .org)_get_info.org
-}
-
-# Function that returns the unique directory name given an experiment unique identifier
-function dir_file {
- UNIQUE=${1:-}
- if [ -z "$UNIQUE" ]; then
- error_script "Error: Unique key not passed as parameter to dir_file."
- fi
- echo $(basename $UNIQUE .org).dir
-}
-
-function create_experiment_dir {
- UNIQUE=${1:-}
-
- if [ -z "$UNIQUE" ]; then
- error_script "Error: Unique key not passed as parameter to create_experiment_directory."
- fi
-
- # Preparation of the output (org and dir)
- OUTPUTORG=$(org_file $UNIQUE)
- OUTPUTORGDIR=$(dir_file $UNIQUE)
- if [ -e $OUTPUTORG ]; then
- error_script "$OUTPUTORG already exists. Please, remove it or use another unique name."
- usage;
- exit;
- else
- info "Experiment Org file: $OUTPUTORG"
- fi
- if [ -d $OUTPUTORGDIR ]; then
- error_script "$OUTPUTORGDIR directory already exists. Please, remove it or use another unique name."
- usage;
- exit;
- else
- info "Experiment Directory: $OUTPUTORGDIR"
- fi
- # Everything seems okay, let's prepare the terrain
- touch $OUTPUTORG
- if [ $? -ne 0 ]; then
- error_script "Creation of $OUTPUTORG did not work."
- exit;
- fi
- mkdir $OUTPUTORGDIR
- if [ $? -ne 0 ]; then
- error_script "Creation of $OUTPUTORGDIR did not work."
- exit;
- fi
-}
-#+end_src
-*** Detect Hyperthreading
-Inspired by:
-http://unix.stackexchange.com/questions/33450/checking-if-hyperthreading-is-enabled-or-not/33509#33509
-#+begin_src R :results output :session :exports both :tangle scripts/hyperthreading.sh :tangle-mode (identity #o755)
-#!/bin/bash
-CPUFILE=/proc/cpuinfo
-test -f $CPUFILE || exit 1
-NUMPHYCPU=`grep "physical id" $CPUFILE | sort -u | wc -l`
-NUMLOGCORE=`grep "processor" $CPUFILE | wc -l`
-NUMPHYCORE=`grep "core id" $CPUFILE | sort -u | wc -l`
-TOTALNUMPHYCORE=$(echo "$NUMPHYCPU * $NUMPHYCORE" | bc)
-MODEL=`grep "model name" $CPUFILE | sort -u | cut -d : -f 2- | sed "s/^[[:space:]]*//"`
-echo "This system has $NUMPHYCPU CPUs, of model \"$MODEL\"."
-echo "Each physical CPU is equipped with $NUMPHYCORE physical cores (total is $TOTALNUMPHYCORE)."
-if [ $TOTALNUMPHYCORE -ne $NUMLOGCORE ]; then
- echo "Hyperthreading is ON. So, there are $NUMLOGCORE logical cores."
-else
- echo "Hyperthreading is OFF."
-fi
-exit
-#+end_src
-*** Disable Hyperthreading
-#+begin_src sh :results output :tangle scripts/disable_hyperthreading.sh :tangle-mode (identity #o755)
-#!/bin/bash
-DIR=$(dirname $0)
-
-#First, enable all cores
-for PU in `find /sys/devices/system/cpu/ |grep cpu[0-9]*$`; do
- echo "Enabling $PU now."
- sudo zsh -c "echo 1 > ${PU}/online"
-done
-
-HYPERTHREADING=`$DIR/hyperthreading.sh | grep -e "Hyperthreading is ON" | wc -l`
-if [ $HYPERTHREADING -eq 0 ]; then
- echo "Hyperthreading is OFF, so disabling is not necessary."
- exit
-else
- echo "Hyperthreading is ON."
-fi
-echo "The number of PUs now is $(hwloc-ls --only PU | wc -l)."
-echo "I will disable hyperthreading now."
-# Disable hyperthreading
-# Only run this if you are sure
-# - Hyperthreading is enabled
-# - Each physical core has two processing units (PU)
-# - hwloc-ls is installed and reports two PU per physical core
-for PU in `hwloc-ls --only PU | cat -n | grep -e "[[:space:]]*[0-9]*[02468][[:space:]]*PU" | sed -e "s/^[^(]*(P#\\([0-9]*\))/\1/"`; do
- echo "Disabling PU $PU now."
- sudo zsh -c "echo 0 > /sys/devices/system/cpu/cpu${PU}/online"
-done
-echo "The number of PUs now is $(hwloc-ls --only PU | wc -l)."
-#+end_src
-*** Disable TurboBoost
-#+begin_src sh :results output :tangle scripts/disable_turboboost.sh :tangle-mode (identity #o755)
-#!/bin/bash
-DIR=$(dirname $0)
-
-if [ `lsmod | grep msr | wc -l` -ne 1 ]; then
- echo "The =msr= module is not loaded. It should be."
- exit 1;
-fi
-
-# Get the list of online cores
-ONLINECPUS=$(for CPU in $(find /sys/devices/system/cpu/ | grep -v cpu0 | grep cpu[0-9]*$); do [[ $(cat $CPU/online) -eq 1 ]] && echo $CPU; done | grep cpu[0-9]*$ | sed 's/.*cpu//')
-
-# Enable
-for PU in ${ONLINECPUS}; do
- sudo zsh -c "/usr/sbin/wrmsr -p${PU} 0x1a0 0x850089"
-done
-
-# Disable & Check
-for PU in ${ONLINECPUS}; do
- echo "Disabling turbo boost mode for PU $PU."
- sudo zsh -c "/usr/sbin/wrmsr -p${PU} 0x1a0 0x4000850089"
- TURBOBOOST=$(sudo zsh -c "/usr/sbin/rdmsr -p${PU} 0x1a0 -f 38:38")
- if [[ "0" = $TURBOBOOST ]]; then
- echo "Failed to disable turbo boost for PU number $cpu. Aborting."
- exit 1
- fi
-done
-#+end_src
-*** CPU Selection
-#+begin_src sh :results output :tangle scripts/cpu_selection.sh :tangle-mode (identity #o755)
-#!/bin/bash
-DIR=$(dirname $0)
-
-function usage()
-{
- echo "Input: number of CPUs to be used"
- echo "Output: core identifiers (NUMA-aware)"
- echo "$0 ";
-}
-
-NCPUS=${1:-}
-if [ -z "$NCPUS" ]; then
- echo "Error: is empty"
- usage;
- exit;
-fi
-
-# Check total number of CPUs
-NUMCPU=$($DIR/hyperthreading.sh | head -n1 | sed 's/^This system has \([0-9]*\) CPUs.*$/\1/')
-if [ $NCPUS -gt $NUMCPU ]; then
- echo "You request $NCPUS CPUs, but this system has only $NUMCPU. Sorry."
- exit
-fi
-
-RES=$(lscpu | grep "NUMA\ node[0-9]* " | sed "s/^.*:[[:space:]]*//" | head -n${NCPUS} | tr '\n' ' ' | sed "s/ $//")
-FINAL=$(for i in $RES; do
- if [ $(echo $i | grep "-" | wc -l) -eq 0 ]; then
- #just replace commas by spaces
- echo $i | sed "s/,/ /g"
- else
- #use seq to replace core ranges by actual core ids
- S=$(echo $i | sed "s/-.*$//");
- E=$(echo $i | sed "s/^.*-//");
- seq $S $E
- fi
-done | tr '\n' ' ')
-echo "Use these: $FINAL"
-#+end_src
-*** Detect DVFS driver (acpi-cpufreq)
-#+begin_src sh :results output :tangle scripts/detect_acpidriver.sh :tangle-mode (identity #o755)
-#!/bin/bash
-DIR=$(dirname $0)
-
-function usage()
-{
- echo "Input: number of CPUs to be used"
- echo "Output: core identifiers (NUMA-aware)"
- echo "$0 ";
-}
-
-PRESENT=$(cpufreq-info | grep driver | uniq | grep acpi-cpufreq | wc -l)
-if [ $PRESENT -ne 1 ]; then
- exit 1;
-fi
-
-exit 0
-#+end_src
-*** Set frequency to maximum
-#+begin_src sh :results output :tangle scripts/set_maximum_frequency.sh :tangle-mode (identity #o755)
-#!/bin/bash
-DIR=$(dirname $0)
-
-MAXFREQ=$(cpufreq-info | grep limits | sed -e "s/.*- //" -e "s/ //g" | uniq)
-
-# Get all online cores
-ONLINECPUS=$(for CPU in $(find /sys/devices/system/cpu/ | grep -v cpu0 | grep cpu[0-9]*$); do [[ $(cat $CPU/online) -eq 1 ]] && echo $CPU; done | grep cpu[0-9]*$ | sed 's/.*cpu//')
-
-# Core 0 is always online
-ONLINECPUS="0 ${ONLINECPUS}"
-
-for PU in ${ONLINECPUS}; do
- echo "Setting the frequency of PU ${PU} to ${MAXFREQ}"
- sudo cpufreq-set -c ${PU} -f ${MAXFREQ}
- echo "After setting to max, the frequency is now $(cat /sys/devices/system/cpu/cpu${PU}/cpufreq/scaling_cur_freq)"
-done
-#+end_src
-** Configure multi-node experimental settings
-A script to control the experiment.
-- [X] Input is a file with node names
-
-For each machine:
-- [X] Disable turboboost
-- [X] Disable hyperthreading
-- [X] The =acpi_cpufreq=
- - [X] Check for its use
- - [X] Fix processor frequency to maximum
-- [X] Call the getinfo script after everything
-
-#+begin_src sh :results output :tangle scripts/control_experiment.sh :tangle-mode (identity #o755)
-#!/bin/bash
-DIR=$(dirname $0)
-
-function usage()
-{
- echo "$0 [dest]";
- echo " where is a file with a list of hostnames, one per line";
- echo " where is a directory where captured files should be moved to";
-}
-
-##############################
-# Parameter handling section #
-##############################
-MACHINEFILE=${1:-}
-if [ -z "$MACHINEFILE" ]; then
- echo "Error: is empty"
- usage;
- exit;
-fi
-
-DEST=${2:-}
-if [ -z "$DEST" ]; then
- DEST="."
-fi
-
-##############################
-# For every host #
-##############################
-while IFS='' read -u10 -r HOST; do
- # Ignore comments and empty lines
- [[ "$(echo $HOST | sed 's/ //g')" =~ ^$ ]] && continue
- [[ "$HOST" =~ ^#.*$ ]] && continue
-
- # Clear anything after the colon
- HOST=$(echo $HOST | sed "s/:.*$//g")
-
- echo "** $HOST"
- echo "*** passwordless ssh"
-
- # Check is passwordless ssh is possible
- X=$(eval ssh -o 'PreferredAuthentications=publickey' ${HOST} "echo -n"; echo $?)
- if [ $X -ne 0 ]; then
- echo "passwordless ssh is not possible to ${HOST}."
- exit ;
- else
- echo "${HOST} is acessible through passwordless ssh."
- fi
-
- echo "*** Check presence of acpi-cpufreq driver"
-
- # Check for acpi-cpufreq
- scp $DIR/detect_acpidriver.sh ${HOST}:/tmp/
- PRESENT=$(eval ssh ${HOST} "/tmp/detect_acpidriver.sh ; echo $?")
- if [ $PRESENT -ne 0 ]; then
- echo "The driver acpi-cpufreq is not available in ${HOST}"
- exit
- else
- echo "I believe the ${HOST} is using the acpi-cpufreq DVFS driver."
- fi
-
- echo "*** Disable hyperthreading"
-
- # Disable hyperthreading
- scp $DIR/hyperthreading.sh ${HOST}:/tmp/
- scp $DIR/disable_hyperthreading.sh ${HOST}:/tmp/
- ssh ${HOST} /tmp/disable_hyperthreading.sh
-
- echo "*** Disable turboboost"
-
- # Disable turboboost
- scp $DIR/disable_turboboost.sh ${HOST}:/tmp/
- ssh ${HOST} /tmp/disable_turboboost.sh
-
- echo "*** Set the frequency to maximum"
-
- # Manually set the frequency to maximum
- scp $DIR/set_maximum_frequency.sh ${HOST}:/tmp/
- ssh ${HOST} /tmp/set_maximum_frequency.sh
-
- echo "*** Call the get_info.sh"
-
- # After everything, call the get_info.sh script
- scp $DIR/get_info.sh ${HOST}:/tmp/
- GETINFO=$(echo ${HOST} | sed "s/\./_/g" | sed "s/$/_get_info.org/")
- ssh ${HOST} "rm -f /tmp/${GETINFO}"
- ssh ${HOST} "/tmp/get_info.sh -t 'Get information from the ${HOST}' /tmp/${GETINFO}"
- scp ${HOST}:/tmp/$GETINFO .
- echo $GETINFO
- if [ "$DEST" != "." ]; then
- mv -f $GETINFO $DEST
- fi
-
-done 10< ${MACHINEFILE}
-#+end_src
-** Run an Alya experiment
-Input:
-- [X] The machine file with the list of reserved machines
-- [X] The number of MPI processes to launch
-- [X] Path to Alya binary
-- [X] Path to the DAT file
-
-Everything should be ran in =/tmp/= directory, outside of NFS. The
-script will pushd to the directory of the DAT file (that should be
-outside of NFS) and ran from there.
-
-#+begin_src sh :results output :tangle scripts/run_alya_experiment.sh :tangle-mode (identity #o755)
-#!/bin/bash
-DIR=$(dirname $0)
-
-source $DIR/create_experiment_dir.sh
-
-function usage()
-{
- echo "$0 ";
- echo " where is a file with a list of hostnames, one per line";
- echo " where is the number of MPI process";
- echo " where is the path to the Alya binary";
- echo " where is the path to the DAT file";
- echo " where is a comment without white spaces";
- echo " where (for infiniband) is true or false";
-}
-
-function now() { date +%s; }
-
-##############################
-# Parameter handling section #
-##############################
-MACHINEFILE=${1:-}
-if [ -z "$MACHINEFILE" ]; then
- echo "Error: is empty"
- usage;
- exit;
-fi
-MACHINEFILE=$(readlink -f $MACHINEFILE)
-
-if [ ! -e "$MACHINEFILE" ]; then
- echo "Error: '$MACHINEFILE' does not exists."
- usage;
- exit
-fi
-
-NP=${2:-}
-if [ -z "$NP" ]; then
- echo "Error: is empty"
- usage;
- exit;
-fi
-
-ALYA=${3:-}
-if [ -z "$ALYA" ]; then
- echo "Error: is empty"
- usage;
- exit;
-fi
-ALYA=$(readlink -f $ALYA)
-
-DAT=${4:-}
-if [ -z "$DAT" ]; then
- echo "Error: is empty"
- usage;
- exit;
-fi
-DAT=$(readlink -f $DAT)
-
-UNIQUE=${5:-}
-if [ -z "$UNIQUE" ]; then
- echo "Error: is empty"
- usage;
- exit;
-else
- # Make sure unique is without spaces
- UNIQUE=$(echo $UNIQUE | sed "s/ /_/g")
-fi
-
-IB=${6:-}
-if [ -z "$IB" ]; then
- echo "Error: is empty";
- exit;
-fi
-
-##############################
-# Create the directory #
-##############################
-# The org file of this script
-OUTPUTORG=$(org_file $UNIQUE)
-OUTPUTORGERR=$(org_err_file $UNIQUE)
-# The getinfo of where this script has been executed
-OUTPUTORGINFO=$(org_info_file $UNIQUE)
-# The place where results will be stored
-OUTPUTORGDIR=$(dir_file $UNIQUE)
-# Create the experimental directory
-create_experiment_dir $UNIQUE
-# Create the two org files
-touch $OUTPUTORG $OUTPUTORGINFO
-# From now on, all standard output goes to $OUTPUTORG
-exec > $OUTPUTORG
-exec 2> $OUTPUTORGERR
-
-DRYRUN=false
-
-##############################
-# Prepare the experiment #
-##############################
-echo "* Prepare the environment"
-
-# Call control the experiment
-if [ "$DRYRUN" = "false" ]; then
- $DIR/control_experiment.sh ${MACHINEFILE} ${OUTPUTORGDIR}
-else
- echo "This is a dry run, not calling the control experiment"
-fi
-
-# Save the machinefile forever
-cp $MACHINEFILE ${OUTPUTORGDIR}
-
-echo "* Alya"
-echo "** Binary"
-echo "The alya binary: $ALYA"
-ls -lh $ALYA
-# ldd is rather important
-ldd $ALYA
-# Heinrich told me he saves the binary, let's do the same
-cp $ALYA $OUTPUTORGDIR
-echo "** Input"
-
-echo "*** DAT"
-echo "The contents of the dat file:"
-ls -lh $DAT
-org_example_file $DAT
-DATDIR=$(dirname $DAT)
-DATBASE=$(basename $DAT .dat)
-echo "The DAT dir is ${DATDIR}, the DAT base is ${DATBASE}"
-
-echo "*** DOM.DAT"
-DOMDAT=${DATDIR}/${DATBASE}.dom.dat
-echo "The contents of the $DOMDAT"
-ls -lh $DOMDAT
-org_example_file $DOMDAT
-
-echo "*** KER.DAT"
-KERDAT=${DATDIR}/${DATBASE}.ker.dat
-echo "The contents of the $KERDAT"
-ls -lh $KERDAT
-org_example_file $KERDAT
-
-echo "*** NSI.DAT"
-NSIDAT=${DATDIR}/${DATBASE}.ker.dat
-echo "The contents of the $NSIDAT"
-ls -lh $NSIDAT
-org_example_file $NSIDAT
-
-if [ "$DRYRUN" = "false" ]; then
- $DIR/get_info.sh -t "Get information from launcher host $(hostname)" $OUTPUTORGINFO
-else
- echo "This is a dry run, not calling the get info script"
-fi
-
-##############################
-# Run the experiment #
-##############################
-echo "* ScoreP directory"
-SCOREPDIR="$HOME/scorep_${UNIQUE}"
-echo "$SCOREPDIR (finally moved to this experiment directory)"
-rm -rf $SCOREPDIR
-
-echo "* The Experiment"
-if [ "$IB" = "true" ]; then
- OPENIB=""
-else
- OPENIB="OMPI_MCA_btl=\"tcp,self,sm\""
-fi
-
-# btl_base_verbose is set to 40 "info"
-COMMAND="${OPENIB} mpirun --mca btl_base_verbose 40 -x SCOREP_MPI_ENABLE_GROUPS=no -x SCOREP_VERBOSE=true -x SCOREP_TOTAL_MEMORY=4G -x SCOREP_EXPERIMENT_DIRECTORY=${SCOREPDIR} -x SCOREP_OVERWRITE_EXPERIMENT_DIRECTORY=true -x SCOREP_ENABLE_TRACING=true --bind-to core --report-bindings -np $NP -machinefile $MACHINEFILE $ALYA ${DATDIR}/${DATBASE}"
-echo "The command to be executed: '$COMMAND'"
-echo "The epoch time now, before the execution, is: $(now)"
-echo "#+BEGIN_EXAMPLE"
-if [ "$DRYRUN" = "false" ]; then
- eval $COMMAND
-fi
-echo "#+END_EXAMPLE"
-echo "The epoch time now, after the execution, is: $(now)"
-
-##############################
-# Copy LOGs #
-##############################
-echo "* The Logs"
-ls -lh ${DATDIR}/*.log
-cp ${DATDIR}/*.log $OUTPUTORGDIR
-
-echo "* Copy other small produced data"
-ls -lh ${DATDIR}/*.cvg ${DATDIR}/*.msh ${DATDIR}/*.res ${DATDIR}/*.sol ${DATDIR}/*.set
-cp ${DATDIR}/*.cvg ${DATDIR}/*.msh ${DATDIR}/*.res ${DATDIR}/*.sol ${DATDIR}/*.set $OUTPUTORGDIR
-
-echo "* Copy results of our manual instrumentation of alya code"
-ls -lh results_NPOIN_NELEM_NELEW_NBOUN.log
-mv -f results_NPOIN_NELEM_NELEW_NBOUN.log $OUTPUTORGDIR
-
-##############################
-# Save in the directory #
-##############################
-# Move scorep directory to
-mv -f $SCOREPDIR $OUTPUTORGDIR
-
-# Move org file to directory
-mv -f $OUTPUTORG $OUTPUTORGDIR
-
-# Move org err file to directory
-mv -f $OUTPUTORGERR $OUTPUTORGDIR
-
-# Move get_info org to directory
-mv -f $OUTPUTORGINFO $OUTPUTORGDIR
-
-# Copy myself to directory
-cp -f $0 $OUTPUTORGDIR
-
-# Copy all scripts to directory
-cp -f $DIR/*.sh $OUTPUTORGDIR
-
-# Exit graciously
-exit 0
-#+end_src
-
-#+RESULTS:
-** Convert traces.otf2 to CSV (version with =pj_dump=)
-#+begin_src sh :results output :tangle scripts/otf22csv.sh :tangle-mode (identity #o755)
-#!/bin/bash
-DIR=$(dirname $0)
-
-otf22csv() {
- pushd $(dirname $otf2)
- otf22paje traces.otf2 | pj_dump | grep ^State | cut -d, -f2,4,5,8 | sed -e "s/ //g" -e "s/MPIRank//" | gzip > traces.csv.gz
- popd
-}
-
-function usage()
-{
- echo "$0 "
- echo " where is a directory that contains .otf2 files";
-}
-
-##############################
-# Parameter handling section #
-##############################
-OTF2DIR=${1:-}
-if [ -z "$OTF2DIR" ]; then
- echo "Error: is empty"
- usage;
- exit;
-fi
-OTF2DIR=$(readlink -f $OTF2DIR)
-
-if [ ! -d "$OTF2DIR" ]; then
- echo "Error: $OTF2DIR is not a directory"
- usage
- exit
-fi
-
-##############################
-# Check for necessary tools #
-##############################
-if [ -z "$(which pj_dump)" ]; then
- echo "Error: pj_dump is not in the PATH"
- exit
-fi
-if [ -z "$(which otf22paje)" ]; then
- echo "Error: otf22paje is not in the PATH"
- exit
-fi
-
-# Files already converted (whose CSV size is not zero)
-EXISTINGFILE=$(tempfile)
-OTF2FILE=$(tempfile)
-find $OTF2DIR -not -empty | grep csv.gz$ | sed -e "s/.csv.gz$//" | sort > $EXISTINGFILE
-find $OTF2DIR | grep \.otf2$ | sed -e "s/.otf2$//" | sort > $OTF2FILE
-
-for otf2 in $(comm -3 $OTF2FILE $EXISTINGFILE | sed "s/$/.otf2/"); do
- echo $otf2
- otf22csv $otf2
-done
-#+end_src
-** Convert traces.otf2 to CSV (version *without* =pj_dump=, only *otf22csv*)
-#+begin_src sh :results output :tangle scripts/otf22csv_faster.sh :tangle-mode (identity #o755)
-#!/bin/bash
-DIR=$(dirname $0)
-
-otf22csv_faster() {
- pushd $(dirname $otf2)
- otf22csv traces.otf2 | gzip > traces.csv.gz
- popd
-}
-
-function usage()
-{
- echo "$0 "
- echo " where is a directory that contains .otf2 files";
-}
-
-##############################
-# Parameter handling section #
-##############################
-OTF2DIR=${1:-}
-if [ -z "$OTF2DIR" ]; then
- echo "Error: is empty"
- usage;
- exit;
-fi
-OTF2DIR=$(readlink -f $OTF2DIR)
-
-if [ ! -d "$OTF2DIR" ]; then
- echo "Error: $OTF2DIR is not a directory"
- usage
- exit
-fi
-
-##############################
-# Check for necessary tools #
-##############################
-if [ -z "$(which otf22csv)" ]; then
- echo "Error: otf22csv is not in the PATH"
- exit
-fi
-
-# Files already converted (whose CSV size is not zero)
-EXISTINGFILE=$(tempfile)
-OTF2FILE=$(tempfile)
-find $OTF2DIR -not -empty | grep csv.gz$ | sed -e "s/.csv.gz$//" | sort > $EXISTINGFILE
-find $OTF2DIR | grep \.otf2$ | sed -e "s/.otf2$//" | sort > $OTF2FILE
-
-for otf2 in $(comm -3 $OTF2FILE $EXISTINGFILE | sed "s/$/.otf2/"); do
- echo $otf2
- otf22csv_faster $otf2
-done
-#+end_src
-** Post-processing in R
-#+begin_src R :results output :session :exports both :tangle scripts/trace2summary.R :tangle-mode (identity #o755)
-#!/usr/bin/Rscript
-library(readr);
-library(dplyr);
-alya_scorep_trace_read <- function(filename)
-{
- meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[3], "_"));
- read_csv(filename,
- col_names=c("Rank", "Start", "End", "Value"),
- progress=TRUE) %>%
- # Transform Value to factor
- mutate(Value = as.factor(Value)) %>%
- # Detect begin and end of iterations
- mutate(Iteration = case_when(
- grepl("timste", .$Value) ~ 1,
- grepl("endste", .$Value) ~ -1,
- TRUE ~ 0)) %>%
- group_by(Rank) %>%
- # Create a logical to detect observations within iterations
- mutate(Iteration = as.logical(cumsum(Iteration))) %>%
- # Get only observations that belongs to some iteration
- filter(Iteration == TRUE) %>%
- ungroup() %>%
- # Create the iteration by cumsum
- mutate(Iteration = case_when(
- grepl("timste", .$Value) ~ 1,
- TRUE ~ 0)) %>%
- group_by(Rank) %>%
- mutate(Iteration = as.integer(cumsum(Iteration))) %>%
- ungroup() %>%
- # Define metadata
- mutate(EID = meta[2],
- Platform = meta[3],
- Nodes = meta[4],
- NP = meta[5],
- Partitioning = meta[6],
- Infiniband = as.logical(meta[7]));
-}
-
-alya_scorep_trace_iterations <- function(filename)
-{
- alya_scorep_trace_read(filename) %>%
- group_by(Rank, Iteration, Platform, Nodes, NP, Partitioning, EID, Infiniband) %>%
- filter(grepl("MPI_", Value)) %>%
- summarize(N=n(), S=min(Start), E=max(End), Comm=sum(End-Start), Comp=(E-S)-Comm);
-}
-
-args = commandArgs(trailingOnly=TRUE)
-if (length(args) < 1) {
- stop("Usage: trace2summary.R ", call.=FALSE)
-}
-
-#
-# parse arguments
-#
-outputFilename = args[[1]];
-inputs = unlist(args[2:length(args)]);
-
-df <- do.call("rbind", lapply(inputs, function(x) { alya_scorep_trace_iterations(x) }));
-write.csv(df, outputFilename);
-#+end_src
-* Guilherme entries
-* Meeting
-** 2018-04-18 Arnaud and Lucas
-
-At http://rendez-vous.renater.fr/alyalbsfc
-
-Rick's goal: 10K cores in MN4
-
-Eeach iteration has: Assemblage and Solver
-
-Rick has made a perfect LB in the Assembly phase only.
-
-The idea now is to make him repeat our observation methodology.
-
-** 2018-04-18 Paper Outline (Discussion @ Barcelona w. Arnaud and Ricard)
-*** General information
-- Target: Physics journal/conf, multi-physics simulation
- developers. Note that I think this kind of work would totally make
- sense in CCPE or JPDC.
-- Thoughts about our work:
- - Nothing specific to Alya
- - Load balance in-depth study and mitigation is interesting as
- everyone relies on METIS or SFC.
- - Profiling technique not new for HPC performance experts but
- interesting for physics simulation developers.
- - 1D load balancing strategy not new from an algorithmic perspective
- - Refinement process expected to work but could have failed as
- well...
- - Doing a good load estimation and a good load balance in
- practice on a real code is not that easy common.
-- Goal: explain how this can be done as it could apply to many other
- codes
-*** Outline
-**** Intro
-**** Context :Guillaume:Ricard:
- - Alya structure
- - 3 typical test cases)
- - respiratory (up to ~1 K cpus - ~25M elem. mesh)
- - combustor (up to ~3K cpus - ~50M elem. mesh)
- - plane (up to ~6K cpus - 160M elem. mesh)
- - Scalability not satifying regardless of the partitioner in use
- (METIS, SFC).
-**** In-depth Load Analysis of one test case (mostly :Lucas:Arnaud:
- - Tracing is intrusive and cumbersome but an informed profiling with
- e.g., Score-P is cheap and requires almost no modification of the
- application.
- - Illustrate the load imbalance and regularity of the application
- across iterations. Illustrate the fact that the load imbalance is
- different for assembly and for the sparse solver.
- - Explain the reason behind the bad load balance (e.g., with SFC but
- this does not depend on the partitioner). There is no magic
- formula relating geometric characteristics with the actual
- execution time.
-**** Proposal: iterative improvement of the load distribution on an SFC :Lucas:Arnaud:Ricard:
- parition based on a few iterations of the application.
- - Illustration of the quick improvements of the load balance
-**** Evaluation of the technique on the three test cases at different scales :Lucas:Arnaud:Ricard:
- - Report both:
- - the parallel execution time
- - an evaluation of the load imbalance on the assembly and on the
- solver to know how much gain can still be expected
- - the amount of communications
-**** Conclusion
- - The main issue is the load estimation which seems very hard to do
- without running the code for real.
- - Our iterative load refinement allows significant gains but
- requires a few "dry runs". Ideally, it would be possible to
- rebalance the load at runtime, which is something on which we are
- working.
-*** Issues to solve (Ordered by priority)
-- Built-in standard profiling :: Ricard will run the experiments as he
- has more experience in this but we need systematic and precise
- load profiles for both writing the paper and deciding how to
- efficiently balance (e.g., balancing the assembly vs. balancing
- the sparse solver vs. both at the same time). To this end, Lucas
- should install score-p on MN4 and explain Ricard how to obtain
- the traces/profiles. Arnaud and Lucas will then do the analysis
- and graph generation.
-- Granularity problem :: Too coarse granularity is the best
- explanation (simpler than unstable memory ordering) for the lack
- of good performance at scale and the "jumps" we observed with
- Lucas. It seems there was some memory issue preventing Ricard to
- run with a finer granularity (bin size)
-- Pre-processing :: Solve pre-processing problems to run SFC on large
- cases.
-*** Possible future work
-- Could DLB be applied with our static iterative improvemet (e.g., by
- grouping heavily loaded nodes with lightly loaded ones if we cannot
- solve our granularity problem). Guillaume is not convinced as there
- is some overhead.
-- Would it make sense to have different bin sizes, i.e., an SFC more
- or less fine grained ? If so would it be possible to adjust the bin
- size when reading the data structure ? What about a simple quad-tree
- that would automatically split when bins are too large ? First, we
- would record where elements lie, and estimate their density locally
- (Note that this could be a pre-processing step independant of the
- execution of Alya...) then, we would merge the density from the
- different ranks. The density could then be used to determine at
- which granularity the SFC should be managed.
-
-* Temporal entries
-** Old entries (from my journal)
-*** 2017-02-06 Alya Meeting :MEET:
- Two alternatives with just one physics model:
- - METIS partitioning
- - Space filling curves
-
- Tasks:
- 1. Check if there is really a difference
- 2. Compare the two .par files
- - With GID or ours (ggplot2)
-
- Sources of problems:
- - Load balancing problem
- - Too many communication
- - Badly distributed
-
- Entered on [2017-02-06 Mon 17:46]
-*** 2017-02-07 Installing and Testing GID :BOOT:
-
-http://www.gidhome.com/download/official-versions/linux64/
-
-Download the =tar.gz= version:
-#+begin_src shell :results output
-cd /tmp/
-wget -q ftp://www.gidhome.com/pub/GiD_Official_Versions/Linux/amd64/gid13.0.1-linux-x64.tar.gz
-#+end_src
-
-#+RESULTS:
-
-Extract it
-#+begin_src shell :results output
-cd /tmp/
-tar xfzv gid13.0.1-linux-x64.tar.gz | head
-#+end_src
-
-#+RESULTS:
-#+begin_example
-gid13.0.1-x64/
-gid13.0.1-x64/gid.exe
-gid13.0.1-x64/desktop_icons.tcl
-gid13.0.1-x64/scripts/
-gid13.0.1-x64/scripts/StatusLabel.tcl
-gid13.0.1-x64/scripts/ContourColors.tcl
-gid13.0.1-x64/scripts/units.tcl
-gid13.0.1-x64/scripts/gdi/
-gid13.0.1-x64/scripts/gdi/pkgIndex.tcl
-gid13.0.1-x64/scripts/DataWindows.tcl
-#+end_example
-
-Check if there is a binary:
-#+begin_src shell :results output
-cd /tmp/gid13.0.1-x64/
-file gid
-file gid.exe
-#+end_src
-
-#+RESULTS:
-: gid: POSIX shell script, ASCII text executable
-: gid.exe: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.15, BuildID[sha1]=b5e042b34364226055f5446ec033407d51bac59c, stripped
-
-Let's go for the script, using the terminal.
-
-Tool works in my machine, I can read the examples, =render= to make them
-beautiful. I tried the motor example.
-
-Entered on [2017-02-07 Tue 09:57]
-*** 2017-02-07 Discussions avec Guillaume Houzeaux (Arnaud's Notes) :MEET:
-***** Original notes
-- Ouverture de compte sur marenostrum pour Lucas. Guillaume s'en
- occupe et nous dit dès que c'est fait. En début de semaine prochaine
- normalement.
-- Question générale: Pourquoi les space filling curves sont tellement
- plus efficaces sur leur code que METHIS alors que tous les articles
- disent le contraire ?
-- Use case: système respiratoire.
- - visualisation (avec GiD cimne x64, téléchargeable en version démo
- sans problème) des mesh/sous domaine MPI avec (fichier .par) en 3D
- pour visualiser qui est qui et où ils sont
- - Là, le résultat de Methis montrait qu'un des process était
- responsable de zones très éloignées (au début du tube
- respiratoire et dans les bronches). Est-ce que ça explique le
- problème ? Pas clair... Il faudra qu'on regarde ça nous même.
- - Typical size: 256 cores, 5 secondes pour quelques itérations, ce
- qui devrait suffire si le partitionnement est déjà fait.
- - Faire le partitionnement une bonne fois pour toute et faire un
- restart d'Alya à partir de là.
- - Par rapport à
- http://bsccase02.bsc.es/alya/index.html#Solution_procedure, la
- description de Guillaume correspond à ce qui se passe dans "solve
- module 1". Il y a juste un module, pas de couplage. (M= moment,
- S=Schurr).
- - A priori, pas de lien entre les informations applicatives et ce
- qui apparaît dans une trace paraver
- - Outputs:
- - un .log
- - -partition.par.post.msh et -partition.par.post.res
- - Compilation METHIS
- #+BEGIN_SRC
- cd Alya
- cd thirdparties/metis-4.0; make ...
- cd Extecutables/unix; cat config.h ; # Guillaume will provide a typical one
- ./configure nastin parall ; make
- #+END_SRC
- - Run:
- - Guillaume will explain us how to do the partitionning and how to
- reuse them.
- - See the doc:
- http://bsccase02.bsc.es/alya/tutorial/partitions.html. It is
- actually up-to-date
- - Warning: master slave organization. Hence partition on 255 and
- run on 256 procs.
- - Dans gensap-nastin.dat, on peut définir le NUMBER_OF_STEPS
- - Input:
-***** Follow-up
-I've checked the site:
-http://bsccase02.bsc.es/alya/index.html#Solution_procedure
-
-#+BEGIN_EXAMPLE
- - Read files, define mesh dependent arrays
- - Initial solution or read restart
- - Output and postprocess
-
- +--- do time steps
- !
- | - Compute time step
- | - Begin a time step: update bc, etc.
- |
- | do blocks ---------------------------------+
- | |
- | do coupling --------+ | Modules are grouped into blocks
- | - Solve module1 | |
- ! - Solve module2 | |
- ! ... | | Useful if some equations are coupled
- | - Check coupling cvg | | and others decoupled. Example
- | end do coupling --------+ | Block 1: Nastin - Turbul
- | | Block 2: Chemic: species are transported
- | Goto new block |
- | end do blocks ---------------------------------+
- |
- | - End time step
- | - Output and postprocess
- |
- +--- end do time steps
-
- - Output and postprocess
- - End the run
-#+END_EXAMPLE
-
-Then, I've checked the step-by-step tutorial to partition:
-http://bsccase02.bsc.es/alya/tutorial/partitions.html
-
-The key line in my simple view now is this one:
-#+BEGIN_EXAMPLE
-TASK: ONLY_PREPROCESS, BINARY, SUBDOMAIN=num_domains
-#+END_EXAMPLE
-present in the =case.dat= configuration file for ALYA.
-Run the MPI program without =mpirun=, using a single process.
-
-Then, the resulting pre-processing stage can be registered in files
-that are only read, and the actual simulation can take place.
-*** 2017-02-24 Marenostrum account creation :MN3:
-***** E-mails from Guillaume Houzeaux :ATTACH:
- :PROPERTIES:
- :Attachments: Screen%20Shot%202017-02-01%20at%2014.49.05.png Screen%20Shot%202017-02-01%20at%2014.49.31.png
- :ID: 789c7373-448e-44a3-9c5a-a0afa244280d
- :END:
-Two screen shots joined here.
-***** Paper about load balancing in Alya :ATTACH:
- :PROPERTIES:
- :Attachments: IJHPCA.pdf
- :ID: 5439e62d-768a-4a98-a77f-4890b829c6ff
- :END:
-
-Load balancing of an MPI parallel unstructured CFD code using DLB and
-OpenMP. Marta Garcia-Gasulla 1 , Guillaume Houzeaux 1 , Antoni
-Artigues 1 , Jesús Labarta 12 and Mariano Vázquez 1.
-
-- unstructured meshes
-- the assembly step
- - a loop over the elements of the mesh to compute element matrix and right-hand side
- - an then to assemble them into global matrix and right-hand side
-- load balancing is an issue
- - mesh partitioning is never perfect
- - hardware can also be a random source of imbalance
-- approach
- - dynamic load balance library (OpenMP) \to idle MPI tasks by others
-
-***** E-mail of account creation :ATTACH:
- :PROPERTIES:
- :Attachments: user_responsibilities.pdf 0769_001.pdf
- :ID: b16bcfa4-f3db-40da-b443-8db078c817e9
- :END:
-We would like to inform you that the accounts of your activity in the
-supercomputer Marenostrum III have been created. Here you have the basic
-information to access and use the supercomputer.
-
-When accessing to the machine for the first time, an error message
-regarding ssh keys may appear. This is normal and you have to erase (mn1,mn2,mn3)
-entries from:
-=$HOME/.ssh/known_hosts=
-
-We have 3 login nodes for access to MareNostrum III machine
-{mn1,mn2,mn3}.bsc.es,all connections must be done through ssh
-(Secure SHell).
-
-#+BEGIN_EXAMPLE
-ssh username@mn1.bsc.es
-ssh username@mn2.bsc.es
-#+END_EXAMPLE
-
-The usernames associated to your project leaded by Jose M. Cela are:
-
- =Lucas Schnorr:bsc21835=
-
-The password will be sent in a separate mail with no subject unless you already
-had one for dl01.bsc.es. In order to change your password, you have to login at:
-
-=dl01.bsc.es=
-
-with the same username and password than in the cluster. Then, you have to use
-the 'passwd' command. The new password will become effective 10 minutes after
-the change. For security reasons the password must be changed the
-first time every user accesses the machine.
-
-MareNostrum III provides 5 interactive nodes. In order to gain access to
-each of them you need a ssh connection
-
-#+BEGIN_EXAMPLE
-ssh login5
-#+END_EXAMPLE
-
-The login nodes serve as front ends and are used typically for
-editing, compiling, preparation and submition of batch executions.
-It is not permitted the execution of cpu-bound programs on these nodes,
-if some execution needs more cputime than the permitted, this needs
-to be done through the batch queue system.
-
-The basic limits in queues by job are 48 "wall clock" hours and 2048 cpus. If
-you need to submit a job exceeding 1024 cpus, then you must contact the
-support team to study the scalability and schedule its execution.
-
-No connections are allowed from inside Marenostrum III to the outside world,
-so all the file transfers have to be executed from your local machines
-and not inside Marenostrum III.
-
-Example to copy files or directories from MN to external machine:
-
-#+BEGIN_EXAMPLE
-scp -r {username}@dt01.bsc.es:"SOURCE_MN_directory" "DEST_directory"
-#+END_EXAMPLE
-
-Example to copy files or directories from external machine to MN:
-
-#+BEGIN_EXAMPLE
-scp -r "SOURCE_directory" {username}@dt01.bsc.es:"DEST_MN_directory"
-#+END_EXAMPLE
-
-Each project needs to inform us regularly about their evolution to let us
-know the status of each project running on the machine.
-
-All accounts will be for single user access and they must not be shared
-among several users. If your project needs more resources or accounts,
-only the project manager must do this kind of request. You can find
-the user's guide of MareNostrum III at the following link:
-
- http://www.bsc.es/support/MareNostrum3-ug.pdf
-
-Remember to send us the User Responsibilities document. If we do not receive it
-15 days after the account is accepted, the access will be disabled.
-
-All BSC's staff is eager to help you in the development and success of
-your project, please contact us for any question / suggestion:
-
- e-mail: support@bsc.es
- Barcelona Supercomputing Center
- C/ Jordi Girona, 31
- 08034 Barcelona
-
-Entered on [2017-02-24 Fri 10:46]
-*** 2017-02-24 Trying to compile Alya, preprocess, postprocess :ALYA:
-**** Preliminaries (with Alya compilation)
-***** Local SSH configuration for BSC supercomputer
-Put this in your =.ssh/config=
-#+BEGIN_EXAMPLE
-Host bscdata
- Hostname dt01.bsc.es
- User bsc21835
-Host bsc
- Hostname mn3.bsc.es
- User bsc21835
-#+END_EXAMPLE
-***** Source code identification
-Currently in my Dropbox. It corresponds to a tarball Guillaume
-Houseaux sent me a couple of months ago (the same used by M. Camelo
-in his PFE).
-#+begin_src shell :results output
-cd ~/Downloads/
-md5sum AlyaR1Source.tar.gz
-#+end_src
-
-#+RESULTS:
-: e8d20de91b56c64030d04ed1a4b1c559 AlyaR1Source.tar.gz
-***** Copy Alya Sources to MN3
-#+begin_src shell :results output
-rsync -avuzP ~/Downloads/AlyaR1Source.tar.gz bscdata:.
-ssh bscdata md5sum AlyaR1Source.tar.gz
-#+end_src
-
-#+RESULTS:
-: sending incremental file list
-: AlyaR1Source.tar.gz
-:
32,768 0% 0.00kB/s 0:00:00
21,856,256 21% 20.81MB/s 0:00:03
56,590,336 55% 26.98MB/s 0:00:01
93,487,104 92% 29.72MB/s 0:00:00
101,349,570 100% 29.91MB/s 0:00:03 (xfr#1, to-chk=0/1)
-:
-: sent 91,884,154 bytes received 34 bytes 20,418,708.44 bytes/sec
-: total size is 101,349,570 speedup is 1.10
-: e8d20de91b56c64030d04ed1a4b1c559 AlyaR1Source.tar.gz
-***** Alya Compilation
-Execute the following commands there.
-
-#+begin_src shell :results output
-tar xfz AlyaR1Source.tar.gz
-cd ~/AlyaR1Source/Thirdparties/metis-4.0; make
-cd ~/AlyaR1Source/Executables/unix/;
-cp configure.in/config_ifort.in config.in
-./configure nastin parall ; make
-#+end_src
-
-I've got two binaries (g for Debug):
-
-#+BEGIN_EXAMPLE
--rwxr-xr-x 1 bsc21835 bsc21 8.8M Feb 24 14:09 Alya.x
--rwxr-xr-x 1 bsc21835 bsc21 27M Feb 24 14:10 Alya.g
-#+END_EXAMPLE
-**** Preprocess the =4_tufan=
-***** Copy the test case to there
-#+begin_src shell :results output
-md5sum ~/Downloads/4_tufan_run.tar.gz
-rsync -avuzP ~/Downloads/4_tufan_run.tar.gz bscdata:.
-ssh bscdata md5sum 4_tufan_run.tar.gz
-#+end_src
-
-#+RESULTS:
-: b4127cb22bed9309d1234827f41135c9 /home/schnorr/Downloads/4_tufan_run.tar.gz
-: sending incremental file list
-: 4_tufan_run.tar.gz
-:
32,768 0% 0.00kB/s 0:00:00
20,709,376 5% 19.72MB/s 0:00:19
52,789,248 12% 25.16MB/s 0:00:13
82,837,504 20% 26.33MB/s 0:00:12
115,376,128 28% 27.51MB/s 0:00:10
149,848,064 36% 30.80MB/s 0:00:08
183,861,248 45% 31.27MB/s 0:00:07
216,989,696 53% 31.99MB/s 0:00:05
249,724,928 61% 32.04MB/s 0:00:04
283,770,880 69% 31.93MB/s 0:00:03
317,849,600 77% 31.95MB/s 0:00:02
347,111,424 84% 31.02MB/s 0:00:01
377,749,504 92% 30.53MB/s 0:00:00
408,525,195 100% 30.11MB/s 0:00:12 (xfr#1, to-chk=0/1)
-:
-: sent 396,055,909 bytes received 34 bytes 27,314,202.97 bytes/sec
-: total size is 408,525,195 speedup is 1.03
-: b4127cb22bed9309d1234827f41135c9 4_tufan_run.tar.gz
-***** Unpack
-#+begin_src shell :results output
-tar xfz 4_tufan_run.tar.gz
-cd 4_tufan_run/c/
-#+end_src
-***** Preprocess using METIS 4.0
-Everything is done locally, in my notebook, because I don't know yet
-how to submit jobs in MN and Alya.x fails to run on =login= nodes.
-
-See: http://bsccase02.bsc.es/alya/tutorial/partitions.html
-- Partion preprocess step
-
-I am using Metis 4, but the tutorial tells me to use version 5.
-
-Let's proceed anyway.
-
-Create the directories to hold the partitioned meshes (for 16 domains)
-
-#+begin_src shell :results output
-cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
-../../Utils/user/alya-hierarchy 116
-#+end_src
-
-#+RESULTS:
-:
-: --| alya-hierarchy |--
-: --|
-: --| 2 directories have been created
-: --|
-: --| Bye.
-: --|
-: --|
-
-This is the DAT file for preprocessing.
-
-#+begin_src shell :results output
-cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
-cat lucas_preprocess_case.dat
-#+end_src
-
-#+RESULTS:
-#+begin_example
-$-------------------------------------------------------------------
-RUN_DATA
- ALYA: sq_cyl
- RUN_TYPE: noCONTI , PRELIMINARY, FREQUENCY=100
- LATEX_INFO_FILE: Yes
- LIVE_INFORMATION: Screen
-END_RUN_DATA
-$-------------------------------------------------------------------
-PROBLEM_DATA
- TIME_COUPLING: GLOBAL, PRESCR
- TIME_INTERVAL= 0.0,100000.0
- TIME_STEP_SIZE= 0.00025
- NUMBER_OF_STEPS= 100000
- MAXIMUM_NUMBER_GLOBAL= 1
- NASTIN_MODULE: On
- END_NASTIN_MODULE
- PARALL_SERVICE: On
- PARTITION_TYPE: FACES
- FILE_HIERARCHY: ON
- FILE_OPEN_CLOSE: Yes
- VIRTUAL_FILE: On, MAXIMUM_MEMORY=0.5
- TASK: ONLY_PREPROCESS, BINARY, SUBDOMAIN=16
- END_PARALL_SERVICE
-END_PROBLEM_DATA
-$-------------------------------------------------------------------
-#+end_example
-
-Note the customized =SUBDOMAIN=16= setting.
-
-Local execution gives this as output:
-
-#+begin_src shell :results output
-cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
-mpirun -np 1 ../../Executables/unix/Alya.x lucas_preprocess_case
-#+end_src
-
-#+RESULTS:
-#+begin_example
---|
---| ALYA START ALYA FOR PROBLEM: sq_cyl
---|
---| ALYA READ PROBLEM DATA
---| ALYA START CONSTRUCT DOMAIN
---| ALYA READ MESH DATA
---| ALYA READ MESH ARRAYS
---| ALYA READ ELEMENT TYPES
---| ALYA CHECK ELEMENT TYPES
---| ALYA READ ELEMENT CONNECTIVITY
---| ALYA CHECK ELEMENT CONNECTIVITY
---| ALYA READ COORDINATES
---| ALYA READ BOUNDARY CONNECTIVITY, TYPES AND ELEMENT CONNECTIVITY
---| ALYA CHECK BOUNDARY TYPES
---| ALYA CHECK BOUNDARY CONNECTIVITY
---| ALYA CHECK CONNECTIVITY BOUNDARY/ELEMENT
---| ALYA READ BOUNDARY CONDITIONS
---| ALYA WARNING: KERNEL WILL EXTRAPOLATE FROM BOUNDARY CODES TO NODE CODES
---| ALYA WARNING: KERNEL READS BOUNDARY CODES ON BOUNDARIES
---| ALYA END READ MESH ARRAYS
---| ALYA COMPUTE GRAPH
---| ALYA KERNEL: COMPUTE GROUPS OF DEFLATED CG
---| ALYA START MESH PARTITION (# OF SUBDOMAINS= 16)
---| ALYA PARALL PREPROCESS --> ALLOCATE MEMORY...
---| ALYA PARALL PREPROCESS --> SET WEIGHTS...
---| ALYA MASTER COMPUTES ELEMENT GRAPH (# EDGES= 6214430, MAX # EDGES/ELEMENT= 6)
---| ALYA MASTER PARTITIONS ELEMENT GRAPH USING FACE CONNECTIVITY
---| ALYA MASTER COMPUTES COMMUNICATION STRATEGY
---| ALYA MASTER ORDERS INTERIOR AND BOUNDARY NODES
---| ALYA MASTER COMPUTES PERMUTATION ARRAYS
---| ALYA MASTER WRITES MESH AND PARTITION DATA IN RESTART FILES WITH HIERARCHY
---| ALYA PARALL PREPROCESS --> S/R 1 - TOT 16
---| ALYA PARALL PREPROCESS --> S/R 2 - TOT 16
---| ALYA PARALL PREPROCESS --> S/R 3 - TOT 16
---| ALYA PARALL PREPROCESS --> S/R 4 - TOT 16
---| ALYA PARALL PREPROCESS --> S/R 5 - TOT 16
---| ALYA PARALL PREPROCESS --> S/R 6 - TOT 16
---| ALYA PARALL PREPROCESS --> S/R 7 - TOT 16
---| ALYA PARALL PREPROCESS --> S/R 8 - TOT 16
---| ALYA PARALL PREPROCESS --> S/R 9 - TOT 16
---| ALYA PARALL PREPROCESS --> S/R 10 - TOT 16
---| ALYA PARALL PREPROCESS --> S/R 11 - TOT 16
---| ALYA PARALL PREPROCESS --> S/R 12 - TOT 16
---| ALYA PARALL PREPROCESS --> S/R 13 - TOT 16
---| ALYA PARALL PREPROCESS --> S/R 14 - TOT 16
---| ALYA PARALL PREPROCESS --> S/R 15 - TOT 16
---| ALYA PARALL PREPROCESS --> S/R 16 - TOT 16
---| ALYA PARALL PREPROCESS --> SEND/REC 16 - TOTAL 16
---| ALYA PARALL PREPROCESS --> COORD, LTYPE, LNODS, LTYPB, LNODB...
---| ALYA PARALL PREPROCESS --> LBOEL...
---| ALYA PARALL PREPROCESS --> KFL_FIELD, XFIEL, TIME_FIELD,...
---| ALYA PARALL PREPROCESS --> SET DATA...
---| ALYA PARALL PREPROCESS --> BC DATA...
---| ALYA PARALL PREPROCESS --> COMMUNICATION ARRAYS...
---| ALYA PARALL PREPROCESS --> SEND DATA TO SLAVES... DONE.
---| ALYA END MESH PARTITION
---|
---| ALYA CALCULATIONS CORRECT
---|
-#+end_example
-
-Here are the contents of the =PAR*= dirs:
-
-#+begin_src shell :results output
-cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
-ls -1hl PAR000*
-#+end_src
-
-#+RESULTS:
-#+begin_example
-PAR000000:
-total 4.0K
--rw-r--r-- 1 schnorr schnorr 2.9K Feb 24 16:02 lucas_preprocess_case.par.rst0
-
-PAR000100:
-total 102M
--rw-r--r-- 1 schnorr schnorr 6.4M Feb 24 16:02 lucas_preprocess_case.par.rst1
--rw-r--r-- 1 schnorr schnorr 6.4M Feb 24 16:02 lucas_preprocess_case.par.rst10
--rw-r--r-- 1 schnorr schnorr 6.5M Feb 24 16:02 lucas_preprocess_case.par.rst11
--rw-r--r-- 1 schnorr schnorr 6.4M Feb 24 16:02 lucas_preprocess_case.par.rst12
--rw-r--r-- 1 schnorr schnorr 6.3M Feb 24 16:02 lucas_preprocess_case.par.rst13
--rw-r--r-- 1 schnorr schnorr 6.7M Feb 24 16:02 lucas_preprocess_case.par.rst14
--rw-r--r-- 1 schnorr schnorr 6.6M Feb 24 16:02 lucas_preprocess_case.par.rst15
--rw-r--r-- 1 schnorr schnorr 6.3M Feb 24 16:02 lucas_preprocess_case.par.rst16
--rw-r--r-- 1 schnorr schnorr 6.3M Feb 24 16:02 lucas_preprocess_case.par.rst2
--rw-r--r-- 1 schnorr schnorr 6.5M Feb 24 16:02 lucas_preprocess_case.par.rst3
--rw-r--r-- 1 schnorr schnorr 6.4M Feb 24 16:02 lucas_preprocess_case.par.rst4
--rw-r--r-- 1 schnorr schnorr 6.3M Feb 24 16:02 lucas_preprocess_case.par.rst5
--rw-r--r-- 1 schnorr schnorr 6.5M Feb 24 16:02 lucas_preprocess_case.par.rst6
--rw-r--r-- 1 schnorr schnorr 6.3M Feb 24 16:02 lucas_preprocess_case.par.rst7
--rw-r--r-- 1 schnorr schnorr 6.3M Feb 24 16:02 lucas_preprocess_case.par.rst8
--rw-r--r-- 1 schnorr schnorr 6.6M Feb 24 16:02 lucas_preprocess_case.par.rst9
-#+end_example
-
-Okay, now let's try to use them avoiding the preprocess.
-**** Postprocess the =4_tufan=
-***** Postprocess
-Still local (notebook), for the same reasons.
-
-See: http://bsccase02.bsc.es/alya/tutorial/partitions.html
-- Partion postprocess step
-
-#+begin_src shell :results output
-cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
-cat lucas_postprocess_case.dat
-#+end_src
-
-#+RESULTS:
-#+begin_example
-$-------------------------------------------------------------------
-RUN_DATA
- ALYA: sq_cyl
- RUN_TYPE: noCONTI , PRELIMINARY, FREQUENCY=100
- LATEX_INFO_FILE: Yes
- LIVE_INFORMATION: Screen
-END_RUN_DATA
-$-------------------------------------------------------------------
-PROBLEM_DATA
- TIME_COUPLING: GLOBAL, PRESCR
- TIME_INTERVAL= 0.0,100000.0
- TIME_STEP_SIZE= 0.00025
- NUMBER_OF_STEPS=2
- MAXIMUM_NUMBER_GLOBAL= 1
- NASTIN_MODULE: On
- END_NASTIN_MODULE
-
-$ this is the section that does the postprocess
- PARALL_SERVICE: On
- PARTITION_TYPE: FACES
- FILE_HIERARCHY: ON
- FILE_OPEN_CLOSE: Yes
- VIRTUAL_FILE: On, MAXIMUM_MEMORY=0.5
- TASK: READ_PREPROCESS, BINARY
- END_PARALL_SERVICE
-
-$ This part has been removed to only preprocess
-$ PARALL_SERVICE: On
-$ OUTPUT_FILE: OFF
-$ POSTPROCESS: MASTER
-$ PARTITION_TYPE: FACES
-$ $ COMMUNICATION: ASYNCRONOUS
-$ END_PARALL_SERVICE
-END_PROBLEM_DATA
-$-------------------------------------------------------------------
-#+end_example
-
-The machine file I have:
-
-#+begin_src shell :results output
-cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
-cat machinefile
-#+end_src
-
-#+RESULTS:
-: localhost slots=32 max_slots=32
-
-Now, let's run again with 17 procs:
-
-#+begin_src shell :results output
-cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
-mpirun -machinefile machinefile -np 17 ../../Executables/unix/Alya.x lucas_postprocess_case
-#+end_src
-
-#+RESULTS:
-
-This has failed. The log tells me why:
-
-#+begin_src shell :results output
-cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
-cat lucas_postprocess_case.log
-#+end_src
-
-#+RESULTS:
-#+begin_example
-
-
-
- --------
- |- A L Y A: sq_cyl
- --------
-
-
-
-
-
- --------------------------------------------------------
-
- |- AN ERROR HAS BEEN DETECTED:
-
- ERROR WHEN OPENING THE PARALL RESTART FILE: PAR000000/lucas_postprocess_case.par.rst0
-
-#+end_example
-
-Looks like I can't change the DAT filename.
-
-Let's rename it.
-
-Next code block is non-reproducible.
-
-#+begin_src shell :results output
-cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
-#cp lucas_preprocess_case.dat lucas_preprocess_case.dat.bak
-#cp lucas_postprocess_case.dat lucas_preprocess_case.dat
-cat lucas_preprocess_case.dat
-#+end_src
-
-#+RESULTS:
-#+begin_example
-$-------------------------------------------------------------------
-RUN_DATA
- ALYA: sq_cyl
- RUN_TYPE: noCONTI , PRELIMINARY, FREQUENCY=100
- LATEX_INFO_FILE: Yes
- LIVE_INFORMATION: Screen
-END_RUN_DATA
-$-------------------------------------------------------------------
-PROBLEM_DATA
- TIME_COUPLING: GLOBAL, PRESCR
- TIME_INTERVAL= 0.0,100000.0
- TIME_STEP_SIZE= 0.00025
- NUMBER_OF_STEPS=2
- MAXIMUM_NUMBER_GLOBAL= 1
- NASTIN_MODULE: On
- END_NASTIN_MODULE
-
-$ this is the section that does the postprocess
- PARALL_SERVICE: On
- OUTPUT_FILE: OFF
- PARTITION_TYPE: FACES
- FILE_HIERARCHY: ON
- FILE_OPEN_CLOSE: Yes
- VIRTUAL_FILE: On, MAXIMUM_MEMORY=0.5
- TASK: READ_PREPROCESS, BINARY
- END_PARALL_SERVICE
-
-END_PROBLEM_DATA
-$-------------------------------------------------------------------
-#+end_example
-
-Okay.
-
-Let's run it:
-
-#+begin_src shell :results output
-cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
-mpirun -machinefile machinefile -np 17 ../../Executables/unix/Alya.x lucas_preprocess_case 2>&1 > x.log
-#+end_src
-
-#+RESULTS:
-
-#+begin_src shell :results output
-cd ~/svn/guilherme/alya-imag/4_tufan_run/7/
-cat x.log
-#+end_src
-
-#+RESULTS:
-#+begin_example
---|
---| ALYA START ALYA FOR PROBLEM: sq_cyl
---|
---| ALYA CHECK MPI. 1 MASTER + 16 SUBDOMAINS
---| ALYA MPI IS WORKING WELL
---| ALYA READ PROBLEM DATA
---| ALYA START CONSTRUCT DOMAIN
---| ALYA MASTER/SLAVES READ MESH AND PARTITION DATA FROM RESTART FILE WITH HIERARCHY
---| ALYA MASTER/SLAVES: ALL SLAVES HAVE READ THEIR RESTART FILES
---| ALYA KERMOD: READ DATA
---| ALYA KERMOD FILE DOES NOT EXITS: USE DEFAULT OPTIONS
---| ALYA PARALL: COMPUTE INTER-COLOR COMMUNICATORS
---| ALYA PARALL: COMPUTE BIN STRUCTURE FOR PARTITIONS
---| ALYA RENUMBER ELEMENTS
---| ALYA RECOMPUTE GRAPH FOR PERIODICITY
---| ALYA CHECK ELEMENT ORDERING
---| ALYA CHECK BOUNDARY ORDERING
---| ALYA PARALL: GET HALO GEOMETRY
---| ALYA PARALL: CREATE INTRA-ZONE COMMUNICATION ARRAY
---| ALYA PARALL: CREATE INTRA-SUBDOMAIN COMMUNICATION ARRAY
---| ALYA OUTPUT MESH
---| ALYA COMPUTE LUMPED MASS MATRIX
---| ALYA COMPUTE CLOSED MASS MATRIX
---| ALYA COMPUTE EXTERIOR NORMALS
---| ALYA EXTRAPOLATE BOUNDARY CODES TO NODE CODES
---| ALYA END CONSTRUCT DOMAIN
---| ALYA MODULE DATA
---| ALYA NASTIN: READ DATA
---| ALYA NASTIN: MASTER SENDS PHYSICAL AND NUMERICAL DATA TO SLAVES
---| ALYA NASTIN: MASTER SENDS PHYSICAL AND NUMERICAL ARRAYS TO SLAVES
---| ALYA WARNINGS HAVE BEEN FOUND IN MODULE NASTIN
---|
---| ALYA ABORTED IN |----> 1 ERROR HAS BEEN FOUND IN NASTIN DATA FILE
---|
-#+end_example
-
-**** Next steps
-- Get an updated version of Alya
-- Fix the partition postprocess step
-- Test case
- - The mesh for the respiratory system
- - Adapt preprocess + postprocess to this case
-- Find out how to activate the second method
- for mesh partitioning (space filling curves)
-- Reconfirm the fact that SFC are faster than METIS
-** 2017-03-27 Trying to compile SVN version of Alya locally
-*** Configure with gfortran
-#+begin_src shell :results output
-cd ~/misc/alya/alya-bsc/Executables/unix/
-rm config.in
-ln -s ./configure.in/config_gfortran.in config.in
-./configure nastin parall
-#+end_src
-
-#+RESULTS:
-#+begin_example
-
---| Alya configure |--
-
---| Build Alya makefile
-
- Linux OS identified.
- Using Intel Fortran Compiler ifort.
-
- Creating makefile for the modules demanded, when the source folder exists:
-
- Name: adapti ( service # 59 ) --> compile = no iffold = yes iffake = no
- Name: alefor ( module # 7 ) --> compile = no iffold = yes iffake = no
- Name: apelme ( module # 9 ) --> compile = no iffold = no iffake = no
- Name: casimi ( module # 23 ) --> compile = no iffold = no iffake = no
- Name: cgns24 ( service # 56 ) --> compile = no iffold = yes iffake = yes
- Name: chemic ( module # 19 ) --> compile = no iffold = yes iffake = no
- Name: codire ( module # 3 ) --> compile = no iffold = yes iffake = no
- Name: commdo ( service # 65 ) --> compile = no iffold = no iffake = no
- Name: coupli ( service # 64 ) --> compile = no iffold = no iffake = no
- Name: dodeme ( service # 57 ) --> compile = no iffold = yes iffake = yes
- Name: exmedi ( module # 5 ) --> compile = no iffold = yes iffake = no
- Name: gidpos ( service # 53 ) --> compile = no iffold = yes iffake = yes
- Name: gotita ( module # 11 ) --> compile = no iffold = yes iffake = no
- Name: handfp ( service # 54 ) --> compile = no iffold = yes iffake = yes
- Name: hdfpos ( service # 62 ) --> compile = no iffold = yes iffake = no
- Name: helmoz ( module # 20 ) --> compile = no iffold = yes iffake = no
- Name: immbou ( module # 21 ) --> compile = no iffold = yes iffake = no
- Name: latbol ( module # 8 ) --> compile = no iffold = no iffake = no
- Name: levels ( module # 14 ) --> compile = no iffold = yes iffake = no
- Name: magnet ( module # 16 ) --> compile = no iffold = yes iffake = no
- Name: nasedg ( module # 18 ) --> compile = no iffold = yes iffake = no
- Name: nastal ( module # 6 ) --> compile = no iffold = yes iffake = no
- Name: nastin ( module # 1 ) --> compile = yes iffold = yes iffake = no --> OK to compile!
- Name: neutro ( module # 26 ) --> compile = no iffold = yes iffake = no
- Name: optsol ( service # 63 ) --> compile = no iffold = yes iffake = yes
- Name: parall ( service # 55 ) --> compile = yes iffold = yes iffake = yes --> OK to compile!
- Name: partis ( module # 17 ) --> compile = no iffold = yes iffake = no
- Name: porous ( module # 24 ) --> compile = no iffold = yes iffake = no
- Name: quanty ( module # 15 ) --> compile = no iffold = yes iffake = no
- Name: radiat ( module # 22 ) --> compile = no iffold = yes iffake = no
- Name: shapar ( service # 51 ) --> compile = no iffold = no iffake = no
- Name: solidz ( module # 10 ) --> compile = no iffold = yes iffake = no
- Name: solmum ( service # 52 ) --> compile = no iffold = yes iffake = yes
- Name: solpls ( service # 58 ) --> compile = no iffold = yes iffake = no
- Name: temper ( module # 2 ) --> compile = no iffold = yes iffake = no
- Name: turbul ( module # 4 ) --> compile = no iffold = yes iffake = no
- Name: wavequ ( module # 12 ) --> compile = no iffold = yes iffake = no
- Name: xxxxxx ( module # 25 ) --> compile = no iffold = no iffake = no
-
- Makefile will make the following binaries:
-
- Alya.x --> Production (non debugger) binary
- Alya.g --> Debugger binary
-
-
---| Makefile created. Ready to make.
---| The command you typed has been saved in file conf.log
---| Bye.
-
-
-#+end_example
-*** Make
-#+begin_src shell :results output
-cd ~/misc/alya/alya-bsc/ThirdParties/metis-4.0/; make -j 4 2>&1
-cd ~/misc/alya/alya-bsc/Executables/unix/; make -j 4 2>&1
-#+end_src
-
-#+RESULTS:
-** 2017-03-28 OAR Job Allocation
-#+begin_src shell :results output
-oarsub -p "cluster='taurus'" -l "nodes=4/walltime=03:00:00" -r "$(date '+%Y-%m-%d %H:%M:%S')" -t deploy
-oarsub -C 852259
-#taurus-1.lyon.grid5000.fr
-#taurus-16.lyon.grid5000.fr
-#taurus-2.lyon.grid5000.fr
-#taurus-7.lyon.grid5000.fr
-kadeploy3 --env-file images/stretch_energy.env --file ${OAR_NODE_FILE} -k
-cd ~/alya-bsc/Thirdparties/metis-4.0/; make -j 4
-cd ~/alya-bsc/Executables/unix/; ln -s ./configure.in/config_gfortran.in config.in; make -j 4
-#+end_src
-** 2017-03-29 Experiments to execute
-#+begin_src shell :results output
-for CASE in sfc metis; do
- for NP in 32 64 96 120; do
- NP=$((NP+1))
- UNIQUE=taurus-np${NP}-${CASE}
- echo "./scripts/run_alya_experiment.sh machine-file ${NP} ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE"
- echo "mv $UNIQUE /home/lschnorr/"
- done
-done > script.sh
-cat script.sh
-#+end_src
-
-#+RESULTS:
-#+begin_example
-./scripts/run_alya_experiment.sh machine-file 33 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np33-sfc
-mv taurus-np33-sfc /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 65 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np65-sfc
-mv taurus-np65-sfc /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 97 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np97-sfc
-mv taurus-np97-sfc /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 121 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np121-sfc
-mv taurus-np121-sfc /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 33 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np33-metis
-mv taurus-np33-metis /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 65 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np65-metis
-mv taurus-np65-metis /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 97 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np97-metis
-mv taurus-np97-metis /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 121 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np121-metis
-mv taurus-np121-metis /home/lschnorr/
-#+end_example
-** 2017-03-29 Quick analysis of first results
-I got these:
-
-#+name: first-results
-#+begin_src shell :results output org table
-for case in $(find $HOME/segundo/taurus-np* | grep -e "sfc.org" -e "metis.org"); do echo -n "$case "; cat $case | grep epoch | sed "s/^.*: //" | tr '\n' '-' | sed -e "s/-$//" -e "s/-/ - /" -e "s/$/\n/" | bc -l; done | sed -e "s/^.*np//" -e "s/.org//" -e "s/-/ /g" -e "s/ */|/g"
-#+end_src
-
-#+RESULTS: first-results
-#+BEGIN_SRC org
-| 121 | metis | 0 |
-| 121 | sfc | 1 |
-| 33 | metis | 1160 |
-| 33 | sfc | 938 |
-| 65 | metis | 653 |
-| 65 | sfc | 612 |
-| 97 | metis | 503 |
-| 97 | sfc | 470 |
-#+END_SRC
-
-#+header: :var df=first-results
-#+begin_src R :results output graphics :file img/first-results.png :exports both :width 600 :height 400 :session
-library(dplyr);
-library(ggplot2);
-df %>%
- rename(NP = V1,
- Case = V2,
- Time = V3) %>%
- filter(NP != 121) %>%
- ggplot(aes(x=NP, y=Time, color=Case)) +
- theme_bw() +
- ylim(0,NA) +
- geom_point() +
- geom_line() +
- theme(legend.position = "top");
-#+end_src
-
-#+RESULTS:
-[[file:img/first-results.png]]
-** 2017-03-29 Planning the tracing with Scorep
-
-Score-P 3.0 allows the selective instrumentation of any function using
-a GCC plugin. Do to so, refer to the Section 3.1 of ScoreP user
-manual. The flag is =--instrument-filter=, much better described in
-Section 5.3 of the same document. This method is preferred since it
-reduces the intrusion. To make such method available in a scorep
-installation, we need to previously install the package
-=gcc-6-plugin-dev= or something similar.
-
-It is interesting to disable everything else which is not MPI and the
-necessary to trace filtered user functions. Guilherme, for example,
-had used this line to instrument Alya:
-
-=scorep --nopomp --nocompiler --nocuda --nopdt --nouser --noopencl=
-
-We would have to keep only:
-
-=scorep --nopomp --nocuda --noopencl --nopdt=
-
-This is the filter that Guilherme used dynamically. We want to use it
-statically.
-
-#+begin_src shell :results output :tangle alya-timestep-filters.scorep
-SCOREP_REGION_NAMES_BEGIN
-EXCLUDE *
-INCLUDE MPI COM
-INCLUDE endste_ timste_
-SCOREP_REGION_NAMES_END
-#+end_src
-
-The way to use this filter in runtime is to use the appropriate
-environment variable this way:
-
-#+BEGIN_EXAMPLE
-SCOREP_FILTERING_FILE=name_of_the_file
-#+END_EXAMPLE
-
-_Summary_ of that has worked:
-
-Finally, I've managed to get it working with the following steps.
-
-1. Configure ScoreP with the following line
-#+BEGIN_EXAMPLE
-./configure --without-gui --prefix=/home/lschnorr/install/nova/scorep-3.0-alya/ --with-shmem=no --with-opari2=no
-#+END_EXAMPLE
-- Do not forget to install the gcc plugin package first
-
-2. Configure Alya's =config.in= to use scorep this way
-#+BEGIN_EXAMPLE
-SCOREP=scorep --compiler --instrument-filter=/home/lschnorr/alya-bsc/Executables/unix/alya-timestep-filters.scorep
-#+END_EXAMPLE
-Where the contents of the filter are like this
-#+BEGIN_EXAMPLE
-#SCOREP_FILE_NAMES_BEGIN
-#EXCLUDE *
-#INCLUDE */Alya.f90
-#SCOREP_FILE_NAMES_END
-
-SCOREP_REGION_NAMES_BEGIN
-EXCLUDE *
-INCLUDE endste
-INCLUDE timste
-SCOREP_REGION_NAMES_END
-#+END_EXAMPLE
-
-3. Finally, run the program this way
-#+BEGIN_EXAMPLE
-mpirun -x SCOREP_FILTERING_FILE=/home/lschnorr/alya-bsc/Executables/unix/alya-timestep-filters.scorep -x SCOREP_EXPERIMENT_DIRECTORY=/home/lschnorr/xxx-scorep7 -x SCOREP_OVERWRITE_EXPERIMENT_DIRECTORY=true -x SCOREP_ENABLE_TRACING=true --bind-to core -np 12 -machinefile /tmp/Alya-Perf/machine-file /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap
-#+END_EXAMPLE
-- The traces need to be put in the HOME NFS.
-- Filtering is not obligatory here since we instrumented in compile time.
-** 2017-04-05 Infiniband problems
-In all machines that belong to an Infiniband experiment, do:
-
-1. In file =/etc/security/limits.conf=, add these lines:
- #+BEGIN_EXAMPLE
- * hard memlock unlimited
- * soft memlock unlimited
- #+END_EXAMPLE
-
-2. In file =/etc/systemd/system.conf=, make the line looks like this:
- #+BEGIN_EXAMPLE
- DefaultLimitMEMLOCK=infinity
- #+END_EXAMPLE
-** 2017-04-06 Nose check
-#+name: nose_table
- | Rank | X | Y | Z |
- |------+--------------+--------------+--------------|
- | 1 | 0.919827E-01 | 0.115893E+00 | -.639115E-01 |
- | 2 | 0.103425E+00 | 0.149886E+00 | -.123160E+00 |
- | 3 | 0.103773E+00 | 0.169636E+00 | -.146991E+00 |
- | 4 | 0.961346E-01 | 0.202625E+00 | -.216010E+00 |
- | 5 | 0.139644E+00 | 0.204323E+00 | -.223233E+00 |
- | 6 | 0.101879E+00 | 0.850899E-01 | 0.235732E-01 |
- | 7 | 0.933165E-01 | 0.458008E-01 | 0.261210E-01 |
- | 8 | 0.913297E-01 | 0.682040E-01 | 0.290498E-01 |
- | 9 | 0.952650E-01 | 0.111200E+00 | 0.149948E-01 |
- | 10 | 0.950596E-01 | 0.122169E+00 | -.234605E-01 |
- | 11 | -.403059E+00 | -.400000E+00 | -.489343E+00 |
- | 12 | 0.596941E+00 | -.400000E+00 | -.489343E+00 |
- | 13 | 0.596941E+00 | 0.240397E+00 | -.489343E+00 |
- | 14 | -.403059E+00 | 0.240397E+00 | -.489343E+00 |
- | 15 | -.403059E+00 | -.400000E+00 | 0.510656E+00 |
- | 16 | 0.596941E+00 | -.400000E+00 | 0.510656E+00 |
- | 17 | 0.596941E+00 | 0.240397E+00 | 0.510656E+00 |
- | 18 | -.403059E+00 | 0.240397E+00 | 0.510656E+00 |
-
-#+begin_src R :results output :session :exports both :var df=nose_table
-df;
-#+end_src
-
-#+RESULTS:
-#+begin_example
- Rank X Y Z
-1 1 0.0919827 0.1158930 -0.0639115
-2 2 0.1034250 0.1498860 -0.1231600
-3 3 0.1037730 0.1696360 -0.1469910
-4 4 0.0961346 0.2026250 -0.2160100
-5 5 0.1396440 0.2043230 -0.2232330
-6 6 0.1018790 0.0850899 0.0235732
-7 7 0.0933165 0.0458008 0.0261210
-8 8 0.0913297 0.0682040 0.0290498
-9 9 0.0952650 0.1112000 0.0149948
-10 10 0.0950596 0.1221690 -0.0234605
-11 11 -0.4030590 -0.4000000 -0.4893430
-12 12 0.5969410 -0.4000000 -0.4893430
-13 13 0.5969410 0.2403970 -0.4893430
-14 14 -0.4030590 0.2403970 -0.4893430
-15 15 -0.4030590 -0.4000000 0.5106560
-16 16 0.5969410 -0.4000000 0.5106560
-17 17 0.5969410 0.2403970 0.5106560
-18 18 -0.4030590 0.2403970 0.5106560
-#+end_example
-
-#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 600 :height 400 :session
-library(ggplot2);
-plot(df);
-#+end_src
-
-#+RESULTS:
-[[file:/tmp/babel-22618hTf/figure22618i_y.png]]
-
-** 2017-04-06 Alya modifications to get more info, set balance to 1 :EXP10:
-#+BEGIN_EXAMPLE
-lschnorr@fnancy:~/alya-bsc$ git diff
-diff --git a/Executables/unix/configure.in/config_gfortran.in b/Executables/unix/configure.in/config_gfortran.in
-index 1c73a1a..90de238 100644
---- a/Executables/unix/configure.in/config_gfortran.in
-+++ b/Executables/unix/configure.in/config_gfortran.in
-@@ -4,16 +4,17 @@
- #module load gcc/5.1.0 openmpi/1.8.5 #
- ###################################################################
-
-+SCOREP=scorep --compiler --instrument-filter=/home/lschnorr/alya-bsc/Executables/unix/alya-timestep-filters.scorep
-
--F77 = mpif90
--F90 = mpif90
--FCOCC = mpicc -c
-+F77 = $(SCOREP) mpif90
-+F90 = $(SCOREP) mpif90
-+FCOCC = $(SCOREP) mpicc -c
- FCFLAGS = -c -J$O -I$O -ffree-line-length-none -fimplicit-none
- FPPFLAGS = -x f95-cpp-input
- EXTRALIB = -lc
- EXTRAINC =
--fa2p = mpif90 -c -x f95-cpp-input -DMPI_OFF -J../../Utils/user/alya2pos -I../../Utils/user/alya2pos
--fa2plk = mpif90 -lc
-+fa2p = $(SCOREP) mpif90 -c -x f95-cpp-input -DMPI_OFF -J../../Utils/user/alya2pos -I../../Utils/user/alya2pos
-+fa2plk = $(SCOREP) mpif90 -lc
-
- ###################################################################
- # PERFORMANCE FLAGS #
-diff --git a/Sources/kernel/parall/mod_par_partit_sfc.f90 b/Sources/kernel/parall/mod_par_partit_sfc.f90
-index ee6fd1d..e55185d 100644
---- a/Sources/kernel/parall/mod_par_partit_sfc.f90
-+++ b/Sources/kernel/parall/mod_par_partit_sfc.f90
-@@ -383,7 +383,7 @@ contains
-
- iboxf = lboxf(ielem)
- iboxl = lboxl(ielem)
-- iweig = real(ngaus(abs(ltype_par(ielem))),rp)
-+ iweig = 1_rp !real(ngaus(abs(ltype_par(ielem))),rp)
-
- if( PAR_MY_PARMETIS_RANK /= iboxc-1 )then
- if(bufwei(iboxl*2) == 0_rp) then
-diff --git a/Sources/services/parall/par_outinf.f90 b/Sources/services/parall/par_outinf.f90
-index 6664b93..301522a 100644
---- a/Sources/services/parall/par_outinf.f90
-+++ b/Sources/services/parall/par_outinf.f90
-@@ -15,6 +15,7 @@ subroutine par_outinf()
- use mod_parall
- implicit none
- integer(ip) :: isubd,ksmin(2),ksmax(2),kaver,koutp,nb
-+ integer, parameter :: out_unit_npoin=20
-
- if( IMASTER ) then
-
-@@ -27,7 +27,11 @@ subroutine par_outinf()
- ksmin = huge(1_ip)
- ksmax = -huge(1_ip)
- kaver = 0
-+ ! LUCAS: manual instrumentation starts here, all code marked by LUCAS
-+ open (unit=out_unit_npoin,file="results_NPOIN_NELEM_NELEW_NBOUN.log",action="write",status="replace")
- do isubd=1,npart_par
-+ ! LUCAS: Here I need to print: npoin_par(isubd)
-+ write (out_unit_npoin,*) "Rank ", isubd ," NPOIN ", npoin_par(isubd), " NELEM ", nelem_par(isubd), " NELEW ", nelew_par(isubd), " NBOUN ", nboun_par(isubd), " NNEIG ", lneig_par(isubd), " NBBOU ", npoin_par(isubd)
- if(npoin_par(isubd)%
- select(-X1) %>%
- rename(Start = S, End = E) %>%
- mutate(Duration = End - Start) %>%
- mutate(Platform = as.factor(Platform),
- Partitioning = as.factor(Partitioning),
- Nodes = as.factor(Nodes));
-
-df2 <- read_csv("data/11/326040-b8e2-41f3-8479-c378f70864d9/exp_11-v1_grimoire_8.csv.gz") %>%
- select(-X1) %>%
- rename(Start = S, End = E) %>%
- mutate(Duration = End - Start) %>%
- mutate(Platform = as.factor(Platform),
- Partitioning = as.factor(Partitioning),
- Nodes = as.factor(Nodes));
-
-df <- rbind(df1, df2);
-df %>% head;
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Parsed with column specification:
-cols(
- X1 = col_integer(),
- Rank = col_integer(),
- Iteration = col_integer(),
- Platform = col_character(),
- Nodes = col_integer(),
- NP = col_integer(),
- Partitioning = col_character(),
- EID = col_character(),
- Infiniband = col_logical(),
- N = col_integer(),
- S = col_double(),
- E = col_double(),
- Comm = col_double(),
- Comp = col_double()
-)
-Warning message:
-Missing column names filled in: 'X1' [1]
-Parsed with column specification:
-cols(
- X1 = col_integer(),
- Rank = col_integer(),
- Iteration = col_integer(),
- Platform = col_character(),
- Nodes = col_integer(),
- NP = col_integer(),
- Partitioning = col_character(),
- EID = col_character(),
- Infiniband = col_logical(),
- N = col_integer(),
- S = col_double(),
- E = col_double(),
- Comm = col_double(),
- Comp = col_double()
-)
-Warning message:
-Missing column names filled in: 'X1' [1]
-# A tibble: 6 × 14
- Rank Iteration Platform Nodes NP Partitioning EID Infiniband N
-
-1 0 1 grimoire 8 128 sfc 10-v2 TRUE 5454
-2 0 2 grimoire 8 128 sfc 10-v2 TRUE 4752
-3 0 3 grimoire 8 128 sfc 10-v2 TRUE 4873
-4 0 4 grimoire 8 128 sfc 10-v2 TRUE 4697
-5 0 5 grimoire 8 128 sfc 10-v2 TRUE 4660
-6 0 6 grimoire 8 128 sfc 10-v2 TRUE 4544
-# ... with 5 more variables: Start , End , Comm , Comp ,
-# Duration
-#+end_example
-**** Plot Comp
-#+begin_src R :results output graphics :file img/exp_10-v2_exp_11-v1_comp.png :exports both :width 500 :height 400 :session
-library(tidyr);
-library(ggplot2);
-df %>%
- select(-Start, -End, -N, -Duration) %>%
- mutate(Total = Comm+Comp) %>%
- gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID, -Infiniband) %>%
- filter(Variable == "Comp", Iteration != 10, Rank != 0) %>%
- ggplot(aes(x=as.factor(Iteration), y=Value, color=Infiniband)) +
- theme_bw(base_size=12) +
- geom_point(alpha=.3) +
- ylim(0,NA) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
- theme(legend.position="top") +
- facet_grid(EID~Partitioning)
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_10-v2_exp_11-v1_comp.png]]
-*** Partition data
-**** Process
-#+begin_src shell :results output
-EDIR="exp_11-v1_grimoire_8 exp_10-v2_grimoire_8"
-for file in $(find $EDIR | grep results | grep log$); do
- OUTPUT=$(dirname $file)/$(basename $file .log).csv
- cat $file | tr -s " " | sed "s/^ //" | tr ' ' ',' | cut -d"," -f1,3,5,7,9,11,13 | uniq > $OUTPUT
- cat $file | tr -s " " | sed "s/^ //" | tr ' ' ',' | cut -d"," -f2,4,6,8,10,12,14 >> $OUTPUT
- echo "=> $file <="
- head $OUTPUT
-done
-#+end_src
-
-#+RESULTS:
-#+begin_example
-=> exp_11-v1_grimoire_8/11-v1_grimoire_8_128_sfc_true.dir/results_NPOIN_NELEM_NELEW_NBOUN.log <=
-Rank,NPOIN,NELEM,NELEW,NBOUN,NNEIG,NBBOU
-1,32182,174168,174611,2973,16,32182
-2,20919,53529,174758,5381,15,20919
-3,19727,49972,175172,5020,11,19727
-4,21918,69049,174054,4277,23,21918
-5,21683,64345,174280,4439,14,21683
-6,22205,64122,175002,4582,18,22205
-7,21361,63432,174587,4436,11,21361
-8,22162,75699,174459,3914,8,22162
-9,21389,66339,174799,4332,8,21389
-=> exp_11-v1_grimoire_8/11-v1_grimoire_8_128_metis_true.dir/results_NPOIN_NELEM_NELEW_NBOUN.log <=
-Rank,NPOIN,NELEM,NELEW,NBOUN,NNEIG,NBBOU
-1,22405,85371,172276,3477,4,22405
-2,22923,86049,179769,3742,5,22923
-3,24316,99541,175416,3029,5,24316
-4,22907,90905,170125,3170,4,22907
-5,22636,86964,173194,3448,5,22636
-6,22774,88416,171711,3333,6,22774
-7,23564,91625,177865,3452,7,23564
-8,22211,83394,173364,3612,5,22211
-9,23427,89980,179685,3588,4,23427
-=> exp_10-v2_grimoire_8/10-v2_grimoire_8_128_sfc_true.dir/results_NPOIN_NELEM_NELEW_NBOUN.log <=
-Rank,NPOIN,NELEM,NELEW,NBOUN,NNEIG,NBBOU
-1,14976,72536,72796,2384,17,14976
-2,14641,72392,72575,448,12,14641
-3,22264,72647,167221,4446,16,22264
-4,28129,72464,251684,7187,11,28129
-5,23239,72488,178813,4341,25,23239
-6,26093,72525,209615,5640,14,26093
-7,22996,72391,183401,4422,10,22996
-8,21190,72393,160643,3505,7,21190
-9,25178,72612,201462,5195,10,25178
-=> exp_10-v2_grimoire_8/10-v2_grimoire_8_128_metis_true.dir/results_NPOIN_NELEM_NELEW_NBOUN.log <=
-Rank,NPOIN,NELEM,NELEW,NBOUN,NNEIG,NBBOU
-1,22405,85371,172276,3477,4,22405
-2,22923,86049,179769,3742,5,22923
-3,24316,99541,175416,3029,5,24316
-4,22907,90905,170125,3170,4,22907
-5,22636,86964,173194,3448,5,22636
-6,22774,88416,171711,3333,6,22774
-7,23564,91625,177865,3452,7,23564
-8,22211,83394,173364,3612,5,22211
-9,23427,89980,179685,3588,4,23427
-#+end_example
-**** Read
-#+begin_src R :results output :session :exports both
-library(readr);
-library(dplyr);
-library(tidyr);
-
-read_npoin <- function(filename)
-{
- meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[2], "_"));
- meta <- gsub(".dir", "", meta);
- read_csv(filename) %>%
- gather(Variable, Value, -Rank) %>%
- mutate(EID = meta[1],
- Platform = meta[2],
- Nodes = meta[3],
- NP = meta[4],
- Partitioning = meta[5],
- Infiniband = as.logical(meta[6]));
-
-}
-files <- list.files("exp_10-v2_grimoire_8", pattern="results_NPOIN_NELEM_NELEW_NBOUN.csv", recursive=TRUE, full.names=TRUE);
-files <- c(files, list.files("exp_11-v1_grimoire_8", pattern="results_NPOIN_NELEM_NELEW_NBOUN.csv", recursive=TRUE, full.names=TRUE));
-dfp <- do.call("rbind", lapply(files, function(x) { read_npoin(x) }))
-dfp %>% filter(Rank == 111, Variable == "NELEW")
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- NPOIN = col_integer(),
- NELEM = col_integer(),
- NELEW = col_integer(),
- NBOUN = col_integer(),
- NNEIG = col_integer(),
- NBBOU = col_integer()
-)
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- NPOIN = col_integer(),
- NELEM = col_integer(),
- NELEW = col_integer(),
- NBOUN = col_integer(),
- NNEIG = col_integer(),
- NBBOU = col_integer()
-)
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- NPOIN = col_integer(),
- NELEM = col_integer(),
- NELEW = col_integer(),
- NBOUN = col_integer(),
- NNEIG = col_integer(),
- NBBOU = col_integer()
-)
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- NPOIN = col_integer(),
- NELEM = col_integer(),
- NELEW = col_integer(),
- NBOUN = col_integer(),
- NNEIG = col_integer(),
- NBBOU = col_integer()
-)
-# A tibble: 4 × 9
- Rank Variable Value EID Platform Nodes NP Partitioning Infiniband
-
-1 111 NELEW 172833 10-v2 grimoire 8 128 metis TRUE
-2 111 NELEW 258786 10-v2 grimoire 8 128 sfc TRUE
-3 111 NELEW 172833 11-v1 grimoire 8 128 metis TRUE
-4 111 NELEW 173994 11-v1 grimoire 8 128 sfc TRUE
-#+end_example
-*** Merge
-Get only the first iteration
-#+begin_src R :results output :session :exports both
-dfz <- df %>% filter(Iteration == 1, Rank != 0) %>%
- select(-Platform, -Nodes, -NP, -Infiniband, -N, -Start, -End, -Comm, -Duration, -Iteration) %>%
- gather(Variable, Value, Comp, -EID, -Partitioning) %>%
- select(EID, Partitioning, Rank, Variable, Value);
-dfpz <- dfp %>% select(-Platform, -Nodes, -NP, -Infiniband) %>%
- select(EID, Partitioning, Rank, Variable, Value);
-dfm <- rbind(dfz, dfpz);
-dfm %>% head;
-#+end_src
-
-#+RESULTS:
-: # A tibble: 6 × 5
-: EID Partitioning Rank Variable Value
-:
-: 1 10-v2 sfc 1 Comp 8.811678
-: 2 10-v2 sfc 2 Comp 8.560127
-: 3 10-v2 sfc 3 Comp 12.641175
-: 4 10-v2 sfc 4 Comp 15.811419
-: 5 10-v2 sfc 5 Comp 13.183812
-: 6 10-v2 sfc 6 Comp 14.227162
-*** Plot
-#+begin_src R :results output :session :exports both
-dfm %>% .$Variable %>% unique
-#+end_src
-
-#+RESULTS:
-: [1] "Comp" "NPOIN" "NELEM" "NELEW" "NBOUN" "NNEIG" "NBBOU"
-
-#+begin_src R :results output graphics :file img/exp_10-v2_exp_11-v1_comp_partition.png :exports both :width 800 :height 400 :session
-dfm %>%
- mutate(EID = case_when(
- grepl("10-v2", .$EID) ~ gsub("$", "(Equal Weight)", .$EID),
- grepl("11-v1", .$EID) ~ gsub("$", "(Original Weight)", .$EID),
- TRUE ~ "Undefined")) %>%
- filter(Partitioning != "metis") %>%
- filter(Variable %in% c("Comp", "NELEM", "NELEW")) %>%
- ggplot(aes(x=Rank, y=Value, color=EID)) +
- theme_bw(base_size=12) +
- geom_point() +
- ylim(0,NA) +
- ggtitle ("SFC only, NP=128, 8-nodes@grimoire, Infiniband") +
- theme(legend.position="top") +
- facet_grid(Variable~EID, scales="free_y")
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_10-v2_exp_11-v1_comp_partition.png]]
-
-** 2017-04-07 Check alternative Alya instrumentation :EXP12:
-*** First try - no imbrication
-#+begin_src shell :results output
-ssh nancy.g5k cat ./alya-bsc/Executables/unix/all.scorep
-#+end_src
-
-#+RESULTS:
-#+begin_example
-#SCOREP_FILE_NAMES_BEGIN
-#EXCLUDE *
-#INCLUDE */Alya.f90
-#SCOREP_FILE_NAMES_END
-
-SCOREP_REGION_NAMES_BEGIN
-EXCLUDE *
-INCLUDE endste
-INCLUDE timste
-INCLUDE doiter
-INCLUDE concou
-INCLUDE conblk
-INCLUDE begzon
-INCLUDE endzon
-INCLUDE moduls
-INCLUDE nastin
-INCLUDE parall
-SCOREP_REGION_NAMES_END
-#+end_example
-
-#+begin_src shell :results output
-scp nancy.g5k:./scorep_12-v1_grimoire_6_96_metis_true/traces.csv.gz .
-#+end_src
-
-#+RESULTS:
-
-#+begin_src R :results output :session :exports both
-library(readr);
-library(dplyr);
-df <- read_csv("traces.csv.gz",
- col_names=c("Rank", "Start", "End", "Value"),
- progress=FALSE);
-df %>% group_by(Value) %>% summarize(N=n())
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- Start = col_double(),
- End = col_double(),
- Value = col_character()
-)
-# A tibble: 10 × 2
- Value N
-
-1 begzon 96
-2 conblk 96
-3 concou 96
-4 doiter 96
-5 endste 96
-6 endzon 96
-7 moduls 1248
-8 nastin 1152
-9 parall 431331
-10 timste 96
-#+end_example
-
-Let's concentrate in nastin:
-
-#+begin_src R :results output :session :exports both
-df %>% filter(Rank == 61)
-#+end_src
-
-#+RESULTS:
-#+begin_example
-# A tibble: 4,542 × 4
- Rank Start End Value
-
-1 61 0.010045 0.010054 parall
-2 61 0.012898 0.043018 parall
-3 61 0.043031 0.051135 parall
-4 61 0.051233 0.051235 moduls
-5 61 0.051251 249.036677 parall
-6 61 249.037003 249.094670 parall
-7 61 249.094764 249.094993 parall
-8 61 249.095035 249.095187 parall
-9 61 249.095202 249.095204 parall
-10 61 249.553567 249.709909 parall
-# ... with 4,532 more rows
-#+end_example
-
-I need imbrication to understand that.
-*** With imbrication, plot
-#+begin_src shell :results output
-scp nancy.g5k:./scorep_12-v1_grimoire_6_96_metis_true/traces.csv.gz .
-#+end_src
-
-#+RESULTS:
-
-#+begin_src R :results output :session :exports both
-df <- read_csv("traces.csv.gz",
- col_names=c("Nature", "Rank", "Type", "Start", "End", "Duration", "Imbrication", "Value"),
- progress=FALSE);
-df;
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Parsed with column specification:
-cols(
- Nature = col_character(),
- Rank = col_integer(),
- Type = col_character(),
- Start = col_double(),
- End = col_double(),
- Duration = col_double(),
- Imbrication = col_double(),
- Value = col_character()
-)
-# A tibble: 857,455 × 8
- Nature Rank Type Start End Duration Imbrication Value
-
-1 State 95 STATE 0.000267 0.000279 0.000012 0 parall
-2 State 95 STATE 0.000290 0.001318 0.001029 0 parall
-3 State 95 STATE 0.001326 0.032243 0.030917 0 parall
-4 State 95 STATE 0.032352 0.032354 0.000002 0 moduls
-5 State 95 STATE 0.032367 252.062136 252.029768 0 parall
-6 State 95 STATE 252.062477 252.118046 0.055569 0 parall
-7 State 95 STATE 252.118115 252.118401 0.000286 0 parall
-8 State 95 STATE 252.118442 252.118595 0.000153 0 parall
-9 State 95 STATE 252.118610 252.118644 0.000034 0 parall
-10 State 95 STATE 252.722345 252.732470 0.010126 0 parall
-# ... with 857,445 more rows
-#+end_example
-
-#+begin_src R :results output :session :exports both
-dfplot %>% .$Imbrication %>% unique %>% sort
-startTime <- df %>% filter(Value == "timste") %>% arrange(Start) %>% slice(1) %>% .$Start
-startTime
-endTime <- df %>% filter(Value == "endste") %>% arrange(End) %>% slice(n()) %>% .$End
-endTime
-dfplot %>% filter(Start > startTime, End < endTime) %>% filter(Imbrication != 10) %>% filter(Value == "moduls")
-#+end_src
-
-#+RESULTS:
-#+begin_example
-[1] 0
-[1] 259.3502
-[1] 315.6014
-# A tibble: 15 × 8
- Nature Rank Type Start End Duration Imbrication Value
-
-1 State 4 STATE 259.6966 259.6972 0.000613 0 moduls
-2 State 4 STATE 287.6941 287.7119 0.017868 0 moduls
-3 State 4 STATE 288.0547 288.0553 0.000610 0 moduls
-4 State 3 STATE 259.6966 259.6973 0.000682 0 moduls
-5 State 3 STATE 287.6947 287.7119 0.017238 0 moduls
-6 State 3 STATE 288.0546 288.0553 0.000678 0 moduls
-7 State 2 STATE 259.6966 259.6972 0.000600 0 moduls
-8 State 2 STATE 287.6937 287.7119 0.018168 0 moduls
-9 State 2 STATE 288.0546 288.0552 0.000601 0 moduls
-10 State 1 STATE 259.6966 259.6973 0.000712 0 moduls
-11 State 1 STATE 287.6949 287.7119 0.017059 0 moduls
-12 State 1 STATE 288.0547 288.0554 0.000707 0 moduls
-13 State 0 STATE 259.6968 259.6969 0.000010 0 moduls
-14 State 0 STATE 287.6883 287.7119 0.023637 0 moduls
-15 State 0 STATE 288.0548 288.0548 0.000011 0 moduls
-#+end_example
-
-#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 1200 :height 300 :session
-library(ggplot2);
-
-maxImbrication = df %>% .$Imbrication %>% max;
-plot <- NULL;
-dfplot <- df %>% filter(!(Value %in% c("timste", "endste"))) %>% filter(Rank < 5) %>% filter(Imbrication %in% c(0,1));
-dfplot %>%
- ggplot() +
- coord_cartesian(xlim=c(startTime, endTime)) +
- theme_bw(base_size=12) +
- scale_fill_brewer(palette = "Set1") -> plot;
-
-for (i in (dfplot %>% .$Imbrication %>% unique %>% sort)){
- plot <- plot +
- geom_rect(data=(dfplot %>% filter(Imbrication == i)), aes(fill=Value,
- xmin=Start,
- xmax=End,
- ymin=Rank,
- ymax=Rank+0.9-(Imbrication*(1/5))));
-}
-plot;
-#+end_src
-
-#+RESULTS:
-[[file:/tmp/babel-24853m4f/figure24853J5Y.png]]
-*** New tracing with MPI + nsi functions
-#+begin_src shell :results output
-ssh nancy.g5k cat ./alya-bsc/Executables/unix/all.scorep
-#+end_src
-
-#+RESULTS:
-#+begin_example
-#SCOREP_FILE_NAMES_BEGIN
-#EXCLUDE *
-#INCLUDE */Alya.f90
-#SCOREP_FILE_NAMES_END
-
-SCOREP_REGION_NAMES_BEGIN
-EXCLUDE *
-INCLUDE endste
-INCLUDE timste
-INCLUDE doiter
-INCLUDE moduls
-INCLUDE nastin
-INCLUDE parall
-INCLUDE nsi_doiter
-INCLUDE nsi_turnon
-INCLUDE nsi_parall
-
-SCOREP_REGION_NAMES_END
-#+end_example
-
-#+begin_src shell :results output
-scp nancy.g5k:./scorep_12-v1_grimoire_6_96_metis_true_v3/traces.csv.gz .
-#+end_src
-
-#+RESULTS:
-
-#+begin_src R :results output :session :exports both
-df <- read_csv("traces.csv.gz",
- col_names=c("Nature", "Rank", "Type", "Start", "End", "Duration", "Imbrication", "Value"),
- progress=FALSE);
-df <- df %>% mutate(Value = gsub("MPI_.*", "MPI", Value));
-df;
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Parsed with column specification:
-cols(
- Nature = col_character(),
- Rank = col_integer(),
- Type = col_character(),
- Start = col_double(),
- End = col_double(),
- Duration = col_double(),
- Imbrication = col_double(),
- Value = col_character()
-)
-# A tibble: 2,338,005 × 8
- Nature Rank Type Start End Duration Imbrication Value
-
-1 State 95 STATE 1.770337 1.770339 2.0e-06 0 MPI
-2 State 95 STATE 1.770339 1.770340 1.0e-06 0 MPI
-3 State 95 STATE 1.770348 1.770353 5.0e-06 0 MPI
-4 State 95 STATE 1.770357 1.770449 9.3e-05 0 MPI
-5 State 95 STATE 1.770450 1.770690 2.4e-04 0 MPI
-6 State 95 STATE 1.770691 1.770691 0.0e+00 0 MPI
-7 State 95 STATE 1.770691 1.770691 0.0e+00 0 MPI
-8 State 95 STATE 1.770692 1.770693 1.0e-06 0 MPI
-9 State 95 STATE 1.770693 1.770723 3.0e-05 0 MPI
-10 State 95 STATE 1.770724 1.770731 8.0e-06 0 MPI
-# ... with 2,337,995 more rows
-#+end_example
-
-#+begin_src R :results output :session :exports both
-startTime <- df %>% filter(Value == "timste") %>% arrange(Start) %>% slice(1) %>% .$End
-startTime
-endTime <- df %>% filter(Value == "endste") %>% arrange(End) %>% slice(n()) %>% .$End
-endTime
-maxImbrication = df %>% .$Imbrication %>% max;
-maxImbrication
-#+end_src
-
-#+RESULTS:
-: [1] 126.1242
-: [1] 153.975
-: [1] 5
-
-#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 1200 :height 300 :session
-library(ggplot2);
-
-plot <- NULL;
-dfplot <- df %>%
- filter(Rank != 0) %>%
- filter(!(Value %in% c("timste", "endste", "nsi_turnon"))) %>%
- filter(Imbrication != 5) %>%
- filter(Rank < 5);
-dfplot %>%
- ggplot() +
- coord_cartesian(xlim=c(startTime, endTime)) +
- xlim(startTime, endTime) +
- theme_bw(base_size=12) +
- scale_fill_brewer(palette = "Set1") -> plot;
-
-for (i in (dfplot %>% .$Imbrication %>% unique %>% sort)){
- plot <- plot +
- geom_rect(data=(dfplot %>% filter(Imbrication == i)), aes(fill=Value,
- xmin=Start,
- xmax=End,
- ymin=Rank,
- ymax=Rank+0.9-(Imbrication*(1/10))));
-}
-plot;
-#+end_src
-
-#+RESULTS:
-[[file:/tmp/babel-24853m4f/figure24853ygt.png]]
-
-Conclusions:
-- I should take the end =timste= and =endste= because =timste= is long
-- =nsi_doiter= (or the more general =doiter=) state has the calculations, but it has a lot of MPI calls within
-- =nsi_turnon= is not necessary
-
-#+begin_src R :results output :session :exports both
-df %>% filter(Rank == 4, Start > 135, End < 140) %>% filter(!(Value %in% c("timste", "endste", "doiter", "moduls")))
-#+end_src
-
-#+RESULTS:
-#+begin_example
-# A tibble: 2,674 × 8
- Nature Rank Type Start End Duration Imbrication Value
-
-1 State 4 STATE 135.0003 135.0005 0.000219 4 MPI
-2 State 4 STATE 135.0008 135.0010 0.000210 4 MPI
-3 State 4 STATE 135.0012 135.0015 0.000228 4 MPI
-4 State 4 STATE 135.0017 135.0018 0.000048 4 parall
-5 State 4 STATE 135.0018 135.0018 0.000010 5 MPI
-6 State 4 STATE 135.0018 135.0018 0.000003 5 MPI
-7 State 4 STATE 135.0018 135.0018 0.000004 5 MPI
-8 State 4 STATE 135.0018 135.0018 0.000002 5 MPI
-9 State 4 STATE 135.0018 135.0018 0.000003 5 MPI
-10 State 4 STATE 135.0018 135.0018 0.000001 5 MPI
-# ... with 2,664 more rows
-#+end_example
-
-#+begin_src R :results output :session :exports both
-df %>% filter(Imbrication == 5) %>% .$Value %>% unique
-#+end_src
-
-#+RESULTS:
-: [1] "MPI"
-
-#+begin_src R :results output :session :exports both
-df %>% filter(Value == "nsi_turnon") %>% filter(Rank == 4)
-#+end_src
-
-#+RESULTS:
-: # A tibble: 1 × 8
-: Nature Rank Type Start End Duration Imbrication Value
-:
-: 1 State 4 STATE 125.57 125.6114 0.041432 2 nsi_turnon
-*** New round
-#+begin_src shell :results output
-ssh nancy.g5k cat ./alya-bsc/Executables/unix/all.scorep
-#+end_src
-
-#+RESULTS:
-#+begin_example
-#SCOREP_FILE_NAMES_BEGIN
-#EXCLUDE *
-#INCLUDE */Alya.f90
-#SCOREP_FILE_NAMES_END
-
-SCOREP_REGION_NAMES_BEGIN
-EXCLUDE *
-# Timestep begins when rank leaves the timste state
-INCLUDE timste
-# Timestep ends when rank enters the endste
-INCLUDE endste
-# doiter is the iteration itself
-INCLUDE doiter
-
-# Nastin calls
-INCLUDE nastin
-INCLUDE nsi_turnon
-INCLUDE nsi_timste
-INCLUDE nsi_iniunk
-INCLUDE nsi_begste
-INCLUDE nsi_doiter
-INCLUDE nsi_concou
-INCLUDE nsi_conblk
-INCLUDE nsi_newmsh
-INCLUDE nsi_endste
-INCLUDE nsi_filter
-INCLUDE nsi_output
-INCLUDE nsi_turnof
-SCOREP_REGION_NAMES_END
-#+end_example
-
-
-#+begin_src shell :results output
-scp nancy.g5k:./scorep_12-v1_grimoire_6_96_metis_true_v4/traces.csv.gz .
-#+end_src
-
-#+RESULTS:
-
-#+begin_src R :results output :session :exports both
-df <- read_csv("traces.csv.gz",
- col_names=c("Nature", "Rank", "Type", "Start", "End", "Duration", "Imbrication", "Value"),
- progress=FALSE);
-df <- df %>% mutate(Value = gsub("MPI_.*", "MPI", Value));
-startTime <- df %>% filter(Value == "timste") %>% arrange(Start) %>% slice(1) %>% .$End
-startTime
-endTime <- df %>% filter(Value == "endste") %>% arrange(End) %>% slice(n()) %>% .$End
-endTime
-maxImbrication = df %>% .$Imbrication %>% max;
-maxImbrication
-df;
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Parsed with column specification:
-cols(
- Nature = col_character(),
- Rank = col_integer(),
- Type = col_character(),
- Start = col_double(),
- End = col_double(),
- Duration = col_double(),
- Imbrication = col_double(),
- Value = col_character()
-)
-[1] 123.4664
-[1] 151.6115
-[1] 3
-# A tibble: 1,906,194 × 8
- Nature Rank Type Start End Duration Imbrication Value
-
-1 State 95 STATE 1.110072 1.110075 0.000003 0 MPI
-2 State 95 STATE 1.110076 1.110077 0.000001 0 MPI
-3 State 95 STATE 1.110085 1.110090 0.000005 0 MPI
-4 State 95 STATE 1.110091 1.110191 0.000100 0 MPI
-5 State 95 STATE 1.110191 1.110424 0.000232 0 MPI
-6 State 95 STATE 1.110425 1.110425 0.000000 0 MPI
-7 State 95 STATE 1.110425 1.110425 0.000000 0 MPI
-8 State 95 STATE 1.110426 1.110427 0.000001 0 MPI
-9 State 95 STATE 1.110427 1.110463 0.000036 0 MPI
-10 State 95 STATE 1.110463 1.110470 0.000007 0 MPI
-# ... with 1,906,184 more rows
-#+end_example
-
-#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 1200 :height 300 :session
-library(ggplot2);
-
-plot <- NULL;
-dfplot <- df %>%
- filter(Rank != 0) %>%
- filter(!(Value %in% c("timste", "endste"))) %>%
- filter(Imbrication != 5) %>%
- filter(Rank < 5);
-dfplot %>%
- ggplot() +
- coord_cartesian(xlim=c(135, 145)) + #startTime, endTime)) +
-# xlim(startTime, endTime) +
- theme_bw(base_size=12) +
- scale_fill_brewer(palette = "Set1") -> plot;
-
-for (i in (dfplot %>% .$Imbrication %>% unique %>% sort)){
- plot <- plot +
- geom_rect(data=(dfplot %>% filter(Imbrication == i)), aes(fill=Value,
- xmin=Start,
- xmax=End,
- ymin=Rank,
- ymax=Rank+0.9-(Imbrication*(1/10))));
-}
-plot;
-#+end_src
-
-#+RESULTS:
-[[file:/tmp/babel-24853m4f/figure24853mXu.png]]
-
-Still not there, I can't see any function in between.
-*** New round with =nsi_solite=
-#+begin_src shell :results output
-ssh nancy.g5k cat ./alya-bsc/Executables/unix/all.scorep
-#+end_src
-
-#+RESULTS:
-#+begin_example
-#SCOREP_FILE_NAMES_BEGIN
-#EXCLUDE *
-#INCLUDE */Alya.f90
-#SCOREP_FILE_NAMES_END
-
-SCOREP_REGION_NAMES_BEGIN
-EXCLUDE *
-# Timestep begins when rank leaves the timste state
-INCLUDE timste
-# Timestep ends when rank enters the endste
-INCLUDE endste
-# doiter is the iteration itself
-INCLUDE doiter
-
-# Nastin calls
-INCLUDE nastin
-INCLUDE nsi_turnon
-INCLUDE nsi_timste
-INCLUDE nsi_iniunk
-INCLUDE nsi_begste
-INCLUDE nsi_doiter
-INCLUDE nsi_concou
-INCLUDE nsi_conblk
-INCLUDE nsi_newmsh
-INCLUDE nsi_endste
-INCLUDE nsi_filter
-INCLUDE nsi_output
-INCLUDE nsi_turnof
-
-# the solver
-INCLUDE nsi_solite
-
-SCOREP_REGION_NAMES_END
-#+end_example
-
-
-#+begin_src shell :results output
-scp nancy.g5k:./scorep_12-v1_grimoire_6_96_metis_true_v5/traces.csv.gz .
-#+end_src
-
-#+RESULTS:
-
-#+begin_src R :results output :session :exports both
-df <- read_csv("traces.csv.gz",
- col_names=c("Nature", "Rank", "Type", "Start", "End", "Duration", "Imbrication", "Value"),
- progress=FALSE);
-df <- df %>% mutate(Value = gsub("MPI_.*", "MPI", Value));
-startTime <- df %>% filter(Value == "timste") %>% arrange(Start) %>% slice(1) %>% .$End
-startTime
-endTime <- df %>% filter(Value == "endste") %>% arrange(End) %>% slice(n()) %>% .$End
-endTime
-maxImbrication = df %>% .$Imbrication %>% max;
-maxImbrication
-df;
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Parsed with column specification:
-cols(
- Nature = col_character(),
- Rank = col_integer(),
- Type = col_character(),
- Start = col_double(),
- End = col_double(),
- Duration = col_double(),
- Imbrication = col_double(),
- Value = col_character()
-)
-[1] 125.3257
-[1] 153.2368
-[1] 4
-# A tibble: 1,906,674 × 8
- Nature Rank Type Start End Duration Imbrication Value
-
-1 State 95 STATE 1.025796 1.025799 0.000002 0 MPI
-2 State 95 STATE 1.025800 1.025800 0.000001 0 MPI
-3 State 95 STATE 1.025809 1.025814 0.000005 0 MPI
-4 State 95 STATE 1.025815 1.025919 0.000104 0 MPI
-5 State 95 STATE 1.025919 1.026159 0.000240 0 MPI
-6 State 95 STATE 1.026160 1.026161 0.000000 0 MPI
-7 State 95 STATE 1.026161 1.026161 0.000000 0 MPI
-8 State 95 STATE 1.026162 1.026163 0.000001 0 MPI
-9 State 95 STATE 1.026163 1.026200 0.000037 0 MPI
-10 State 95 STATE 1.026200 1.026206 0.000005 0 MPI
-# ... with 1,906,664 more rows
-#+end_example
-
-#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 1200 :height 300 :session
-library(ggplot2);
-
-plot <- NULL;
-dfplot <- df %>%
- filter(Rank != 0) %>%
- filter(!(Value %in% c("timste", "endste", "doiter", "nastin"))) %>%
- filter(Imbrication != 5) %>%
- filter(Rank < 5);
-dfplot %>%
- ggplot() +
- coord_cartesian(xlim=c(startTime, endTime)) +
- theme_bw(base_size=12) +
- scale_fill_brewer(palette = "Set1") -> plot;
-
-for (i in (dfplot %>% .$Imbrication %>% unique %>% sort)){
- plot <- plot +
- geom_rect(data=(dfplot %>% filter(Imbrication == i)), aes(fill=Value,
- xmin=Start,
- xmax=End,
- ymin=Rank,
- ymax=Rank+0.9-(Imbrication*(1/10))));
-}
-plot;
-#+end_src
-
-#+RESULTS:
-[[file:/tmp/babel-24853m4f/figure24853ZbQ.png]]
-
-
-We have five =nsi_solite= which matches:
-
-#+BEGIN_EXAMPLE
---| ALYA SOLVE NASTIN (MSMSC)
---| ALYA SOLVE NASTIN (MSMSC)
---| ALYA SOLVE NASTIN (MSMSC)
---| ALYA SOLVE NASTIN (MSMSC)
---| ALYA SOLVE NASTIN (MSMSC)
-#+END_EXAMPLE
-
-There are still some MPI calls in between. Perhaps one of each MSMSC.
-*** With many functions called by nsi solite
-
-#+begin_src shell :results output
-ssh nancy.g5k cat ./alya-bsc/Executables/unix/all.scorep
-#+end_src
-
-#+RESULTS:
-#+begin_example
-#SCOREP_FILE_NAMES_BEGIN
-#EXCLUDE *
-#INCLUDE */Alya.f90
-#SCOREP_FILE_NAMES_END
-
-SCOREP_REGION_NAMES_BEGIN
-EXCLUDE *
-# Timestep begins when rank leaves the timste state
-INCLUDE timste
-# Timestep ends when rank enters the endste
-INCLUDE endste
-# doiter is the iteration itself
-INCLUDE doiter
-
-# Nastin calls
-INCLUDE nastin
-INCLUDE nsi_turnon
-INCLUDE nsi_timste
-INCLUDE nsi_iniunk
-INCLUDE nsi_begste
-INCLUDE nsi_doiter
-INCLUDE nsi_concou
-INCLUDE nsi_conblk
-INCLUDE nsi_newmsh
-INCLUDE nsi_endste
-INCLUDE nsi_filter
-INCLUDE nsi_output
-INCLUDE nsi_turnof
-
-# the solver
-INCLUDE nsi_solite
-
-# functions called by the nsi solver
-INCLUDE nsi_updbcs
-INCLUDE nsi_ifconf
-INCLUDE nsi_updunk
-INCLUDE nsi_solsgs
-INCLUDE nsi_inisol
-INCLUDE nsi_matrix
-INCLUDE nsimatndof
-INCLUDE nsi_agmgsol
-INCLUDE solver
-INCLUDE nsi_multi_step_fs_solution
-INCLUDE nsi_fractional_step_solution
-INCLUDE nsi_schur_complement_solution
-INCLUDE nsi_solver_postprocess
-
-SCOREP_REGION_NAMES_END
-#+end_example
-
-
-#+begin_src shell :results output
-scp nancy.g5k:./scorep_12-v1_grimoire_6_96_metis_true_v6/traces.csv.gz .
-#+end_src
-
-#+RESULTS:
-
-#+begin_src R :results output :session :exports both
-df <- read_csv("traces.csv.gz",
- col_names=c("Nature", "Rank", "Type", "Start", "End", "Duration", "Imbrication", "Value"),
- progress=FALSE);
-df <- df %>% mutate(Value = gsub("MPI_.*", "MPI", Value));
-startTime <- df %>% filter(Value == "timste") %>% arrange(Start) %>% slice(1) %>% .$End
-startTime
-endTime <- df %>% filter(Value == "endste") %>% arrange(End) %>% slice(n()) %>% .$End
-endTime
-maxImbrication = df %>% .$Imbrication %>% max;
-maxImbrication
-df;
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Parsed with column specification:
-cols(
- Nature = col_character(),
- Rank = col_integer(),
- Type = col_character(),
- Start = col_double(),
- End = col_double(),
- Duration = col_double(),
- Imbrication = col_double(),
- Value = col_character()
-)
-[1] 124.8219
-[1] 152.4296
-[1] 5
-# A tibble: 1,914,065 × 8
- Nature Rank Type Start End Duration Imbrication Value
-
-1 State 95 STATE 1.014010 1.014012 0.000002 0 MPI
-2 State 95 STATE 1.014013 1.014014 0.000001 0 MPI
-3 State 95 STATE 1.014022 1.014027 0.000004 0 MPI
-4 State 95 STATE 1.014028 1.014128 0.000100 0 MPI
-5 State 95 STATE 1.014128 1.014358 0.000229 0 MPI
-6 State 95 STATE 1.014359 1.014359 0.000000 0 MPI
-7 State 95 STATE 1.014359 1.014359 0.000000 0 MPI
-8 State 95 STATE 1.014360 1.014361 0.000001 0 MPI
-9 State 95 STATE 1.014361 1.014398 0.000037 0 MPI
-10 State 95 STATE 1.014398 1.014405 0.000007 0 MPI
-# ... with 1,914,055 more rows
-#+end_example
-
-#+begin_src R :results output :session :exports both
-df %>% select(Imbrication, Value) %>% unique %>% as.data.frame() %>% filter(Imbrication == 4)
-#+end_src
-
-#+RESULTS:
-: Imbrication Value
-: 1 4 nsi_updbcs
-: 2 4 nsi_ifconf
-: 3 4 nsi_updunk
-: 4 4 nsi_solsgs
-: 5 4 nsi_matrix
-: 6 4 solver
-: 7 4 MPI
-: 8 4 nsi_inisol
-
-#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 1400 :height 300 :session
-library(ggplot2);
-
-plot <- NULL;
-dfplot <- df %>%
- filter(Rank != 0) %>%
- filter(Value %in% c(#"MPI",
- "solver",
- #"nsi_solite",
- "nsi_matrix",
- "nsi_inisol",
- "nsi_updunk",
- "nsi_updbcs",
- "nsi_ifconf",
- "nsi_updbcs")) %>% #, "nsi_matrix")) %>%
-# filter(!(Value %in% c("timste", "endste", "doiter", "nastin", "nsi_begste", "nsi_endste", "nsi_concou", "nsi_output"))) %>%
- filter(Imbrication != 5)# %>%
-# filter(Rank < 5);
-dfplot %>%
- ggplot() +
- coord_cartesian(xlim=c(startTime, endTime)) +
- xlim(startTime, endTime) +
- theme_bw(base_size=12) +
- scale_fill_brewer(palette = "Set1") -> plot;
-
-for (i in (dfplot %>% .$Imbrication %>% unique %>% sort)){
- plot <- plot +
- geom_rect(data=(dfplot %>% filter(Imbrication == i)), aes(fill=Value,
- xmin=Start,
- xmax=End,
- ymin=Rank,
- ymax=Rank+0.9-(Imbrication*(1/10))));
-}
-plot;
-#+end_src
-
-#+RESULTS:
-[[file:/tmp/babel-24853m4f/figure24853PbT.png]]
-
-
-So, for Nastin, computation time is:
-- =nsi_matrix=
-- =solver=
-- =nsi_inisol= (hardly seen)
--
-*** Final attempt
-Check analysis of EXP12.
-*** SCOREP Filter File :ATTACH:
-:PROPERTIES:
-:Attachments: all.scorep
-:ID: ef6feaae-5f92-4706-a250-bde856794f47
-:END:
-
-#+begin_src shell :results output
-scp nancy.g5k:./alya-bsc/Executables/unix/all.scorep .
-#+end_src
-
-#+RESULTS:
-
-** 2017-05-04 Meeting D2.4 " *Report on Multilevel Programming and I/O optimizations"
-Presents: Guillaume, Rick, Lucas, Arnaud
-Link: https://rendez-vous.renater.fr/alya
-*** Introduction (Guillaume's Message)
-Guillaume's Message: I'm responsible for a deliverable of HPC4E, D2.4
-"Report on Multilevel Programming and I/O optimizations". We are going
-to report our work on SFC as a mesh partitioner and I was wondering if
-you want to report the performance analysis you did on this as
-well. No need to be extensive, one page would be okay.
-*** Discussed Topics
-**** Load imbalance (and SFC/Metis)
-When the number of points is higher, the time spent in the "solver"
-function is reduce. That surprises us because it should be the
-contrary. The number of points correspond to the number of rows in the
-matrix associated to each rank. Instead, Rick told us that we should
-look instead to the number of entries in each row; that should
-directly correlate with the workload of each rank.
-
-So we should add a new row with the number of entries associated to
-each rank: =img/exp_13_solver_partition.png=.
-
-Load imbalance is independent of Metis/SFC.
-
-At one point, when load imbalance is perfect, we shall look again to
-the communication issue we have previously seen.
-**** The one-page report
-We still have 10 days to finish the delivrable. So we can check the
-number of entries previously mentionned before writing the report.
-** 2017-05-10 Experimental session
-#+begin_src shell :results output
-oarsub -p "cluster='grisou'" -l "nodes=20,walltime=14:00:00" -r "$(date '+%Y-%m-%d 19:00:00')" -t deploy
-kadeploy3 --env-file /home/lschnorr/images/stretch_energy.env -f ${OAR_NODE_FILE} -k
-#+end_src
-** 2018-01-21 Meeting @BSC during HP4E closure
-*** Pre-meeting notes based on e-mails
-**** Partition quality using adaptation strategy: run-time measurements + unity tests (Ricard)
-The SFC partitioning process has shown to be very fast: a mesh of 30M
-elements can be partitioned in 0.05 seconds using 4K CPUs. As you say,
-next steps must be focused on the partition quality. In particular,
-I’m thinking on a partition adaptation strategy based on run-time
-measurements or some initial measurements on unity tests.
-**** Different input to check for variability (Lucas)
-Sure. Back in those days, we have instrumented the kernels that were
-used in one of the simulations to measure their run-time ie. the
-per-iteration load of each rank. The gathered information can be used
-to understand how such load evolves along the execution. As you plan,
-perhaps we can leverage on that to improve the partition quality. I
-wonder if we can play with many different inputs to check for
-variability.
-**** LB with co-execution on hybrid systems where a proper load distribution is hard to guess (Ricard+Arnaud)
-Le fait de lancer plusieurs codes (couplés ou pas) sur les mêmes
-ressources. Par exemple, dans Alya, qui fait de la multi physique, on
-peut activer plusieurs "modules" (un pour l'atmosphère, un pour
-l'océan, un pour ...) qui sont en fait des programmes autonomes mais
-couplés. Tu peux mettre un module sur une partie des ressources, un
-module sur une autre partie des ressources ou les deux modules sur
-l'ensemble des ressources. Pour moi, quand tu les met au même endroit,
-c'est un exemple de co-execution et sinon, ils se partagent quand même
-les ressources réseau. On parle de co-execution aussi au niveau batch
-scheduler quand on s'autorise à exécuter deux applis vraiment
-indépendantes sur les mêmes ressources mais je ne pense pas que ça
-soit ça dont il parle. Là, c'est plusieurs modules au sein de la même
-application et histoire de simplifier les choses ces modules utilisent
-plus ou moins les GPUs, plus ou moins en même temps selon la façon
-dont c'est programmé..
-*** Meeting
-- Partitioning is very fast, so we can run several times until the
- quality is very good. There are two approaches to improve the
- quality: the first one is based on using realtime measuremnets from
- the Alya; the second is with a unity test (mimicing the Alya
- code). The second has some disavatnages but is pretty much faster.
-- They are using a Hilbert SFC because it provides better locality.
- - Other approaches: Peano, Morton
-- The current weight: the number of elements, mass center of the element
- - Other approaches Ricard tried: more sofisticated approaches but it
- is a manual approach, very slow. A runtime approach with
- monitoring is better.
-- They read the mesh in parallel
- - Boxes and subboxes: the distribution based on SFC and the weight
- of each box is passed throught (acumulated) to other boxes.
-- Test cases:
- - sniff (17.7 M elements, 6.8M nodes); \approx1K cores (60K elements per core)
- - in memory occupation: 50K (1GBytes per core)
- - combustor (28.9 M elements, 7M nodes); \approx2K cores
- - 30 to 60 number of elements per core
-- Cartesian division from the initial global bounding box
- - Each rank reads the mesh in parallel and send to each box (another
- rank is responsible) the number of elements and their weight
- - Starts with a coarse grid, but a finer cartesion grid used for the
- SFC gives better initial LB. He does a very thin grid and its
- good.
-- How to improve the SFC. There are two options
- - You cut the line before if the weight is too much
- - Change the cartesian in some axis
- - Change the granularity of one box with a finer SFC
-- Arnaud (another approach): perturbate the partioning and collect
- measurements with multiple SFC configurations (with cuts slightly
- before and after) and then,
-
-** 2018-01-29 Prep data for Arnaud (three stable runs)
-*** Introduction
-
-We are talking about experiences:
-- EXP24 (5 iterations)
- - It is a bit strange as we previously saw
- - See figure in to remember
- #+BEGIN_EXAMPLE
- ** 4-node grisou (64 cores + 1 core) 1its, new round
- *** Visualization / Analysis
- **** Merge the cumsum of exp24 and exp25
- #+END_EXAMPLE
-- EXP25 (1 iteration)
-- EXP26 (5 iterations)
-
-*** Gather data
-
-#+name: exp24_PREP
-#+header: :var dep0=exp24_enrich
-#+header: :var dep1=exp24_number_of_elements
-#+begin_src R :results output :session :exports both
-exp.ENRICH %>% left_join(exp.ELEMENTS) %>% mutate(Exp = "exp24") -> exp24.PREP
-#+end_src
-
-#+RESULTS: exp24_PREP
-: Joining, by = "Rank"
-
-#+name: exp25_PREP
-#+header: :var dep0=exp25_enrich
-#+header: :var dep1=exp25_number_of_elements
-#+begin_src R :results output :session :exports both
-exp.ENRICH %>% left_join(exp.ELEMENTS) %>% mutate(Exp = "exp25") -> exp25.PREP
-#+end_src
-
-#+RESULTS: exp25_PREP
-: Joining, by = "Rank"
-
-#+name: exp26_PREP
-#+header: :var dep0=exp26_enrich
-#+header: :var dep1=exp26_number_of_elements
-#+begin_src R :results output :session :exports both
-exp.ENRICH %>% left_join(exp.ELEMENTS) %>% mutate(Exp = "exp26") -> exp26.PREP
-#+end_src
-
-#+RESULTS: exp26_PREP
-: Joining, by = "Rank"
-
-#+begin_src R :results output :session :exports both
-exp24.PREP %>%
- bind_rows(exp25.PREP) %>%
- bind_rows(exp26.PREP) %>%
- select(Phase, ID, Rank, NELEM,NPOIN,NBOUN,NPOI32,Exp) -> exp.PREP
-exp.PREP %>%
- select(Phase, Exp) %>% unique
-#+end_src
-
-#+RESULTS:
-#+begin_example
-# A tibble: 11 x 2
- Phase Exp
-
- 1 1 exp24
- 2 2 exp24
- 3 3 exp24
- 4 4 exp24
- 5 5 exp24
- 6 1 exp25
- 7 1 exp26
- 8 2 exp26
- 9 3 exp26
-10 4 exp26
-11 5 exp26
-#+end_example
-
-*** Save the data here :ATTACH:
-:PROPERTIES:
-:Attachments: exp24-exp25-exp26.csv.gz
-:ID: 2c325c57-678e-491d-a7d4-f71be90dbd73
-:END:
-
-#+begin_src R :results output :session :exports both
-write_csv(exp.PREP, "exp24-exp25-exp26.csv.gz")
-#+end_src
-
-#+RESULTS:
-
-** 2018-02-06 New hypothesis Check
-*** PAST: ping-pong effects can be avoided by learning from the past (many rounds)
-
-Arnaud.
-
-*** GRANULARITY: a finer SFC granularity can reveal necessary details to LB
-
-10 iterations of the test case with 128, 256, and 512 workers, for 20
-refinement rounds, with granularity levels for each partitioner
-sub-box of 128_ip (the current def), 256_ip, and 512_ip. Please, let me
-know if you think that is enough. I also intend to keep CRITERIA
-equals to 1 (unchanged) to avoid changing everything at once, except
-if I can allocated another cluster for that specific test.
-
-*** CRITERIA: use a different =CRITERIA = 2_ip= so we consider weighted elements
-
-* Experiments
-** 2017-03-31 Grisou experiments
-#+begin_src shell :results output
-SCRIPT=grisou-script.sh
-rm -f ${SCRIPT}
-echo 'export PATH=$PATH:$HOME/install/nova/scorep-3.0-alya/bin' >> ${SCRIPT}
-echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/install/nova/scorep-3.0-alya/lib' >> ${SCRIPT}
-for CASE in sfc metis; do
- for NP in 127 112 96 80 64 48 32 16; do
- NP=$((NP+1))
- UNIQUE=taurus-np${NP}-${CASE}
- echo "./scripts/run_alya_experiment.sh machine-file ${NP} ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE"
- echo "mv $UNIQUE /home/lschnorr/"
- done
-done >> ${SCRIPT}
-cat ${SCRIPT}
-#+end_src
-
-#+RESULTS:
-#+begin_example
-export PATH=$PATH:$HOME/install/nova/scorep-3.0-alya/bin
-export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/install/nova/scorep-3.0-alya/lib
-./scripts/run_alya_experiment.sh machine-file 128 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np128-sfc
-mv taurus-np128-sfc /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 113 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np113-sfc
-mv taurus-np113-sfc /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 97 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np97-sfc
-mv taurus-np97-sfc /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 81 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np81-sfc
-mv taurus-np81-sfc /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 65 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np65-sfc
-mv taurus-np65-sfc /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 49 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np49-sfc
-mv taurus-np49-sfc /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 33 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np33-sfc
-mv taurus-np33-sfc /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 17 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_sfc/fensap.dat taurus-np17-sfc
-mv taurus-np17-sfc /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 128 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np128-metis
-mv taurus-np128-metis /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 113 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np113-metis
-mv taurus-np113-metis /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 97 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np97-metis
-mv taurus-np97-metis /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 81 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np81-metis
-mv taurus-np81-metis /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 65 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np65-metis
-mv taurus-np65-metis /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 49 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np49-metis
-mv taurus-np49-metis /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 33 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np33-metis
-mv taurus-np33-metis /home/lschnorr/
-./scripts/run_alya_experiment.sh machine-file 17 ~/alya-bsc/Executables/unix/Alya.x ~/WORK-RICARD/resp_metis/fensap.dat taurus-np17-metis
-mv taurus-np17-metis /home/lschnorr/
-#+end_example
-
-Those experiments have failed.
-
-** 2017-03-31 2-node grisou debugging experiments :EXP1:
-
-Job 1224689 @ Nancy
-
-#+begin_src shell :results output
-git clone https://gitlab.inria.fr/schnorr/Alya-Perf.git
-#+end_src
-
-Requirements
-
-- The mesh: =$HOME/node_denis/=
-- The case study: =$HOME/WORK-RICARD/=
-- Scorep 3.0 installed at: =$HOME/install/nova/scorep-3.0-alya/=
-- Alya: =$HOME/alya-bsc/Executables/unix/Alya.x=
- - Instrumented with the aid of scorep
- #+BEGIN_EXAMPLE
- SCOREP=scorep --compiler --instrument-filter=/home/lschnorr/alya-bsc/Executables/unix/alya-timestep-filters.scorep
- #+END_EXAMPLE
- Contents of the filter
- #+BEGIN_EXAMPLE
-SCOREP_REGION_NAMES_BEGIN
-EXCLUDE *
-INCLUDE endste
-INCLUDE timste
-SCOREP_REGION_NAMES_END
- #+END_EXAMPLE
-
-Experimental design
-
-#+begin_src shell :results output
-CLUSTER=grisou
-NODES=2
-MACHINEFILE=machine-file
-SCRIPT="script_${CLUSTER}_${NODES}_nodes.sh"
-rm -f ${SCRIPT}
-for NP in 31 16; do
- for CASE in sfc metis; do
- NPN=$((NP+1))
- UNIQUE=${CLUSTER}_${NODES}_${NPN}_${CASE}
- echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE"
- echo "mv $UNIQUE.dir /home/lschnorr/"
- done
-done >> ${SCRIPT}
-chmod 755 ${SCRIPT}
-cat ${SCRIPT}
-#+end_src
-
-#+RESULTS:
-: ./scripts/run_alya_experiment.sh machine-file 32 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat grisou_2_32_sfc
-: mv grisou_2_32_sfc.dir /home/lschnorr/
-: ./scripts/run_alya_experiment.sh machine-file 32 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat grisou_2_32_metis
-: mv grisou_2_32_metis.dir /home/lschnorr/
-: ./scripts/run_alya_experiment.sh machine-file 17 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat grisou_2_17_sfc
-: mv grisou_2_17_sfc.dir /home/lschnorr/
-: ./scripts/run_alya_experiment.sh machine-file 17 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat grisou_2_17_metis
-: mv grisou_2_17_metis.dir /home/lschnorr/
-** 2017-03-31 10-node grisou debugging experiments :EXP2:EXP3:
-
-Same as previous entry, but with 10 nodes.
-
-Job 1224741 @ Nancy
-
-Kadeploy the image:
-=/home/lschnorr/images/stretch_energy_v6_2017_03_30.tgz=
-
-Same requirements as previous entry.
-
-Experimental design
-
-#+begin_src shell :results output
-EID=3
-CLUSTER=grisou
-NODES=10
-MACHINEFILE=machine-file
-G5KHOME=/home/lschnorr/
-SCRIPT="script_${CLUSTER}_${NODES}_nodes.sh"
-
-rm -f $SCRIPT
-
-EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
-echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
-echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
-echo " exit" >> $SCRIPT
-echo "fi" >> $SCRIPT
-
-# Create experimental directory in the home
-echo "mkdir -p $EDIR" >> $SCRIPT
-
-# The design
-for NP in 159 144 128 112 96; do
- for CASE in sfc metis; do
- NPN=$((NP+1))
- UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}
- echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE"
- echo "cp -prfv $UNIQUE.dir ${EDIR}"
- done
-done >> ${SCRIPT}
-echo "mv $SCRIPT $EDIR" >> $SCRIPT
-chmod 755 ${SCRIPT}
-cat ${SCRIPT}
-echo "This script is at '${SCRIPT}', commit it."
-#+end_src
-
-#+RESULTS:
-#+begin_example
-if [ -e /home/lschnorr//exp_3_grisou_10 ]; then
- echo "Directory/File '/home/lschnorr//exp_3_grisou_10' already exists. Remove it or rename it."
- exit
-fi
-mkdir -p /home/lschnorr//exp_3_grisou_10
-./scripts/run_alya_experiment.sh machine-file 160 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 3_grisou_10_160_sfc
-cp -prfv 3_grisou_10_160_sfc.dir /home/lschnorr//exp_3_grisou_10
-./scripts/run_alya_experiment.sh machine-file 160 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 3_grisou_10_160_metis
-cp -prfv 3_grisou_10_160_metis.dir /home/lschnorr//exp_3_grisou_10
-./scripts/run_alya_experiment.sh machine-file 145 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 3_grisou_10_145_sfc
-cp -prfv 3_grisou_10_145_sfc.dir /home/lschnorr//exp_3_grisou_10
-./scripts/run_alya_experiment.sh machine-file 145 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 3_grisou_10_145_metis
-cp -prfv 3_grisou_10_145_metis.dir /home/lschnorr//exp_3_grisou_10
-./scripts/run_alya_experiment.sh machine-file 129 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 3_grisou_10_129_sfc
-cp -prfv 3_grisou_10_129_sfc.dir /home/lschnorr//exp_3_grisou_10
-./scripts/run_alya_experiment.sh machine-file 129 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 3_grisou_10_129_metis
-cp -prfv 3_grisou_10_129_metis.dir /home/lschnorr//exp_3_grisou_10
-./scripts/run_alya_experiment.sh machine-file 113 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 3_grisou_10_113_sfc
-cp -prfv 3_grisou_10_113_sfc.dir /home/lschnorr//exp_3_grisou_10
-./scripts/run_alya_experiment.sh machine-file 113 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 3_grisou_10_113_metis
-cp -prfv 3_grisou_10_113_metis.dir /home/lschnorr//exp_3_grisou_10
-./scripts/run_alya_experiment.sh machine-file 97 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 3_grisou_10_97_sfc
-cp -prfv 3_grisou_10_97_sfc.dir /home/lschnorr//exp_3_grisou_10
-./scripts/run_alya_experiment.sh machine-file 97 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 3_grisou_10_97_metis
-cp -prfv 3_grisou_10_97_metis.dir /home/lschnorr//exp_3_grisou_10
-mv script_grisou_10_nodes.sh /home/lschnorr//exp_3_grisou_10
-This script is at 'script_grisou_10_nodes.sh', commit it.
-#+end_example
-
-Faster way to copy is to tar (without compression) all files, and then
-copy using (not the g5kg)
-#+BEGIN_EXAMPLE
-rsync --progress grisou-51.nancy.g5kg:~/exp_2_grisou_10.tar .
-#+END_EXAMPLE
-
-Two replications have been made:
-- exp 2
-- exp 3
-** 2017-04-03 23-node (368 cores) nova in April 3rd :EXP4:
-*** Preparation for nova (prep-4-v[12])
-
-#+begin_src shell :results output
-EID=prep-4
-CLUSTER=grisou
-NODES=4
-MACHINEFILE=machine-file
-G5KHOME=/home/lschnorr/
-SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
-
-rm -f $SCRIPT
-
-EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
-echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
-echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
-echo " exit" >> $SCRIPT
-echo "fi" >> $SCRIPT
-
-# Create experimental directory in the home
-echo "mkdir -p $EDIR" >> $SCRIPT
-
-# The design
-for NP in 62 48 32 17; do
- for CASE in sfc metis; do
- NPN=$((NP+1))
- UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}
- echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE"
- echo "cp -prfv $UNIQUE.dir ${EDIR}"
- done
-done >> ${SCRIPT}
-echo "mv $SCRIPT $EDIR" >> $SCRIPT
-chmod 755 ${SCRIPT}
-cat ${SCRIPT}
-echo "This script is at '${SCRIPT}', commit it."
-#+end_src
-
-#+RESULTS:
-#+begin_example
-if [ -e /home/lschnorr//exp_prep-4_grisou_4 ]; then
- echo "Directory/File '/home/lschnorr//exp_prep-4_grisou_4' already exists. Remove it or rename it."
- exit
-fi
-mkdir -p /home/lschnorr//exp_prep-4_grisou_4
-./scripts/run_alya_experiment.sh machine-file 63 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat prep-4_grisou_4_63_sfc
-cp -prfv prep-4_grisou_4_63_sfc.dir /home/lschnorr//exp_prep-4_grisou_4
-./scripts/run_alya_experiment.sh machine-file 63 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat prep-4_grisou_4_63_metis
-cp -prfv prep-4_grisou_4_63_metis.dir /home/lschnorr//exp_prep-4_grisou_4
-./scripts/run_alya_experiment.sh machine-file 49 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat prep-4_grisou_4_49_sfc
-cp -prfv prep-4_grisou_4_49_sfc.dir /home/lschnorr//exp_prep-4_grisou_4
-./scripts/run_alya_experiment.sh machine-file 49 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat prep-4_grisou_4_49_metis
-cp -prfv prep-4_grisou_4_49_metis.dir /home/lschnorr//exp_prep-4_grisou_4
-./scripts/run_alya_experiment.sh machine-file 33 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat prep-4_grisou_4_33_sfc
-cp -prfv prep-4_grisou_4_33_sfc.dir /home/lschnorr//exp_prep-4_grisou_4
-./scripts/run_alya_experiment.sh machine-file 33 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat prep-4_grisou_4_33_metis
-cp -prfv prep-4_grisou_4_33_metis.dir /home/lschnorr//exp_prep-4_grisou_4
-./scripts/run_alya_experiment.sh machine-file 18 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat prep-4_grisou_4_18_sfc
-cp -prfv prep-4_grisou_4_18_sfc.dir /home/lschnorr//exp_prep-4_grisou_4
-./scripts/run_alya_experiment.sh machine-file 18 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat prep-4_grisou_4_18_metis
-cp -prfv prep-4_grisou_4_18_metis.dir /home/lschnorr//exp_prep-4_grisou_4
-mv script_prep-4_grisou_4_nodes.sh /home/lschnorr//exp_prep-4_grisou_4
-This script is at 'script_prep-4_grisou_4_nodes.sh', commit it.
-#+end_example
-*** Full 23-node nova
-
-JobId 854496 @ Lyon.
-
-#+begin_src shell :results output
-EID=4
-CLUSTER=grisou
-NODES=23
-MACHINEFILE=machine-file
-G5KHOME=/home/lschnorr/
-SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
-
-rm -f $SCRIPT
-
-EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
-echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
-echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
-echo " exit" >> $SCRIPT
-echo "fi" >> $SCRIPT
-
-# Create experimental directory in the home
-echo "mkdir -p $EDIR" >> $SCRIPT
-
-# The design
-for NP in 367 352 336 272 208 144 80 16; do
- for CASE in sfc metis; do
- NPN=$((NP+1))
- UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}
- echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE"
- echo "cp -prfv $UNIQUE.dir ${EDIR}"
- done
-done >> ${SCRIPT}
-echo "cp $SCRIPT $EDIR" >> $SCRIPT
-chmod 755 ${SCRIPT}
-cat ${SCRIPT}
-echo "This script is at '${SCRIPT}', commit it."
-#+end_src
-
-#+RESULTS:
-#+begin_example
-if [ -e /home/lschnorr//exp_4_grisou_23 ]; then
- echo "Directory/File '/home/lschnorr//exp_4_grisou_23' already exists. Remove it or rename it."
- exit
-fi
-mkdir -p /home/lschnorr//exp_4_grisou_23
-./scripts/run_alya_experiment.sh machine-file 368 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 4_grisou_23_368_sfc
-cp -prfv 4_grisou_23_368_sfc.dir /home/lschnorr//exp_4_grisou_23
-./scripts/run_alya_experiment.sh machine-file 368 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 4_grisou_23_368_metis
-cp -prfv 4_grisou_23_368_metis.dir /home/lschnorr//exp_4_grisou_23
-./scripts/run_alya_experiment.sh machine-file 353 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 4_grisou_23_353_sfc
-cp -prfv 4_grisou_23_353_sfc.dir /home/lschnorr//exp_4_grisou_23
-./scripts/run_alya_experiment.sh machine-file 353 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 4_grisou_23_353_metis
-cp -prfv 4_grisou_23_353_metis.dir /home/lschnorr//exp_4_grisou_23
-./scripts/run_alya_experiment.sh machine-file 337 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 4_grisou_23_337_sfc
-cp -prfv 4_grisou_23_337_sfc.dir /home/lschnorr//exp_4_grisou_23
-./scripts/run_alya_experiment.sh machine-file 337 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 4_grisou_23_337_metis
-cp -prfv 4_grisou_23_337_metis.dir /home/lschnorr//exp_4_grisou_23
-./scripts/run_alya_experiment.sh machine-file 273 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 4_grisou_23_273_sfc
-cp -prfv 4_grisou_23_273_sfc.dir /home/lschnorr//exp_4_grisou_23
-./scripts/run_alya_experiment.sh machine-file 273 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 4_grisou_23_273_metis
-cp -prfv 4_grisou_23_273_metis.dir /home/lschnorr//exp_4_grisou_23
-./scripts/run_alya_experiment.sh machine-file 209 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 4_grisou_23_209_sfc
-cp -prfv 4_grisou_23_209_sfc.dir /home/lschnorr//exp_4_grisou_23
-./scripts/run_alya_experiment.sh machine-file 209 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 4_grisou_23_209_metis
-cp -prfv 4_grisou_23_209_metis.dir /home/lschnorr//exp_4_grisou_23
-./scripts/run_alya_experiment.sh machine-file 145 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 4_grisou_23_145_sfc
-cp -prfv 4_grisou_23_145_sfc.dir /home/lschnorr//exp_4_grisou_23
-./scripts/run_alya_experiment.sh machine-file 145 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 4_grisou_23_145_metis
-cp -prfv 4_grisou_23_145_metis.dir /home/lschnorr//exp_4_grisou_23
-./scripts/run_alya_experiment.sh machine-file 81 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 4_grisou_23_81_sfc
-cp -prfv 4_grisou_23_81_sfc.dir /home/lschnorr//exp_4_grisou_23
-./scripts/run_alya_experiment.sh machine-file 81 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 4_grisou_23_81_metis
-cp -prfv 4_grisou_23_81_metis.dir /home/lschnorr//exp_4_grisou_23
-./scripts/run_alya_experiment.sh machine-file 17 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 4_grisou_23_17_sfc
-cp -prfv 4_grisou_23_17_sfc.dir /home/lschnorr//exp_4_grisou_23
-./scripts/run_alya_experiment.sh machine-file 17 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 4_grisou_23_17_metis
-cp -prfv 4_grisou_23_17_metis.dir /home/lschnorr//exp_4_grisou_23
-cp script_4_grisou_23_nodes.sh /home/lschnorr//exp_4_grisou_23
-This script is at 'script_4_grisou_23_nodes.sh', commit it.
-#+end_example
-
-** 2017-04-03 50-node (800 cores) grisou in April 4th :EXP5:
-#+begin_src shell :results output
-EID=5
-VERSION=1
-CLUSTER=grisou
-NODES=50
-MACHINEFILE=machine-file
-G5KHOME=/home/lschnorr/
-SCRIPT="script_${EID}_v${VERSION}_${CLUSTER}_${NODES}_nodes.sh"
-
-rm -f $SCRIPT
-
-EDIR="$G5KHOME/exp_${EID}-v${VERSION}_${CLUSTER}_${NODES}"
-echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
-echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
-echo " exit" >> $SCRIPT
-echo "fi" >> $SCRIPT
-
-# Create experimental directory in the home
-echo "mkdir -p $EDIR" >> $SCRIPT
-
-# The design
-for NP in 800 720 640 560 480 400; do
- for CASE in sfc metis; do
- NPN=$((NP))
- UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}
- echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE false"
- echo "cp -prfv $UNIQUE.dir ${EDIR}"
- done
-done >> ${SCRIPT}
-echo "cp $SCRIPT $EDIR" >> $SCRIPT
-chmod 755 ${SCRIPT}
-cat ${SCRIPT}
-echo "This script is at '${SCRIPT}', commit it."
-#+end_src
-
-#+RESULTS:
-#+begin_example
-if [ -e /home/lschnorr//exp_5-v1_grisou_50 ]; then
- echo "Directory/File '/home/lschnorr//exp_5-v1_grisou_50' already exists. Remove it or rename it."
- exit
-fi
-mkdir -p /home/lschnorr//exp_5-v1_grisou_50
-./scripts/run_alya_experiment.sh machine-file 800 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 5_grisou_50_800_sfc false
-cp -prfv 5_grisou_50_800_sfc.dir /home/lschnorr//exp_5-v1_grisou_50
-./scripts/run_alya_experiment.sh machine-file 800 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 5_grisou_50_800_metis false
-cp -prfv 5_grisou_50_800_metis.dir /home/lschnorr//exp_5-v1_grisou_50
-./scripts/run_alya_experiment.sh machine-file 720 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 5_grisou_50_720_sfc false
-cp -prfv 5_grisou_50_720_sfc.dir /home/lschnorr//exp_5-v1_grisou_50
-./scripts/run_alya_experiment.sh machine-file 720 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 5_grisou_50_720_metis false
-cp -prfv 5_grisou_50_720_metis.dir /home/lschnorr//exp_5-v1_grisou_50
-./scripts/run_alya_experiment.sh machine-file 640 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 5_grisou_50_640_sfc false
-cp -prfv 5_grisou_50_640_sfc.dir /home/lschnorr//exp_5-v1_grisou_50
-./scripts/run_alya_experiment.sh machine-file 640 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 5_grisou_50_640_metis false
-cp -prfv 5_grisou_50_640_metis.dir /home/lschnorr//exp_5-v1_grisou_50
-./scripts/run_alya_experiment.sh machine-file 560 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 5_grisou_50_560_sfc false
-cp -prfv 5_grisou_50_560_sfc.dir /home/lschnorr//exp_5-v1_grisou_50
-./scripts/run_alya_experiment.sh machine-file 560 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 5_grisou_50_560_metis false
-cp -prfv 5_grisou_50_560_metis.dir /home/lschnorr//exp_5-v1_grisou_50
-./scripts/run_alya_experiment.sh machine-file 480 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 5_grisou_50_480_sfc false
-cp -prfv 5_grisou_50_480_sfc.dir /home/lschnorr//exp_5-v1_grisou_50
-./scripts/run_alya_experiment.sh machine-file 480 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 5_grisou_50_480_metis false
-cp -prfv 5_grisou_50_480_metis.dir /home/lschnorr//exp_5-v1_grisou_50
-./scripts/run_alya_experiment.sh machine-file 400 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 5_grisou_50_400_sfc false
-cp -prfv 5_grisou_50_400_sfc.dir /home/lschnorr//exp_5-v1_grisou_50
-./scripts/run_alya_experiment.sh machine-file 400 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 5_grisou_50_400_metis false
-cp -prfv 5_grisou_50_400_metis.dir /home/lschnorr//exp_5-v1_grisou_50
-cp script_5_v1_grisou_50_nodes.sh /home/lschnorr//exp_5-v1_grisou_50
-This script is at 'script_5_v1_grisou_50_nodes.sh', commit it.
-#+end_example
-
-** 2017-04-03 7-node (112 cores) grimoire with Infiniband :EXP6:
-#+begin_src shell :results output
-EID=6
-CLUSTER=grimoire
-NODES=7
-MACHINEFILE=machine-file
-G5KHOME=/home/lschnorr/
-SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
-
-rm -f $SCRIPT
-
-EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
-echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
-echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
-echo " exit" >> $SCRIPT
-echo "fi" >> $SCRIPT
-
-# Create experimental directory in the home
-echo "mkdir -p $EDIR" >> $SCRIPT
-
-# The design
-for NP in 111 96; do
- for CASE in sfc metis; do
- NPN=$((NP+1))
- UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}
- echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE true"
- echo "cp -prfv $UNIQUE.dir ${EDIR}"
- done
-done >> ${SCRIPT}
-echo "cp $SCRIPT $EDIR" >> $SCRIPT
-chmod 755 ${SCRIPT}
-cat ${SCRIPT}
-echo "This script is at '${SCRIPT}', commit it."
-#+end_src
-
-#+RESULTS:
-#+begin_example
-if [ -e /home/lschnorr//exp_6_grimoire_7 ]; then
- echo "Directory/File '/home/lschnorr//exp_6_grimoire_7' already exists. Remove it or rename it."
- exit
-fi
-mkdir -p /home/lschnorr//exp_6_grimoire_7
-./scripts/run_alya_experiment.sh machine-file 112 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 6_grimoire_7_112_sfc true
-cp -prfv 6_grimoire_7_112_sfc.dir /home/lschnorr//exp_6_grimoire_7
-./scripts/run_alya_experiment.sh machine-file 112 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 6_grimoire_7_112_metis true
-cp -prfv 6_grimoire_7_112_metis.dir /home/lschnorr//exp_6_grimoire_7
-./scripts/run_alya_experiment.sh machine-file 97 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 6_grimoire_7_97_sfc true
-cp -prfv 6_grimoire_7_97_sfc.dir /home/lschnorr//exp_6_grimoire_7
-./scripts/run_alya_experiment.sh machine-file 97 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 6_grimoire_7_97_metis true
-cp -prfv 6_grimoire_7_97_metis.dir /home/lschnorr//exp_6_grimoire_7
-cp script_6_grimoire_7_nodes.sh /home/lschnorr//exp_6_grimoire_7
-This script is at 'script_6_grimoire_7_nodes.sh', commit it.
-#+end_example
-
-** 2017-04-04 4-node (64 cores) grimoire with Infiniband :EXP7:
-
-The goal is simply confirm experimentally that OpenMPI is doing openib
-communications instead of ethernet when told so.
-
-#+begin_src shell :results output
-EID=7
-CLUSTER=grimoire
-NODES=4
-MACHINEFILE=machine-file
-G5KHOME=/home/lschnorr/
-SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
-
-rm -f $SCRIPT
-
-EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
-echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
-echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
-echo " exit" >> $SCRIPT
-echo "fi" >> $SCRIPT
-
-# Create experimental directory in the home
-echo "mkdir -p $EDIR" >> $SCRIPT
-
-# The design
-NPN=112
-for INFINIBAND in true false; do
- for CASE in sfc metis; do
- UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}
- echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
- echo "cp -prfv $UNIQUE.dir ${EDIR}"
- done
-done >> ${SCRIPT}
-echo "cp $SCRIPT $EDIR" >> $SCRIPT
-chmod 755 ${SCRIPT}
-cat ${SCRIPT}
-echo "This script is at '${SCRIPT}', commit it."
-#+end_src
-
-#+RESULTS:
-#+begin_example
-if [ -e /home/lschnorr//exp_7_grimoire_4 ]; then
- echo "Directory/File '/home/lschnorr//exp_7_grimoire_4' already exists. Remove it or rename it."
- exit
-fi
-mkdir -p /home/lschnorr//exp_7_grimoire_4
-./scripts/run_alya_experiment.sh machine-file 112 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 7_grimoire_4_112_sfc true
-cp -prfv 7_grimoire_4_112_sfc.dir /home/lschnorr//exp_7_grimoire_4
-./scripts/run_alya_experiment.sh machine-file 112 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 7_grimoire_4_112_metis true
-cp -prfv 7_grimoire_4_112_metis.dir /home/lschnorr//exp_7_grimoire_4
-./scripts/run_alya_experiment.sh machine-file 112 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 7_grimoire_4_112_sfc false
-cp -prfv 7_grimoire_4_112_sfc.dir /home/lschnorr//exp_7_grimoire_4
-./scripts/run_alya_experiment.sh machine-file 112 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 7_grimoire_4_112_metis false
-cp -prfv 7_grimoire_4_112_metis.dir /home/lschnorr//exp_7_grimoire_4
-cp script_7_grimoire_4_nodes.sh /home/lschnorr//exp_7_grimoire_4
-This script is at 'script_7_grimoire_4_nodes.sh', commit it.
-#+end_example
-
-** 2017-04-05 2-node (32 cores) and 4-node (64 cores) grimoire to test Infiniband :EXP8:
-#+begin_src shell :results output
-EID=8
-VERSION=4
-CLUSTER=grimoire
-NODES=4
-MACHINEFILE=machine-file
-G5KHOME=/home/lschnorr/
-EID=${EID}-v${VERSION}
-SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
-
-rm -f $SCRIPT
-
-EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
-echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
-echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
-echo " exit" >> $SCRIPT
-echo "fi" >> $SCRIPT
-
-# Create experimental directory in the home
-echo "mkdir -p $EDIR" >> $SCRIPT
-
-# The design
-NP=60
-for INFINIBAND in true false; do
- for CASE in metis sfc; do
- NPN=$((NP))
- UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}_${INFINIBAND}
- echo "# "
- echo "# Experiment $UNIQUE"
- echo "# "
- echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
- echo "cp -prfv $UNIQUE.dir ${EDIR}"
- echo "rm -rf $UNIQUE.dir"
- done
-done >> ${SCRIPT}
-echo "# " >> ${SCRIPT}
-echo "# Final" >> ${SCRIPT}
-echo "# " >> ${SCRIPT}
-echo "cp $SCRIPT $EDIR" >> $SCRIPT
-chmod 755 ${SCRIPT}
-cat ${SCRIPT}
-echo "This script is at '${SCRIPT}', commit it."
-#+end_src
-
-#+RESULTS:
-#+begin_example
-if [ -e /home/lschnorr//exp_8-v4_grimoire_4 ]; then
- echo "Directory/File '/home/lschnorr//exp_8-v4_grimoire_4' already exists. Remove it or rename it."
- exit
-fi
-mkdir -p /home/lschnorr//exp_8-v4_grimoire_4
-#
-# Experiment 8-v4_grimoire_4_60_metis_true
-#
-./scripts/run_alya_experiment.sh machine-file 60 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 8-v4_grimoire_4_60_metis_true true
-cp -prfv 8-v4_grimoire_4_60_metis_true.dir /home/lschnorr//exp_8-v4_grimoire_4
-rm -rf 8-v4_grimoire_4_60_metis_true.dir
-#
-# Experiment 8-v4_grimoire_4_60_sfc_true
-#
-./scripts/run_alya_experiment.sh machine-file 60 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 8-v4_grimoire_4_60_sfc_true true
-cp -prfv 8-v4_grimoire_4_60_sfc_true.dir /home/lschnorr//exp_8-v4_grimoire_4
-rm -rf 8-v4_grimoire_4_60_sfc_true.dir
-#
-# Experiment 8-v4_grimoire_4_60_metis_false
-#
-./scripts/run_alya_experiment.sh machine-file 60 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 8-v4_grimoire_4_60_metis_false false
-cp -prfv 8-v4_grimoire_4_60_metis_false.dir /home/lschnorr//exp_8-v4_grimoire_4
-rm -rf 8-v4_grimoire_4_60_metis_false.dir
-#
-# Experiment 8-v4_grimoire_4_60_sfc_false
-#
-./scripts/run_alya_experiment.sh machine-file 60 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 8-v4_grimoire_4_60_sfc_false false
-cp -prfv 8-v4_grimoire_4_60_sfc_false.dir /home/lschnorr//exp_8-v4_grimoire_4
-rm -rf 8-v4_grimoire_4_60_sfc_false.dir
-#
-# Final
-#
-cp script_8-v4_grimoire_4_nodes.sh /home/lschnorr//exp_8-v4_grimoire_4
-This script is at 'script_8-v4_grimoire_4_nodes.sh', commit it.
-#+end_example
-
-** 2017-04-05 6-node grimoire with Infiniband :EXP9:
-#+begin_src shell :results output
-EID=9
-VERSION=1
-CLUSTER=grimoire
-NODES=6
-MACHINEFILE=machine-file
-G5KHOME=/home/lschnorr/
-EID=${EID}-v${VERSION}
-SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
-
-rm -f $SCRIPT
-
-EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
-echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
-echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
-echo " exit" >> $SCRIPT
-echo "fi" >> $SCRIPT
-
-# Create experimental directory in the home
-echo "mkdir -p $EDIR" >> $SCRIPT
-
-# The design
-NP=96
-for INFINIBAND in true false; do
- for CASE in metis sfc; do
- NPN=$((NP))
- UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}_${INFINIBAND}
- echo "# "
- echo "# Experiment $UNIQUE"
- echo "# "
- echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
- echo "cp -prfv $UNIQUE.dir ${EDIR}"
- echo "rm -rf $UNIQUE.dir"
- done
-done >> ${SCRIPT}
-echo "# " >> ${SCRIPT}
-echo "# Final" >> ${SCRIPT}
-echo "# " >> ${SCRIPT}
-echo "cp $SCRIPT $EDIR" >> $SCRIPT
-chmod 755 ${SCRIPT}
-cat ${SCRIPT}
-echo "This script is at '${SCRIPT}', commit it."
-#+end_src
-
-#+RESULTS:
-#+begin_example
-if [ -e /home/lschnorr//exp_9-v1_grimoire_6 ]; then
- echo "Directory/File '/home/lschnorr//exp_9-v1_grimoire_6' already exists. Remove it or rename it."
- exit
-fi
-mkdir -p /home/lschnorr//exp_9-v1_grimoire_6
-#
-# Experiment 9-v1_grimoire_6_96_metis_true
-#
-./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 9-v1_grimoire_6_96_metis_true true
-cp -prfv 9-v1_grimoire_6_96_metis_true.dir /home/lschnorr//exp_9-v1_grimoire_6
-rm -rf 9-v1_grimoire_6_96_metis_true.dir
-#
-# Experiment 9-v1_grimoire_6_96_sfc_true
-#
-./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 9-v1_grimoire_6_96_sfc_true true
-cp -prfv 9-v1_grimoire_6_96_sfc_true.dir /home/lschnorr//exp_9-v1_grimoire_6
-rm -rf 9-v1_grimoire_6_96_sfc_true.dir
-#
-# Experiment 9-v1_grimoire_6_96_metis_false
-#
-./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 9-v1_grimoire_6_96_metis_false false
-cp -prfv 9-v1_grimoire_6_96_metis_false.dir /home/lschnorr//exp_9-v1_grimoire_6
-rm -rf 9-v1_grimoire_6_96_metis_false.dir
-#
-# Experiment 9-v1_grimoire_6_96_sfc_false
-#
-./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 9-v1_grimoire_6_96_sfc_false false
-cp -prfv 9-v1_grimoire_6_96_sfc_false.dir /home/lschnorr//exp_9-v1_grimoire_6
-rm -rf 9-v1_grimoire_6_96_sfc_false.dir
-#
-# Final
-#
-cp script_9-v1_grimoire_6_nodes.sh /home/lschnorr//exp_9-v1_grimoire_6
-This script is at 'script_9-v1_grimoire_6_nodes.sh', commit it.
-#+end_example
-
-** 2017-04-06 8-node grimoire infiniband-only after Alya modifications :EXP10:
-#+begin_src shell :results output
-EID=10
-VERSION=1
-CLUSTER=grimoire
-NODES=8
-MACHINEFILE=machine-file
-G5KHOME=/home/lschnorr/
-EID=${EID}-v${VERSION}
-SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
-
-rm -f $SCRIPT
-
-EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
-echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
-echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
-echo " exit" >> $SCRIPT
-echo "fi" >> $SCRIPT
-
-# Create experimental directory in the home
-echo "mkdir -p $EDIR" >> $SCRIPT
-
-# The design
-NP=128
-for INFINIBAND in true; do
- for CASE in metis sfc; do
- NPN=$((NP))
- UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}_${INFINIBAND}
- echo "# "
- echo "# Experiment $UNIQUE"
- echo "# "
- echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
- echo "cp -prfv $UNIQUE.dir ${EDIR}"
- echo "rm -rf $UNIQUE.dir"
- done
-done >> ${SCRIPT}
-echo "# " >> ${SCRIPT}
-echo "# Final" >> ${SCRIPT}
-echo "# " >> ${SCRIPT}
-echo "cp $SCRIPT $EDIR" >> $SCRIPT
-chmod 755 ${SCRIPT}
-cat ${SCRIPT}
-echo "This script is at '${SCRIPT}', commit it."
-#+end_src
-
-#+RESULTS:
-#+begin_example
-if [ -e /home/lschnorr//exp_10-v1_grimoire_8 ]; then
- echo "Directory/File '/home/lschnorr//exp_10-v1_grimoire_8' already exists. Remove it or rename it."
- exit
-fi
-mkdir -p /home/lschnorr//exp_10-v1_grimoire_8
-#
-# Experiment 10-v1_grimoire_8_128_metis_true
-#
-./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 10-v1_grimoire_8_128_metis_true true
-cp -prfv 10-v1_grimoire_8_128_metis_true.dir /home/lschnorr//exp_10-v1_grimoire_8
-rm -rf 10-v1_grimoire_8_128_metis_true.dir
-#
-# Experiment 10-v1_grimoire_8_128_sfc_true
-#
-./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 10-v1_grimoire_8_128_sfc_true true
-cp -prfv 10-v1_grimoire_8_128_sfc_true.dir /home/lschnorr//exp_10-v1_grimoire_8
-rm -rf 10-v1_grimoire_8_128_sfc_true.dir
-#
-# Final
-#
-cp script_10-v1_grimoire_8_nodes.sh /home/lschnorr//exp_10-v1_grimoire_8
-This script is at 'script_10-v1_grimoire_8_nodes.sh', commit it.
-#+end_example
-
-** 2017-04-06 8-node grimoire infiniband-only without Alya modifications :EXP11:
-I will compile Alya before the partitioning weight modifications.
-#+begin_src shell :results output
-EID=11
-VERSION=1
-CLUSTER=grimoire
-NODES=8
-MACHINEFILE=machine-file
-G5KHOME=/home/lschnorr/
-EID=${EID}-v${VERSION}
-SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
-
-rm -f $SCRIPT
-
-EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
-echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
-echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
-echo " exit" >> $SCRIPT
-echo "fi" >> $SCRIPT
-
-# Create experimental directory in the home
-echo "mkdir -p $EDIR" >> $SCRIPT
-
-# The design
-NP=128
-for INFINIBAND in true; do
- for CASE in metis sfc; do
- NPN=$((NP))
- UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}_${INFINIBAND}
- echo "# "
- echo "# Experiment $UNIQUE"
- echo "# "
- echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
- echo "cp -prfv $UNIQUE.dir ${EDIR}"
- echo "rm -rf $UNIQUE.dir"
- done
-done >> ${SCRIPT}
-echo "# " >> ${SCRIPT}
-echo "# Final" >> ${SCRIPT}
-echo "# " >> ${SCRIPT}
-echo "cp $SCRIPT $EDIR" >> $SCRIPT
-chmod 755 ${SCRIPT}
-cat ${SCRIPT}
-echo "This script is at '${SCRIPT}', commit it."
-#+end_src
-
-#+RESULTS:
-#+begin_example
-if [ -e /home/lschnorr//exp_11-v1_grimoire_8 ]; then
- echo "Directory/File '/home/lschnorr//exp_11-v1_grimoire_8' already exists. Remove it or rename it."
- exit
-fi
-mkdir -p /home/lschnorr//exp_11-v1_grimoire_8
-#
-# Experiment 11-v1_grimoire_8_128_metis_true
-#
-./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 11-v1_grimoire_8_128_metis_true true
-cp -prfv 11-v1_grimoire_8_128_metis_true.dir /home/lschnorr//exp_11-v1_grimoire_8
-rm -rf 11-v1_grimoire_8_128_metis_true.dir
-#
-# Experiment 11-v1_grimoire_8_128_sfc_true
-#
-./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 11-v1_grimoire_8_128_sfc_true true
-cp -prfv 11-v1_grimoire_8_128_sfc_true.dir /home/lschnorr//exp_11-v1_grimoire_8
-rm -rf 11-v1_grimoire_8_128_sfc_true.dir
-#
-# Final
-#
-cp script_11-v1_grimoire_8_nodes.sh /home/lschnorr//exp_11-v1_grimoire_8
-This script is at 'script_11-v1_grimoire_8_nodes.sh', commit it.
-#+end_example
-
-** 2017-04-07 6-node grimoire (no Alya modifications, full computing instrumentations) :EXP12:
-
-For this experiment, we shall:
-- remove MPI instrumentation completely
-- adopt compiler instrumentation to obtain some application functions
-
-#+begin_src shell :results output
-EID=12
-VERSION=1
-CLUSTER=grimoire
-NODES=6
-MACHINEFILE=machine-file
-G5KHOME=/home/lschnorr/
-EID=${EID}-v${VERSION}
-SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
-
-rm -f $SCRIPT
-
-EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
-echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
-echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
-echo " exit" >> $SCRIPT
-echo "fi" >> $SCRIPT
-
-# Create experimental directory in the home
-echo "mkdir -p $EDIR" >> $SCRIPT
-
-# The design
-for NP in 96 48; do
- for INFINIBAND in true; do
- for CASE in metis sfc; do
- NPN=$((NP))
- UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}_${INFINIBAND}
- echo "# "
- echo "# Experiment $UNIQUE"
- echo "# "
- echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
- echo "cp -prfv $UNIQUE.dir ${EDIR}"
- echo "rm -rf $UNIQUE.dir"
- done
- done
-done >> ${SCRIPT}
-echo "# " >> ${SCRIPT}
-echo "# Final" >> ${SCRIPT}
-echo "# " >> ${SCRIPT}
-echo "cp $SCRIPT $EDIR" >> $SCRIPT
-chmod 755 ${SCRIPT}
-cat ${SCRIPT}
-echo "This script is at '${SCRIPT}', commit it."
-#+end_src
-
-#+RESULTS:
-#+begin_example
-if [ -e /home/lschnorr//exp_12-v1_grimoire_6 ]; then
- echo "Directory/File '/home/lschnorr//exp_12-v1_grimoire_6' already exists. Remove it or rename it."
- exit
-fi
-mkdir -p /home/lschnorr//exp_12-v1_grimoire_6
-#
-# Experiment 12-v1_grimoire_6_96_metis_true
-#
-./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 12-v1_grimoire_6_96_metis_true true
-cp -prfv 12-v1_grimoire_6_96_metis_true.dir /home/lschnorr//exp_12-v1_grimoire_6
-rm -rf 12-v1_grimoire_6_96_metis_true.dir
-#
-# Experiment 12-v1_grimoire_6_96_sfc_true
-#
-./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 12-v1_grimoire_6_96_sfc_true true
-cp -prfv 12-v1_grimoire_6_96_sfc_true.dir /home/lschnorr//exp_12-v1_grimoire_6
-rm -rf 12-v1_grimoire_6_96_sfc_true.dir
-#
-# Experiment 12-v1_grimoire_6_48_metis_true
-#
-./scripts/run_alya_experiment.sh machine-file 48 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 12-v1_grimoire_6_48_metis_true true
-cp -prfv 12-v1_grimoire_6_48_metis_true.dir /home/lschnorr//exp_12-v1_grimoire_6
-rm -rf 12-v1_grimoire_6_48_metis_true.dir
-#
-# Experiment 12-v1_grimoire_6_48_sfc_true
-#
-./scripts/run_alya_experiment.sh machine-file 48 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 12-v1_grimoire_6_48_sfc_true true
-cp -prfv 12-v1_grimoire_6_48_sfc_true.dir /home/lschnorr//exp_12-v1_grimoire_6
-rm -rf 12-v1_grimoire_6_48_sfc_true.dir
-#
-# Final
-#
-cp script_12-v1_grimoire_6_nodes.sh /home/lschnorr//exp_12-v1_grimoire_6
-This script is at 'script_12-v1_grimoire_6_nodes.sh', commit it.
-#+end_example
-
-** 2017-04-10 12-node grisou (computing instrumentations, Alya modifs yes/no) :EXP13:
-#+begin_src shell :results output
-EID=13
-VERSION=1
-CLUSTER=grisou
-NODES=12
-MACHINEFILE=machine-file
-G5KHOME=/home/lschnorr/
-EID=${EID}-v${VERSION}
-SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
-
-rm -f $SCRIPT
-
-EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
-echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
-echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
-echo " exit" >> $SCRIPT
-echo "fi" >> $SCRIPT
-
-# Create experimental directory in the home
-echo "mkdir -p $EDIR" >> $SCRIPT
-
-# The design
-ALYAO=Alya.x
-ALYAM=Alya.x.modif
-
-for NP in 192 96; do
- for INFINIBAND in true; do
- for CASE in metis sfc; do
- for ALYA in $ALYAO $ALYAM; do
- NPN=$((NP))
- UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}_${INFINIBAND}_${ALYA}
- echo "# "
- echo "# Experiment $UNIQUE"
- echo "# "
- echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} /home/lschnorr/alya-bsc/Executables/unix/$ALYA /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
- echo "cp -prfv $UNIQUE.dir ${EDIR}"
- echo "rm -rf $UNIQUE.dir"
- done
- done
- done >> ${SCRIPT}
-done
-echo "# " >> ${SCRIPT}
-echo "# Final" >> ${SCRIPT}
-echo "# " >> ${SCRIPT}
-echo "cp $SCRIPT $EDIR" >> $SCRIPT
-chmod 755 ${SCRIPT}
-cat ${SCRIPT}
-echo "This script is at '${SCRIPT}', commit it."
-#+end_src
-
-#+RESULTS:
-#+begin_example
-if [ -e /home/lschnorr//exp_13-v1_grisou_12 ]; then
- echo "Directory/File '/home/lschnorr//exp_13-v1_grisou_12' already exists. Remove it or rename it."
- exit
-fi
-mkdir -p /home/lschnorr//exp_13-v1_grisou_12
-#
-# Experiment 13-v1_grisou_12_192_metis_true_Alya.x
-#
-./scripts/run_alya_experiment.sh machine-file 192 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 13-v1_grisou_12_192_metis_true_Alya.x true
-cp -prfv 13-v1_grisou_12_192_metis_true_Alya.x.dir /home/lschnorr//exp_13-v1_grisou_12
-rm -rf 13-v1_grisou_12_192_metis_true_Alya.x.dir
-#
-# Experiment 13-v1_grisou_12_192_metis_true_Alya.x.modif
-#
-./scripts/run_alya_experiment.sh machine-file 192 /home/lschnorr/alya-bsc/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 13-v1_grisou_12_192_metis_true_Alya.x.modif true
-cp -prfv 13-v1_grisou_12_192_metis_true_Alya.x.modif.dir /home/lschnorr//exp_13-v1_grisou_12
-rm -rf 13-v1_grisou_12_192_metis_true_Alya.x.modif.dir
-#
-# Experiment 13-v1_grisou_12_192_sfc_true_Alya.x
-#
-./scripts/run_alya_experiment.sh machine-file 192 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 13-v1_grisou_12_192_sfc_true_Alya.x true
-cp -prfv 13-v1_grisou_12_192_sfc_true_Alya.x.dir /home/lschnorr//exp_13-v1_grisou_12
-rm -rf 13-v1_grisou_12_192_sfc_true_Alya.x.dir
-#
-# Experiment 13-v1_grisou_12_192_sfc_true_Alya.x.modif
-#
-./scripts/run_alya_experiment.sh machine-file 192 /home/lschnorr/alya-bsc/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 13-v1_grisou_12_192_sfc_true_Alya.x.modif true
-cp -prfv 13-v1_grisou_12_192_sfc_true_Alya.x.modif.dir /home/lschnorr//exp_13-v1_grisou_12
-rm -rf 13-v1_grisou_12_192_sfc_true_Alya.x.modif.dir
-#
-# Experiment 13-v1_grisou_12_96_metis_true_Alya.x
-#
-./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 13-v1_grisou_12_96_metis_true_Alya.x true
-cp -prfv 13-v1_grisou_12_96_metis_true_Alya.x.dir /home/lschnorr//exp_13-v1_grisou_12
-rm -rf 13-v1_grisou_12_96_metis_true_Alya.x.dir
-#
-# Experiment 13-v1_grisou_12_96_metis_true_Alya.x.modif
-#
-./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 13-v1_grisou_12_96_metis_true_Alya.x.modif true
-cp -prfv 13-v1_grisou_12_96_metis_true_Alya.x.modif.dir /home/lschnorr//exp_13-v1_grisou_12
-rm -rf 13-v1_grisou_12_96_metis_true_Alya.x.modif.dir
-#
-# Experiment 13-v1_grisou_12_96_sfc_true_Alya.x
-#
-./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 13-v1_grisou_12_96_sfc_true_Alya.x true
-cp -prfv 13-v1_grisou_12_96_sfc_true_Alya.x.dir /home/lschnorr//exp_13-v1_grisou_12
-rm -rf 13-v1_grisou_12_96_sfc_true_Alya.x.dir
-#
-# Experiment 13-v1_grisou_12_96_sfc_true_Alya.x.modif
-#
-./scripts/run_alya_experiment.sh machine-file 96 /home/lschnorr/alya-bsc/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 13-v1_grisou_12_96_sfc_true_Alya.x.modif true
-cp -prfv 13-v1_grisou_12_96_sfc_true_Alya.x.modif.dir /home/lschnorr//exp_13-v1_grisou_12
-rm -rf 13-v1_grisou_12_96_sfc_true_Alya.x.modif.dir
-#
-# Final
-#
-cp script_13-v1_grisou_12_nodes.sh /home/lschnorr//exp_13-v1_grisou_12
-This script is at 'script_13-v1_grisou_12_nodes.sh', commit it.
-#+end_example
-
-** 2017-05-10 20-node grisou :EXP14:
-*** Introduction :ATTACH:
-:PROPERTIES:
-:Attachments: all.scorep
-:ID: b51a80da-07e4-4d58-844c-a73009092eaa
-:END:
-
-The attached file, =all.scorep=, contains the functions that are
-instrumented during compilation by scorep. Those functions are non-MPI
-codes that represent the computation load.
-
-I am using Trunk.6515.Alya, unmodified, to get =Alya.x.orig=. Then, I
-manually modify Alya the way described here: [[*2017-04-06 Alya modifications to get more info, set balance to 1][2017-04-06 Alya
-modifications to get more info, set balance to 1]] to get another binary
-called =Alya.x.modif= (with elements weight set to 1). It seems that the
-latest version of Alya (revision 6515 at least) has a slightly
-different code to deal with element weight (with the =iweig=
-variable). So, for the =modif= version, I am forcing the =iweig= to be
-=1_rp=, removing the =if= that is now there.
-
-#+BEGIN_EXAMPLE
-lschnorr@grisou-1:~/Trunk.6515.Alya/Executables/unix$ svn diff ../../Sources/kernel/parall/mod_par_partit_sfc.f90
-Index: ../../Sources/kernel/parall/mod_par_partit_sfc.f90
-===================================================================
---- ../../Sources/kernel/parall/mod_par_partit_sfc.f90 (revision 6515)
-+++ ../../Sources/kernel/parall/mod_par_partit_sfc.f90 (working copy)
-@@ -459,6 +459,7 @@
- else
- iweig = 1_rp
- endif
-+ iweig = 1_rp
-
- if( PAR_MY_SFC_RANK_WM /= iboxc-1 )then !if true element must be sent
- if(bufwei(iboxl*2) == 0_rp) then
-#+END_EXAMPLE
-
-Note that the file =Sources/services/parall/par_outinf.f90= needs to be
-modified to output per-rank weights, points, etc. This is fundamental
-to conduct our analysis. Here's the diff:
-
-#+BEGIN_EXAMPLE
-lschnorr@grisou-1:~/Trunk.6515.Alya/Executables/unix$ svn diff ../../Sources/services/parall/par_outinf.f90
-Index: ../../Sources/services/parall/par_outinf.f90
-===================================================================
---- ../../Sources/services/parall/par_outinf.f90 (revision 6515)
-+++ ../../Sources/services/parall/par_outinf.f90 (working copy)
-@@ -15,6 +15,7 @@
- use mod_parall
- implicit none
- integer(ip) :: isubd,ksmin(2),ksmax(2),kaver,koutp,nb
-+ integer, parameter :: out_unit_npoin=20
-
- if( IMASTER ) then
-
-@@ -27,7 +28,11 @@
- ksmin = huge(1_ip)
- ksmax = -huge(1_ip)
- kaver = 0
-+ ! LUCAS: manual instrumentation starts here, all code marked by LUCAS
-+ open (unit=out_unit_npoin,file="results_NPOIN_NELEM_NELEW_NBOUN.log",action="write",status="replace")
- do isubd=1,npart_par
-+ ! LUCAS: Here I need to print: npoin_par(isubd)
-+ write (out_unit_npoin,*) "Rank ", isubd ," NPOIN ", npoin_par(isubd), " NELEM ", nelem_par(isubd), " NELEW ", nelew_par(isubd), " NBOUN ", nboun_par(isubd), " NNEIG ", lneig_par(isubd), " NBBOU ", npoin_par(isubd)
- if(npoin_par(isubd)> $SCRIPT
-echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
-echo " exit" >> $SCRIPT
-echo "fi" >> $SCRIPT
-
-# Create experimental directory in the home
-echo "mkdir -p $EDIR" >> $SCRIPT
-
-# The design
-ALYAO=Alya.x.orig
-ALYAM=Alya.x.modif
-ALYAPATH=/home/lschnorr/Trunk.6515.Alya
-
-for NP in 320; do
- for INFINIBAND in false; do
- for CASE in metis sfc; do
- for ALYA in $ALYAO $ALYAM; do
- NPN=$((NP))
- UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}_${INFINIBAND}_${ALYA}
- echo "# "
- echo "# Experiment $UNIQUE"
- echo "# "
- echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} ${ALYAPATH}/Executables/unix/$ALYA /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
- echo "cp -prfv $UNIQUE.dir ${EDIR}"
- echo "rm -rf $UNIQUE.dir"
- done
- done
- done >> ${SCRIPT}
-done
-echo "# " >> ${SCRIPT}
-echo "# Final" >> ${SCRIPT}
-echo "# " >> ${SCRIPT}
-echo "cp $SCRIPT $EDIR" >> $SCRIPT
-chmod 755 ${SCRIPT}
-cat ${SCRIPT}
-echo "This script is at '${SCRIPT}', commit it."
-#+end_src
-
-#+RESULTS:
-#+begin_example
-if [ -e /home/lschnorr//exp_14-v1_grisou_20 ]; then
- echo "Directory/File '/home/lschnorr//exp_14-v1_grisou_20' already exists. Remove it or rename it."
- exit
-fi
-mkdir -p /home/lschnorr//exp_14-v1_grisou_20
-#
-# Experiment 14-v1_grisou_20_320_metis_false_Alya.x.orig
-#
-./scripts/run_alya_experiment.sh machine-file 320 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.orig /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 14-v1_grisou_20_320_metis_false_Alya.x.orig false
-cp -prfv 14-v1_grisou_20_320_metis_false_Alya.x.orig.dir /home/lschnorr//exp_14-v1_grisou_20
-rm -rf 14-v1_grisou_20_320_metis_false_Alya.x.orig.dir
-#
-# Experiment 14-v1_grisou_20_320_metis_false_Alya.x.modif
-#
-./scripts/run_alya_experiment.sh machine-file 320 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 14-v1_grisou_20_320_metis_false_Alya.x.modif false
-cp -prfv 14-v1_grisou_20_320_metis_false_Alya.x.modif.dir /home/lschnorr//exp_14-v1_grisou_20
-rm -rf 14-v1_grisou_20_320_metis_false_Alya.x.modif.dir
-#
-# Experiment 14-v1_grisou_20_320_sfc_false_Alya.x.orig
-#
-./scripts/run_alya_experiment.sh machine-file 320 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.orig /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 14-v1_grisou_20_320_sfc_false_Alya.x.orig false
-cp -prfv 14-v1_grisou_20_320_sfc_false_Alya.x.orig.dir /home/lschnorr//exp_14-v1_grisou_20
-rm -rf 14-v1_grisou_20_320_sfc_false_Alya.x.orig.dir
-#
-# Experiment 14-v1_grisou_20_320_sfc_false_Alya.x.modif
-#
-./scripts/run_alya_experiment.sh machine-file 320 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 14-v1_grisou_20_320_sfc_false_Alya.x.modif false
-cp -prfv 14-v1_grisou_20_320_sfc_false_Alya.x.modif.dir /home/lschnorr//exp_14-v1_grisou_20
-rm -rf 14-v1_grisou_20_320_sfc_false_Alya.x.modif.dir
-#
-# Final
-#
-cp script_14-v1_grisou_20_nodes.sh /home/lschnorr//exp_14-v1_grisou_20
-This script is at 'script_14-v1_grisou_20_nodes.sh', commit it.
-#+end_example
-
-** 2017-05-11 44-node grisou (same as previous entry) :EXP15:
-*** Design
-
-#+begin_src shell :results output
-EID=15
-VERSION=1
-CLUSTER=grisou
-NODES=44
-MACHINEFILE=machine-file
-G5KHOME=/home/lschnorr/
-EID=${EID}-v${VERSION}
-SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
-
-rm -f $SCRIPT
-
-EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
-echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
-echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
-echo " exit" >> $SCRIPT
-echo "fi" >> $SCRIPT
-
-# Create experimental directory in the home
-echo "mkdir -p $EDIR" >> $SCRIPT
-
-# The design
-ALYAO=Alya.x.orig
-ALYAM=Alya.x.modif
-ALYAPATH=/home/lschnorr/Trunk.6515.Alya
-
-for NP in 704; do
- for INFINIBAND in false; do
- for CASE in metis sfc; do
- for ALYA in $ALYAO $ALYAM; do
- NPN=$((NP))
- UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}_${INFINIBAND}_${ALYA}
- echo "# "
- echo "# Experiment $UNIQUE"
- echo "# "
- echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} ${ALYAPATH}/Executables/unix/$ALYA /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
- echo "cp -prfv $UNIQUE.dir ${EDIR}"
- echo "rm -rf $UNIQUE.dir"
- done
- done
- done >> ${SCRIPT}
-done
-echo "# " >> ${SCRIPT}
-echo "# Final" >> ${SCRIPT}
-echo "# " >> ${SCRIPT}
-echo "cp $SCRIPT $EDIR" >> $SCRIPT
-chmod 755 ${SCRIPT}
-cat ${SCRIPT}
-echo "This script is at '${SCRIPT}', commit it."
-#+end_src
-
-#+RESULTS:
-#+begin_example
-if [ -e /home/lschnorr//exp_15-v1_grisou_44 ]; then
- echo "Directory/File '/home/lschnorr//exp_15-v1_grisou_44' already exists. Remove it or rename it."
- exit
-fi
-mkdir -p /home/lschnorr//exp_15-v1_grisou_44
-#
-# Experiment 15-v1_grisou_44_704_metis_false_Alya.x.orig
-#
-./scripts/run_alya_experiment.sh machine-file 704 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.orig /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 15-v1_grisou_44_704_metis_false_Alya.x.orig false
-cp -prfv 15-v1_grisou_44_704_metis_false_Alya.x.orig.dir /home/lschnorr//exp_15-v1_grisou_44
-rm -rf 15-v1_grisou_44_704_metis_false_Alya.x.orig.dir
-#
-# Experiment 15-v1_grisou_44_704_metis_false_Alya.x.modif
-#
-./scripts/run_alya_experiment.sh machine-file 704 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 15-v1_grisou_44_704_metis_false_Alya.x.modif false
-cp -prfv 15-v1_grisou_44_704_metis_false_Alya.x.modif.dir /home/lschnorr//exp_15-v1_grisou_44
-rm -rf 15-v1_grisou_44_704_metis_false_Alya.x.modif.dir
-#
-# Experiment 15-v1_grisou_44_704_sfc_false_Alya.x.orig
-#
-./scripts/run_alya_experiment.sh machine-file 704 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.orig /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 15-v1_grisou_44_704_sfc_false_Alya.x.orig false
-cp -prfv 15-v1_grisou_44_704_sfc_false_Alya.x.orig.dir /home/lschnorr//exp_15-v1_grisou_44
-rm -rf 15-v1_grisou_44_704_sfc_false_Alya.x.orig.dir
-#
-# Experiment 15-v1_grisou_44_704_sfc_false_Alya.x.modif
-#
-./scripts/run_alya_experiment.sh machine-file 704 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 15-v1_grisou_44_704_sfc_false_Alya.x.modif false
-cp -prfv 15-v1_grisou_44_704_sfc_false_Alya.x.modif.dir /home/lschnorr//exp_15-v1_grisou_44
-rm -rf 15-v1_grisou_44_704_sfc_false_Alya.x.modif.dir
-#
-# Final
-#
-cp script_15-v1_grisou_44_nodes.sh /home/lschnorr//exp_15-v1_grisou_44
-This script is at 'script_15-v1_grisou_44_nodes.sh', commit it.
-#+end_example
-
-** 2017-05-11 8-node grimoire (as previous, Infiniband and Ethernet) :EXP16:
-*** Design
-
-#+begin_src shell :results output
-EID=16
-VERSION=1
-CLUSTER=grimoire
-NODES=8
-MACHINEFILE=machine-file
-G5KHOME=/home/lschnorr/
-EID=${EID}-v${VERSION}
-SCRIPT="script_${EID}_${CLUSTER}_${NODES}_nodes.sh"
-
-rm -f $SCRIPT
-
-EDIR="$G5KHOME/exp_${EID}_${CLUSTER}_${NODES}"
-echo "if [ -e "$EDIR" ]; then" >> $SCRIPT
-echo " echo \"Directory/File '${EDIR}' already exists. Remove it or rename it.\"" >> $SCRIPT
-echo " exit" >> $SCRIPT
-echo "fi" >> $SCRIPT
-
-# Create experimental directory in the home
-echo "mkdir -p $EDIR" >> $SCRIPT
-
-# The design
-ALYAO=Alya.x.orig
-ALYAM=Alya.x.modif
-ALYAPATH=/home/lschnorr/Trunk.6515.Alya
-
-for NP in 128; do
- for INFINIBAND in true false; do
- for CASE in metis sfc; do
- for ALYA in $ALYAO $ALYAM; do
- NPN=$((NP))
- UNIQUE=${EID}_${CLUSTER}_${NODES}_${NPN}_${CASE}_${INFINIBAND}_${ALYA}
- echo "# "
- echo "# Experiment $UNIQUE"
- echo "# "
- echo "./scripts/run_alya_experiment.sh ${MACHINEFILE} ${NPN} ${ALYAPATH}/Executables/unix/$ALYA /home/lschnorr/WORK-RICARD/resp_${CASE}/fensap.dat $UNIQUE ${INFINIBAND}"
- echo "cp -prfv $UNIQUE.dir ${EDIR}"
- echo "rm -rf $UNIQUE.dir"
- done
- done
- done >> ${SCRIPT}
-done
-echo "# " >> ${SCRIPT}
-echo "# Final" >> ${SCRIPT}
-echo "# " >> ${SCRIPT}
-echo "cp $SCRIPT $EDIR" >> $SCRIPT
-chmod 755 ${SCRIPT}
-cat ${SCRIPT}
-echo "This script is at '${SCRIPT}', commit it."
-#+end_src
-
-#+RESULTS:
-#+begin_example
-if [ -e /home/lschnorr//exp_16-v1_grimoire_8 ]; then
- echo "Directory/File '/home/lschnorr//exp_16-v1_grimoire_8' already exists. Remove it or rename it."
- exit
-fi
-mkdir -p /home/lschnorr//exp_16-v1_grimoire_8
-#
-# Experiment 16-v1_grimoire_8_128_metis_true_Alya.x.orig
-#
-./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.orig /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 16-v1_grimoire_8_128_metis_true_Alya.x.orig true
-cp -prfv 16-v1_grimoire_8_128_metis_true_Alya.x.orig.dir /home/lschnorr//exp_16-v1_grimoire_8
-rm -rf 16-v1_grimoire_8_128_metis_true_Alya.x.orig.dir
-#
-# Experiment 16-v1_grimoire_8_128_metis_true_Alya.x.modif
-#
-./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 16-v1_grimoire_8_128_metis_true_Alya.x.modif true
-cp -prfv 16-v1_grimoire_8_128_metis_true_Alya.x.modif.dir /home/lschnorr//exp_16-v1_grimoire_8
-rm -rf 16-v1_grimoire_8_128_metis_true_Alya.x.modif.dir
-#
-# Experiment 16-v1_grimoire_8_128_sfc_true_Alya.x.orig
-#
-./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.orig /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 16-v1_grimoire_8_128_sfc_true_Alya.x.orig true
-cp -prfv 16-v1_grimoire_8_128_sfc_true_Alya.x.orig.dir /home/lschnorr//exp_16-v1_grimoire_8
-rm -rf 16-v1_grimoire_8_128_sfc_true_Alya.x.orig.dir
-#
-# Experiment 16-v1_grimoire_8_128_sfc_true_Alya.x.modif
-#
-./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 16-v1_grimoire_8_128_sfc_true_Alya.x.modif true
-cp -prfv 16-v1_grimoire_8_128_sfc_true_Alya.x.modif.dir /home/lschnorr//exp_16-v1_grimoire_8
-rm -rf 16-v1_grimoire_8_128_sfc_true_Alya.x.modif.dir
-#
-# Experiment 16-v1_grimoire_8_128_metis_false_Alya.x.orig
-#
-./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.orig /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 16-v1_grimoire_8_128_metis_false_Alya.x.orig false
-cp -prfv 16-v1_grimoire_8_128_metis_false_Alya.x.orig.dir /home/lschnorr//exp_16-v1_grimoire_8
-rm -rf 16-v1_grimoire_8_128_metis_false_Alya.x.orig.dir
-#
-# Experiment 16-v1_grimoire_8_128_metis_false_Alya.x.modif
-#
-./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_metis/fensap.dat 16-v1_grimoire_8_128_metis_false_Alya.x.modif false
-cp -prfv 16-v1_grimoire_8_128_metis_false_Alya.x.modif.dir /home/lschnorr//exp_16-v1_grimoire_8
-rm -rf 16-v1_grimoire_8_128_metis_false_Alya.x.modif.dir
-#
-# Experiment 16-v1_grimoire_8_128_sfc_false_Alya.x.orig
-#
-./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.orig /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 16-v1_grimoire_8_128_sfc_false_Alya.x.orig false
-cp -prfv 16-v1_grimoire_8_128_sfc_false_Alya.x.orig.dir /home/lschnorr//exp_16-v1_grimoire_8
-rm -rf 16-v1_grimoire_8_128_sfc_false_Alya.x.orig.dir
-#
-# Experiment 16-v1_grimoire_8_128_sfc_false_Alya.x.modif
-#
-./scripts/run_alya_experiment.sh machine-file 128 /home/lschnorr/Trunk.6515.Alya/Executables/unix/Alya.x.modif /home/lschnorr/WORK-RICARD/resp_sfc/fensap.dat 16-v1_grimoire_8_128_sfc_false_Alya.x.modif false
-cp -prfv 16-v1_grimoire_8_128_sfc_false_Alya.x.modif.dir /home/lschnorr//exp_16-v1_grimoire_8
-rm -rf 16-v1_grimoire_8_128_sfc_false_Alya.x.modif.dir
-#
-# Final
-#
-cp script_16-v1_grimoire_8_nodes.sh /home/lschnorr//exp_16-v1_grimoire_8
-This script is at 'script_16-v1_grimoire_8_nodes.sh', commit it.
-#+end_example
-
-* Analysis
-** 2-node grisou (17 and 32 ranks) :EXP1:
-*** Rough epoch-based execution time
-#+name: exp_1_grisou_2_epoch
-#+begin_src shell :results output org table
-EDIR=exp_1_grisou_2
-for case in $(find $EDIR | grep -e "sfc.org" -e "metis.org"); do
- echo -n "$(basename $case) "
- cat $case | grep epoch | sed "s/^.*: //" | tr '\n' '-' | sed -e "s/-$//" -e "s/-/ - /" -e "s/$/\n/" | bc -l;
-done | sed -e "s/.org//" | sed -e "s/_/|/g" -e "s/-//g"
-#+end_src
-
-#+RESULTS: exp_1_grisou_2_epoch
-#+BEGIN_SRC org
-| grisou | 2 | 32 | metis | 951 |
-| grisou | 2 | 32 | sfc | 762 |
-| grisou | 2 | 17 | metis | 1417 |
-| grisou | 2 | 17 | sfc | 1225 |
-#+END_SRC
-
-#+header: :var df=exp_1_grisou_2_epoch
-#+begin_src R :results output graphics :file img/exp_1_grisou_2_epoch.png :exports both :width 600 :height 400 :session
-library(dplyr);
-library(ggplot2);
-df %>%
- rename(Cluster= V1,
- Nodes = V2,
- NP = V3,
- Case = V4,
- Time = V5) %>%
- filter(NP != 121) %>%
- ggplot(aes(x=NP, y=Time, color=Case)) +
- theme_bw() +
- ylim(0,NA) +
- ylab("Time [s]") +
- geom_point() +
- geom_line() +
- theme(legend.position = "top");
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_1_grisou_2_epoch.png]]
-** 10-node grisou (97, 113, 129, 145, 160) :EXP2:
-*** Rough epoch-based execution time
-#+name: exp_2_grisou_10_epoch
-#+begin_src shell :results output org table
-EDIR=exp_2_grisou_10
-for case in $(find $EDIR | grep -e "sfc.org" -e "metis.org"); do
- echo -n "$(basename $case) "
- cat $case | grep epoch | sed "s/^.*: //" | tr '\n' '-' | sed -e "s/-$//" -e "s/-/ - /" -e "s/$/\n/" | bc -l;
-done | sed -e "s/.org//" | sed -e "s/_/|/g" -e "s/-//g"
-#+end_src
-
-#+RESULTS: exp_2_grisou_10_epoch
-#+BEGIN_SRC org
-| grisou | 10 | 160 | metis | 304 |
-| grisou | 10 | 129 | sfc | 337 |
-| grisou | 10 | 97 | metis | 401 |
-| grisou | 10 | 145 | sfc | 315 |
-| grisou | 10 | 129 | metis | 345 |
-| grisou | 10 | 145 | metis | 321 |
-| grisou | 10 | 97 | sfc | 380 |
-| grisou | 10 | 160 | sfc | 301 |
-| grisou | 10 | 113 | metis | 363 |
-| grisou | 10 | 113 | sfc | 359 |
-#+END_SRC
-
-#+header: :var df=exp_2_grisou_10_epoch
-#+begin_src R :results output graphics :file img/exp_2_grisou_10_epoch.png :exports both :width 200 :height 400 :session
-library(dplyr);
-library(ggplot2);
-df %>%
- rename(Cluster= V1,
- Nodes = V2,
- NP = V3,
- Case = V4,
- Time = V5) %>%
- filter(NP != 121) %>%
- ggplot(aes(x=NP, y=Time, color=Case)) +
- theme_bw() +
- ylim(0,NA) +
- ylab("Time [s]") +
- geom_point() +
- geom_line() +
- theme(legend.position = "top");
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_2_grisou_10_epoch.png]]
-*** Convert to CSV
-I need on the $PATH:
-- akypuera's otf22paje (compiled using OTF2 libraries of ScoreP 3.0)
-- pajeng's =pj_dump=
-
-#+begin_src shell :results output
-export PATH=$PATH:/home/schnorr/install/stow/bin/
-export PATH=$PATH:/home/schnorr/dev/pajeng/b/
-
-convert() {
- pushd $(dirname $otf2)
- otf22paje traces.otf2 | pj_dump | grep ^State | cut -d, -f2,4,5,8 | sed -e "s/ //g" -e "s/MPIRank//" > traces.csv
- popd
-}
-
-EDIR=exp_2_grisou_10
-N=4 # number of parallel batches
-for otf2 in $(find $EDIR | grep otf2$); do
- ((i=i%N)); ((i++==0)) && wait
- convert $otf2 &
-done
-#+end_src
-
-#+RESULTS:
-#+begin_example
-/home/schnorr/install/stow/bin//otf22paje
-Usage: otf22paje [OPTION...] ANCHORFILE
-Converts an OTF2 archive to the Paje file format
-
- -b, --basic Avoid extended events (impoverished trace file)
- -d, --dummy Read input traces but won't translate (no output)
- -f, --flat Flat hierarchy (only MPI ranks)
- -h, --hostfile=FILE MPI hostfile to create system hierarchy
- -i, --ignore-errors Ignore errors
- -l, --no-links Don't convert links
- -m, --comment=COMMENT Comment is echoed to output
- -n, --commentfile=FILE Comments (from file) echoed to output
- -o, --only-mpi Only convert MPI states
- -s, --no-states Don't convert states
- -z, --normalize-mpi Try to normalize MPI state names
- -?, --help Give this help list
- --usage Give a short usage message
-
-Mandatory or optional arguments to long options are also mandatory or optional
-for any corresponding short options.
-~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_160_metis.dir/scorep_grisou_10_160_metis ~/dev/Alya-Perf
-~/dev/Alya-Perf
-~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_129_sfc.dir/scorep_grisou_10_129_sfc ~/dev/Alya-Perf
-~/dev/Alya-Perf
-~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_97_metis.dir/scorep_grisou_10_97_metis ~/dev/Alya-Perf
-~/dev/Alya-Perf
-~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_145_sfc.dir/scorep_grisou_10_145_sfc ~/dev/Alya-Perf
-~/dev/Alya-Perf
-~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_129_metis.dir/scorep_grisou_10_129_metis ~/dev/Alya-Perf
-~/dev/Alya-Perf
-~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_145_metis.dir/scorep_grisou_10_145_metis ~/dev/Alya-Perf
-~/dev/Alya-Perf
-~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_97_sfc.dir/scorep_grisou_10_97_sfc ~/dev/Alya-Perf
-~/dev/Alya-Perf
-~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_160_sfc.dir/scorep_grisou_10_160_sfc ~/dev/Alya-Perf
-~/dev/Alya-Perf
-~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_113_metis.dir/scorep_grisou_10_113_metis ~/dev/Alya-Perf
-~/dev/Alya-Perf
-~/dev/Alya-Perf/exp_2_grisou_10/grisou_10_113_sfc.dir/scorep_grisou_10_113_sfc ~/dev/Alya-Perf
-~/dev/Alya-Perf
-#+end_example
-
-#+begin_src shell :results output
-export PATH=$PATH:/home/schnorr/install/stow/bin/
-export PATH=$PATH:/home/schnorr/dev/pajeng/b/
-
-convert() {
- pushd $(dirname $otf2)
- otf22paje traces.otf2 | pj_dump | grep ^State | cut -d, -f2,4,5,8 | sed -e "s/ //g" -e "s/MPIRank//" > traces.csv
- popd
-}
-
-EDIR=exp_2_grisou_10
-# Files already converted (whose CSV size is not zero)
-EXISTINGFILE=$(tempfile)
-OTF2FILE=$(tempfile)
-find $EDIR -not -empty | grep csv$ | sed -e "s/.csv$//" | sort > $EXISTINGFILE
-find $EDIR | grep otf2 | sed -e "s/.otf2$//" | sort > $OTF2FILE
-
-for otf2 in $(comm -3 $OTF2FILE $EXISTINGFILE | sed "s/$/.otf2/"); do
- convert $otf2
-done
-#+end_src
-
-#+RESULTS:
-*** Read and analysis
-Check read and analysis of next section.
-** 10-node grisou (97, 113, 129, 145, 160) :EXP3:
-*** Rough epoch-based execution time
-#+name: exp_3_grisou_10_epoch
-#+begin_src shell :results output org table
-EDIR=exp_3_grisou_10
-for case in $(find $EDIR | grep -e "sfc.org" -e "metis.org"); do
- echo -n "$(basename $case) "
- cat $case | grep epoch | sed "s/^.*: //" | tr '\n' '-' | sed -e "s/-$//" -e "s/-/ - /" -e "s/$/\n/" | bc -l;
-done | sed -e "s/.org//" | sed -e "s/_/|/g" -e "s/-//g"
-#+end_src
-
-#+RESULTS: exp_3_grisou_10_epoch
-#+BEGIN_SRC org
-| 3 | grisou | 10 | 145 | sfc | 320 |
-| 3 | grisou | 10 | 113 | sfc | 356 |
-| 3 | grisou | 10 | 97 | metis | 394 |
-| 3 | grisou | 10 | 129 | sfc | 336 |
-| 3 | grisou | 10 | 129 | metis | 344 |
-| 3 | grisou | 10 | 160 | sfc | 308 |
-| 3 | grisou | 10 | 145 | metis | 319 |
-| 3 | grisou | 10 | 113 | metis | 362 |
-| 3 | grisou | 10 | 160 | metis | 306 |
-| 3 | grisou | 10 | 97 | sfc | 380 |
-#+END_SRC
-
-#+header: :var df=exp_3_grisou_10_epoch
-#+begin_src R :results output graphics :file img/exp_3_grisou_10_epoch.png :exports both :width 200 :height 400 :session
-library(dplyr);
-library(ggplot2);
-df %>%
- rename(EID = V1,
- Cluster= V2,
- Nodes = V3,
- NP = V4,
- Case = V5,
- Time = V6) %>%
- filter(NP != 121) %>%
- ggplot(aes(x=NP, y=Time, color=Case)) +
- theme_bw() +
- ylim(0,NA) +
- ylab("Time [s]") +
- geom_point() +
- geom_line() +
- theme(legend.position = "top");
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_3_grisou_10_epoch.png]]
-*** Convert to CSV
-I need on the $PATH:
-- akypuera's otf22paje (compiled using OTF2 libraries of ScoreP 3.0)
-- pajeng's =pj_dump=
-
-#+begin_src shell :results output
-export PATH=$PATH:/home/schnorr/install/stow/bin/
-export PATH=$PATH:/home/schnorr/dev/pajeng/b/
-
-convert() {
- pushd $(dirname $otf2)
- otf22paje traces.otf2 | pj_dump | grep ^State | cut -d, -f2,4,5,8 | sed -e "s/ //g" -e "s/MPIRank//" > traces.csv
- popd
-}
-
-EDIR=exp_3_grisou_10
-# Files already converted (whose CSV size is not zero)
-EXISTINGFILE=$(tempfile)
-OTF2FILE=$(tempfile)
-find $EDIR -not -empty | grep csv$ | sed -e "s/.csv$//" | sort > $EXISTINGFILE
-find $EDIR | grep otf2 | sed -e "s/.otf2$//" | sort > $OTF2FILE
-
-for otf2 in $(comm -3 $OTF2FILE $EXISTINGFILE | sed "s/$/.otf2/"); do
- echo $otf2
- convert $otf2
-done
-#+end_src
-
-#+RESULTS:
-#+begin_example
-exp_3_grisou_10/3_grisou_10_113_metis.dir/scorep_3_grisou_10_113_metis/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_113_metis.dir/scorep_3_grisou_10_113_metis ~/dev/Alya-Perf
-~/dev/Alya-Perf
-exp_3_grisou_10/3_grisou_10_113_sfc.dir/scorep_3_grisou_10_113_sfc/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_113_sfc.dir/scorep_3_grisou_10_113_sfc ~/dev/Alya-Perf
-~/dev/Alya-Perf
-exp_3_grisou_10/3_grisou_10_129_metis.dir/scorep_3_grisou_10_129_metis/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_129_metis.dir/scorep_3_grisou_10_129_metis ~/dev/Alya-Perf
-~/dev/Alya-Perf
-exp_3_grisou_10/3_grisou_10_129_sfc.dir/scorep_3_grisou_10_129_sfc/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_129_sfc.dir/scorep_3_grisou_10_129_sfc ~/dev/Alya-Perf
-~/dev/Alya-Perf
-exp_3_grisou_10/3_grisou_10_145_metis.dir/scorep_3_grisou_10_145_metis/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_145_metis.dir/scorep_3_grisou_10_145_metis ~/dev/Alya-Perf
-~/dev/Alya-Perf
-exp_3_grisou_10/3_grisou_10_145_sfc.dir/scorep_3_grisou_10_145_sfc/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_145_sfc.dir/scorep_3_grisou_10_145_sfc ~/dev/Alya-Perf
-~/dev/Alya-Perf
-exp_3_grisou_10/3_grisou_10_160_metis.dir/scorep_3_grisou_10_160_metis/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_160_metis.dir/scorep_3_grisou_10_160_metis ~/dev/Alya-Perf
-~/dev/Alya-Perf
-exp_3_grisou_10/3_grisou_10_160_sfc.dir/scorep_3_grisou_10_160_sfc/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_160_sfc.dir/scorep_3_grisou_10_160_sfc ~/dev/Alya-Perf
-~/dev/Alya-Perf
-exp_3_grisou_10/3_grisou_10_97_metis.dir/scorep_3_grisou_10_97_metis/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_97_metis.dir/scorep_3_grisou_10_97_metis ~/dev/Alya-Perf
-~/dev/Alya-Perf
-exp_3_grisou_10/3_grisou_10_97_sfc.dir/scorep_3_grisou_10_97_sfc/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_97_sfc.dir/scorep_3_grisou_10_97_sfc ~/dev/Alya-Perf
-~/dev/Alya-Perf
-#+end_example
-*** Read in R
-#+begin_src R :results output :session :exports both :tangle do.R :tangle-mode (identity #o755)
-#!/usr/bin/Rscript
-library(readr);
-library(dplyr);
-alya_scorep_trace_read <- function(filename)
-{
- meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[3], "_"));
- read_csv(filename,
- col_names=c("Rank", "Start", "End", "Value"),
- progress=TRUE) %>%
- # Transform Value to factor
- mutate(Value = as.factor(Value)) %>%
- # Detect begin and end of iterations
- mutate(Iteration = case_when(
- grepl("timste", .$Value) ~ 1,
- grepl("endste", .$Value) ~ -1,
- TRUE ~ 0)) %>%
- group_by(Rank) %>%
- # Create a logical to detect observations within iterations
- mutate(Iteration = as.logical(cumsum(Iteration))) %>%
- # Get only observations that belongs to some iteration
- filter(Iteration == TRUE) %>%
- ungroup() %>%
- # Create the iteration by cumsum-
- mutate(Iteration = case_when(
- grepl("timste", .$Value) ~ 1,
- TRUE ~ 0)) %>%
- group_by(Rank) %>%
- mutate(Iteration = cumsum(Iteration)) %>%
- mutate(EID = meta[2],
- Platform = meta[3],
- Nodes = meta[4],
- NP = meta[5],
- Partitioning = meta[6]);
-}
-
-alya_scorep_trace_iterations <- function(filename)
-{
- alya_scorep_trace_read(filename) %>%
- group_by(Rank, Iteration, Platform, Nodes, NP, Partitioning, EID) %>%
- summarize(N=n(), S=min(Start), E=max(End));
-}
-
-args = commandArgs(trailingOnly=TRUE)
-print(args);
-df <- do.call("rbind", lapply(args, function(x) { alya_scorep_trace_iterations(x) }));
-write.csv(df, "exp2_exp3_iterations.csv");
-#+end_src
-
-#+RESULTS:
-*** Analysis :ATTACH:
-:PROPERTIES:
-:Attachments: exp2_exp3_iterations.csv
-:ID: d2233039-ad84-42ac-ad70-545877ba7e8e
-:END:
-**** Iteration-only shows SFC is worse than Metis (average per-iteration span)
-#+begin_src R :results output :session :exports both
-library(readr);
-library(dplyr);
-df <- read_csv("data/d2/233039-ad84-42ac-ad70-545877ba7e8e/exp2_exp3_iterations.csv") %>%
- select(-X1) %>%
- rename(Start = S, End = E) %>%
- mutate(Duration = End - Start) %>%
- mutate(Platform = as.factor(Platform),
- Partitioning = as.factor(Partitioning),
- Nodes = as.factor(Nodes));
-df %>% summary;
-#+end_src
-
-#+RESULTS:
-#+begin_example
-
-Attaching package: ‘dplyr’
-
-The following objects are masked from ‘package:stats’:
-
- filter, lag
-
-The following objects are masked from ‘package:base’:
-
- intersect, setdiff, setequal, union
-Parsed with column specification:
-cols(
- X1 = col_integer(),
- Rank = col_integer(),
- Iteration = col_integer(),
- Platform = col_character(),
- Nodes = col_integer(),
- NP = col_integer(),
- Partitioning = col_character(),
- EID = col_integer(),
- N = col_integer(),
- S = col_double(),
- E = col_double()
-)
-Warning message:
-Missing column names filled in: 'X1' [1]
- Rank Iteration Platform Nodes NP
- Min. : 0.00 Min. : 1.0 grisou:25760 10:25760 Min. : 97.0
- 1st Qu.: 32.00 1st Qu.: 3.0 1st Qu.:113.0
- Median : 64.00 Median : 5.5 Median :129.0
- Mean : 65.84 Mean : 5.5 Mean :132.7
- 3rd Qu.: 96.00 3rd Qu.: 8.0 3rd Qu.:145.0
- Max. :159.00 Max. :10.0 Max. :160.0
- Partitioning EID N Start End
- metis:12880 Min. :2.0 Min. : 3580 Min. :120.2 Min. :139.9
- sfc :12880 1st Qu.:2.0 1st Qu.:17618 1st Qu.:168.8 1st Qu.:188.9
- Median :2.5 Median :24816 Median :220.5 Median :241.4
- Mean :2.5 Mean :28263 Mean :222.6 Mean :243.2
- 3rd Qu.:3.0 3rd Qu.:37184 3rd Qu.:271.5 3rd Qu.:293.7
- Max. :3.0 Max. :85082 Max. :375.5 Max. :397.2
- Duration
- Min. :13.57
- 1st Qu.:18.05
- Median :19.93
- Mean :20.63
- 3rd Qu.:23.17
- Max. :29.10
-#+end_example
-
-Execution time
-
-#+begin_src R :results output graphics :file img/exp2_exp3_iterations_only.png :exports both :width 600 :height 400 :session
-df %>%
- group_by(Platform, Nodes, NP, Partitioning, EID) %>%
- summarize(Time = max(End) - min(Start)) %>%
- group_by(Platform, Nodes, NP, Partitioning) %>%
- summarize(Time = mean(Time), N=n()) %>%
- print %>%
- ggplot(aes(x=NP, y=Time, color=Partitioning)) +
- theme_bw(base_size=12) +
- geom_point() +
- geom_line() +
- ylim(0, NA)
-#+end_src
-
-#+RESULTS:
-[[file:img/exp2_exp3_iterations_only.png]]
-
-#+begin_src R :results output :session :exports both
-df %>%
- group_by(NP, Iteration, Partitioning) %>%
- # calculate the mean of number of operations per iteration
- summarize(N=mean(N), Z=n());
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Source: local data frame [100 x 5]
-Groups: NP, Iteration [?]
-
- NP Iteration Partitioning N Z
-
-1 97 1 metis 19204.61 194
-2 97 1 sfc 40975.99 194
-3 97 2 metis 18361.41 194
-4 97 2 sfc 39909.78 194
-5 97 3 metis 18358.87 194
-6 97 3 sfc 39712.10 194
-7 97 4 metis 17679.86 194
-8 97 4 sfc 38238.62 194
-9 97 5 metis 17201.62 194
-10 97 5 sfc 37063.47 194
-# ... with 90 more rows
-#+end_example
-
-#+begin_src R :results output graphics :file img/exp2_exp3_number_of_operations.png :exports both :width 600 :height 400 :session
-library(ggplot2);
-df %>%
- group_by(NP, Iteration, Partitioning) %>%
- # calculate the mean of number of operations per iteration
- summarize(N=mean(N), Z=n()) %>%
- ungroup() %>%
- ggplot(aes(x=Iteration, y=N, color=Partitioning)) +
- theme_bw(base_size=12) +
- ylim(0,NA) +
- geom_point() +
- theme(legend.position="top") +
- facet_wrap(~NP, nrow=1)
-#+end_src
-
-#+RESULTS:
-[[file:img/exp2_exp3_number_of_operations.png]]
-**** Addressing Rick's questions (on max per-iteration)
-:RICK:
-Could you give us the results shown in =exp2_exp3_iterations_only.png=, but
-considering the maximum time for each CPU-core instead of the average?
-:END:
-
-:LUCAS:
-If I understood correctly your question, you want to see the time each
-CPU-core took to compute the 10 iterations. So, I put Rank in the X
-axis, and on Y, I put the difference of time between the moment where
-a given rank enters the first iteration until the moment it leaves the
-10th iteration. I select only one experiment (exp2) because they are
-very similar (on the same set of machines). The exp2 experiment
-contained 5 executions with different NP (97, 113, 129, 145, and
-160). The five plots are in the attached file
-=exp2_exp3_10iter_time_of_each_rank.png=
-:END:
-
-#+begin_src R :results output graphics :file img/exp2_10iter_time_of_each_rank.png :exports both :width 600 :height 600 :session
-library(ggplot2);
-df %>%
- filter(EID==2) %>%
- group_by(Rank, NP, Partitioning, EID) %>%
- summarize(Time = max(End) - min(Start)) %>%
- ggplot(aes(x=Rank, y=Time, color=Partitioning)) +
- theme_bw(base_size=12) +
- geom_point() +
- ylim(0, NA) +
- facet_grid(EID~NP) +
- theme(legend.position="top")
-#+end_src
-
-#+RESULTS:
-[[file:img/exp2_10iter_time_of_each_rank.png]]
-
-:LUCAS:
-Since I am not sure I understood correctly your request, the plot
-=img/exp2_exp3_iterations_only_max.png= contains the max execution time
-(on Y) of each of the 10 iterations (the horizontal facets) taking
-into account all ranks belonging to each of the five NP experiments
-(on X).
-:END:
-
-#+begin_src R :results output graphics :file img/exp2_exp3_iterations_only_max.png :exports both :width 1200 :height 400 :session
-library(ggplot2);
-df %>%
- select(-Start, -End) %>%
- group_by(Iteration, NP, Partitioning) %>%
- summarize(Time = max(Duration), N=n()) %>%
- ungroup() %>%
- mutate(NP=as.factor(NP)) %>%
- ggplot(aes(x=NP, y=Time, color=Partitioning, group=Partitioning)) +
- theme_bw(base_size=12) +
- geom_point() +
- geom_line() +
- ylim(0, NA) +
- facet_grid(~Iteration) +
- theme(legend.position="top")
-#+end_src
-
-#+RESULTS:
-[[file:img/exp2_exp3_iterations_only_max.png]]
-** 23-node nova (367 352 336 272 208 144 80 16) :EXP4:
-*** Preparation (2-nodes only)
-**** Convert to CSV
-I need on the $PATH:
-- akypuera's otf22paje (compiled using OTF2 libraries of ScoreP 3.0)
-- pajeng's =pj_dump=
-
-#+begin_src shell :results output
-export PATH=$PATH:/home/schnorr/install/stow/bin/
-export PATH=$PATH:/home/schnorr/dev/pajeng/b/
-
-convert() {
- pushd $(dirname $otf2)
- otf22paje traces.otf2 | pj_dump | grep ^State | cut -d, -f2,4,5,8 | sed -e "s/ //g" -e "s/MPIRank//" | gzip > traces.csv.gz
- popd
-}
-
-EDIR=exp_prep-4-v2_grisou_4
-# Files already converted (whose CSV size is not zero)
-EXISTINGFILE=$(tempfile)
-OTF2FILE=$(tempfile)
-find $EDIR -not -empty | grep csv$ | sed -e "s/.csv$//" | sort > $EXISTINGFILE
-find $EDIR | grep otf2 | sed -e "s/.otf2$//" | sort > $OTF2FILE
-
-for otf2 in $(comm -3 $OTF2FILE $EXISTINGFILE | sed "s/$/.otf2/"); do
- echo $otf2
- convert $otf2
-done
-#+end_src
-
-#+RESULTS:
-: exp_prep-4-v2_grisou_4/prep-4_grisou_4_63_metis.dir/scorep_prep-4_grisou_4_63_metis/traces.otf2
-: ~/dev/Alya-Perf/exp_prep-4-v2_grisou_4/prep-4_grisou_4_63_metis.dir/scorep_prep-4_grisou_4_63_metis ~/dev/Alya-Perf
-: ~/dev/Alya-Perf
-: exp_prep-4-v2_grisou_4/prep-4_grisou_4_63_sfc.dir/scorep_prep-4_grisou_4_63_sfc/traces.otf2
-: ~/dev/Alya-Perf/exp_prep-4-v2_grisou_4/prep-4_grisou_4_63_sfc.dir/scorep_prep-4_grisou_4_63_sfc ~/dev/Alya-Perf
-: ~/dev/Alya-Perf
-**** Extract iteration information :ATTACH:
-:PROPERTIES:
-:Attachments: exp_prep-4_iterations.csv.gz
-:ID: b0e3978c-4131-4005-b979-a86aad031816
-:END:
-#+begin_src shell :results output
-./do.R \
-exp_prep-4-v1_grisou_4/prep-4_grisou_4_18_sfc.dir/scorep_prep-4_grisou_4_18_sfc/traces.csv.gz \
-exp_prep-4-v1_grisou_4/prep-4_grisou_4_33_sfc.dir/scorep_prep-4_grisou_4_33_sfc/traces.csv.gz \
-exp_prep-4-v1_grisou_4/prep-4_grisou_4_49_metis.dir/scorep_prep-4_grisou_4_49_metis/traces.csv.gz \
-exp_prep-4-v1_grisou_4/prep-4_grisou_4_63_sfc.dir/scorep_prep-4_grisou_4_63_sfc/traces.csv.gz \
-exp_prep-4-v1_grisou_4/prep-4_grisou_4_33_metis.dir/scorep_prep-4_grisou_4_33_metis/traces.csv.gz \
-exp_prep-4-v1_grisou_4/prep-4_grisou_4_18_metis.dir/scorep_prep-4_grisou_4_18_metis/traces.csv.gz \
-exp_prep-4-v1_grisou_4/prep-4_grisou_4_49_sfc.dir/scorep_prep-4_grisou_4_49_sfc/traces.csv.gz \
-exp_prep-4-v1_grisou_4/prep-4_grisou_4_63_metis.dir/scorep_prep-4_grisou_4_63_metis/traces.csv.gz \
-exp_prep-4-v2_grisou_4/prep-4_grisou_4_63_sfc.dir/scorep_prep-4_grisou_4_63_sfc/traces.csv.gz \
-exp_prep-4-v2_grisou_4/prep-4_grisou_4_63_metis.dir/scorep_prep-4_grisou_4_63_metis/traces.csv.gz \
-#+end_src
-
-Results are saved in file compressed CSV file called =exp_prep-4.csv.gz=.
-**** Analysis
-Forthcoming.
-*** 23-Node data
-**** Convert to CSV
-I need on the $PATH:
-- akypuera's otf22paje (compiled using OTF2 libraries of ScoreP 3.0)
-- pajeng's =pj_dump=
-
-#+begin_src shell :results output
-export PATH=$PATH:/home/lschnorr/akypuera/b/
-export PATH=$PATH:/home/lschnorr/pajeng/b/
-
-convert() {
- pushd $(dirname $otf2)
- otf22paje traces.otf2 | pj_dump | grep ^State | cut -d, -f2,4,5,8 | sed -e "s/ //g" -e "s/MPIRank//" | gzip > traces.csv.gz
- popd
-}
-
-EDIR=exp_4_grisou_23_v1/
-# Files already converted (whose CSV size is not zero)
-EXISTINGFILE=$(tempfile)
-OTF2FILE=$(tempfile)
-find $EDIR -not -empty | grep csv.gz$ | sed -e "s/.csv.gz$//" | sort > $EXISTINGFILE
-find $EDIR | grep \.otf2$ | sed -e "s/.otf2$//" | sort > $OTF2FILE
-
-for otf2 in $(comm -3 $OTF2FILE $EXISTINGFILE | sed "s/$/.otf2/"); do
- echo $otf2
- convert $otf2
-done
-#+end_src
-
-#+RESULTS:
-#+begin_example
-exp_3_grisou_10/3_grisou_10_113_metis.dir/scorep_3_grisou_10_113_metis/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_113_metis.dir/scorep_3_grisou_10_113_metis ~/dev/Alya-Perf
-~/dev/Alya-Perf
-exp_3_grisou_10/3_grisou_10_113_sfc.dir/scorep_3_grisou_10_113_sfc/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_113_sfc.dir/scorep_3_grisou_10_113_sfc ~/dev/Alya-Perf
-~/dev/Alya-Perf
-exp_3_grisou_10/3_grisou_10_129_metis.dir/scorep_3_grisou_10_129_metis/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_129_metis.dir/scorep_3_grisou_10_129_metis ~/dev/Alya-Perf
-~/dev/Alya-Perf
-exp_3_grisou_10/3_grisou_10_129_sfc.dir/scorep_3_grisou_10_129_sfc/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_129_sfc.dir/scorep_3_grisou_10_129_sfc ~/dev/Alya-Perf
-~/dev/Alya-Perf
-exp_3_grisou_10/3_grisou_10_145_metis.dir/scorep_3_grisou_10_145_metis/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_145_metis.dir/scorep_3_grisou_10_145_metis ~/dev/Alya-Perf
-~/dev/Alya-Perf
-exp_3_grisou_10/3_grisou_10_145_sfc.dir/scorep_3_grisou_10_145_sfc/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_145_sfc.dir/scorep_3_grisou_10_145_sfc ~/dev/Alya-Perf
-~/dev/Alya-Perf
-exp_3_grisou_10/3_grisou_10_160_metis.dir/scorep_3_grisou_10_160_metis/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_160_metis.dir/scorep_3_grisou_10_160_metis ~/dev/Alya-Perf
-~/dev/Alya-Perf
-exp_3_grisou_10/3_grisou_10_160_sfc.dir/scorep_3_grisou_10_160_sfc/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_160_sfc.dir/scorep_3_grisou_10_160_sfc ~/dev/Alya-Perf
-~/dev/Alya-Perf
-exp_3_grisou_10/3_grisou_10_97_metis.dir/scorep_3_grisou_10_97_metis/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_97_metis.dir/scorep_3_grisou_10_97_metis ~/dev/Alya-Perf
-~/dev/Alya-Perf
-exp_3_grisou_10/3_grisou_10_97_sfc.dir/scorep_3_grisou_10_97_sfc/traces.otf2
-~/dev/Alya-Perf/exp_3_grisou_10/3_grisou_10_97_sfc.dir/scorep_3_grisou_10_97_sfc ~/dev/Alya-Perf
-~/dev/Alya-Perf
-#+end_example
-**** Post-processing in R
-#+begin_src R :results output :session :exports both :tangle do-exp4.R :tangle-mode (identity #o755)
-#!/usr/bin/Rscript
-library(readr);
-library(dplyr);
-alya_scorep_trace_read <- function(filename)
-{
- meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[3], "_"));
- read_csv(filename,
- col_names=c("Rank", "Start", "End", "Value"),
- progress=TRUE) %>%
- # Transform Value to factor
- mutate(Value = as.factor(Value)) %>%
- # Detect begin and end of iterations
- mutate(Iteration = case_when(
- grepl("timste", .$Value) ~ 1,
- grepl("endste", .$Value) ~ -1,
- TRUE ~ 0)) %>%
- group_by(Rank) %>%
- # Create a logical to detect observations within iterations
- mutate(Iteration = as.logical(cumsum(Iteration))) %>%
- # Get only observations that belongs to some iteration
- filter(Iteration == TRUE) %>%
- ungroup() %>%
- # Create the iteration by cumsum
- mutate(Iteration = case_when(
- grepl("timste", .$Value) ~ 1,
- TRUE ~ 0)) %>%
- group_by(Rank) %>%
- mutate(Iteration = as.integer(cumsum(Iteration))) %>%
- ungroup() %>%
- # Define metadata
- mutate(EID = meta[2],
- Platform = meta[3],
- Nodes = meta[4],
- NP = meta[5],
- Partitioning = meta[6]);
-}
-
-alya_scorep_trace_iterations <- function(filename)
-{
- alya_scorep_trace_read(filename) %>%
- group_by(Rank, Iteration, Platform, Nodes, NP, Partitioning, EID) %>%
- filter(grepl("MPI_", Value)) %>%
- summarize(N=n(), S=min(Start), E=max(End), Comm=sum(End-Start), Comp=(E-S)-Comm);
-}
-
-args = commandArgs(trailingOnly=TRUE)
-print(args);
-df <- do.call("rbind", lapply(args, function(x) { alya_scorep_trace_iterations(x) }));
-write.csv(df, "exp4_iterations.csv");
-#+end_src
-
-**** Read in R :ATTACH:
-:PROPERTIES:
-:Attachments: exp4_iterations.csv.gz
-:ID: 3b264f0c-58ba-47c0-b291-b5729da5e669
-:END:
-
-#+begin_src R :results output :session :exports both
-library(readr);
-library(dplyr);
-df <- read_csv("data/3b/264f0c-58ba-47c0-b291-b5729da5e669/exp4_iterations.csv.gz") %>%
- select(-X1) %>%
- rename(Start = S, End = E) %>%
- mutate(Duration = End - Start) %>%
- mutate(Platform = as.factor(Platform),
- Partitioning = as.factor(Partitioning),
- Nodes = as.factor(Nodes));
-df %>% summary;
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Parsed with column specification:
-cols(
- X1 = col_integer(),
- Rank = col_integer(),
- Iteration = col_integer(),
- Platform = col_character(),
- Nodes = col_integer(),
- NP = col_integer(),
- Partitioning = col_character(),
- EID = col_integer(),
- N = col_integer(),
- S = col_double(),
- E = col_double(),
- Comm = col_double(),
- Comp = col_double()
-)
-Warning message:
-Missing column names filled in: 'X1' [1]
- Rank Iteration Platform Nodes NP
- Min. : 0.0 Min. : 1.000 grisou:19613 23:19613 Min. : 17.0
- 1st Qu.: 61.0 1st Qu.: 2.000 1st Qu.:209.0
- Median :132.0 Median : 5.000 Median :337.0
- Mean :145.2 Mean : 5.091 Mean :291.5
- 3rd Qu.:221.0 3rd Qu.: 8.000 3rd Qu.:353.0
- Max. :367.0 Max. :10.000 Max. :368.0
- Partitioning EID N Start End
- metis: 1783 Min. :4 Min. : 3579 Min. : 131.6 Min. : 145.8
- sfc :17830 1st Qu.:4 1st Qu.: 29737 1st Qu.: 151.6 1st Qu.: 166.4
- Median :4 Median : 38587 Median : 189.7 Median : 201.3
- Mean :4 Mean : 39148 Mean : 199.2 Mean : 214.1
- 3rd Qu.:4 3rd Qu.: 47123 3rd Qu.: 223.9 3rd Qu.: 234.6
- Max. :4 Max. :103825 Max. :1417.6 Max. :1527.0
- Comm Comp Duration
- Min. : 0.539 Min. : 0.01607 Min. : 8.054
- 1st Qu.: 5.304 1st Qu.: 4.92897 1st Qu.: 10.519
- Median : 5.926 Median : 5.75274 Median : 11.189
- Mean : 6.596 Mean : 8.34015 Mean : 14.936
- 3rd Qu.: 7.522 3rd Qu.: 8.05452 3rd Qu.: 15.129
- Max. :169.634 Max. :167.55322 Max. :169.672
-#+end_example
-**** Plot
-#+begin_src R :results output :session :exports both
-df %>% filter(Partitioning == "metis") %>% tail
-#+end_src
-
-#+RESULTS:
-#+begin_example
-# A tibble: 6 x 13
- Rank Iteration Platform Nodes NP Partitioning EID N Start
-
-1 331 1 grisou 23 337 metis 4 19477 149.0451
-2 332 1 grisou 23 337 metis 4 31973 149.0405
-3 333 1 grisou 23 337 metis 4 35097 149.0563
-4 334 1 grisou 23 337 metis 4 35097 149.0551
-5 335 1 grisou 23 337 metis 4 41345 149.0725
-6 336 1 grisou 23 337 metis 4 28849 149.0405
-# ... with 4 more variables: End , Comm , Comp , Duration
-#+end_example
-
-#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 500 :height 400 :session
-library(ggplot2);
-library(tidyr);
-df %>%
- select(-Start, -End, -N, -Duration) %>%
- mutate(Total = Comm+Comp) %>%
- gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID) %>%
- group_by (Iteration, Platform, Nodes, NP, Partitioning, EID, Variable) %>%
- summarize(Mean = mean(Value), SE = 3*sd(Value)/sqrt(n()), N=n()) %>%
- filter(NP == 368) %>%
- ggplot(aes(x=Iteration, y=Mean, ymin=Mean-SE, ymax=Mean+SE, color=Partitioning)) +
- theme_bw(base_size=12) +
- ylim(0,NA) +
- geom_point() +
- geom_errorbar(width=.3) +
- geom_line () +
- theme(legend.position="top") +
- facet_grid(NP~Variable)
-#+end_src
-
-#+RESULTS:
-[[file:/tmp/babel-24763lPR/figure24763z4R.png]]
-
-
-
-Unfortunately, Metis as misconfigured in lyon, it ran only 1
-iteration.
-**** Plot with min/max
-
-#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 500 :height 400 :session
-library(ggplot2);
-library(tidyr);
-df %>%
- filter(Rank != 0) %>%
- filter(Partitioning == "sfc") %>%
- select(-Start, -End, -N, -Duration) %>%
- mutate(Total = Comm+Comp) %>%
- gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID) %>%
- filter(NP == 368) %>%
- ggplot(aes(x=Iteration, y=Value, group=Iteration)) +
- theme_bw(base_size=12) +
- ylim(0,NA) +
- geom_boxplot() +
- theme(legend.position="top") +
- facet_grid(Partitioning~Variable)
-#+end_src
-
-#+RESULTS:
-[[file:/tmp/babel-24763lPR/figure24763AKM.png]]
-**** Gantt to check MPI calls
-***** Read the trace
-
-#+begin_src R :results output :session :exports both
-library(readr);
-library(dplyr);
-alya_scorep_trace_read <- function(filename)
-{
- meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[3], "_"));
- read_csv(filename,
- col_names=c("Rank", "Start", "End", "Value"),
- progress=TRUE) %>%
- # Transform Value to factor
- mutate(Value = as.factor(Value)) %>%
- # Detect begin and end of iterations
- mutate(Iteration = case_when(
- grepl("timste", .$Value) ~ 1,
- grepl("endste", .$Value) ~ -1,
- TRUE ~ 0)) %>%
- group_by(Rank) %>%
- # Create a logical to detect observations within iterations
- mutate(Iteration = as.logical(cumsum(Iteration))) %>%
- # Get only observations that belongs to some iteration
- filter(Iteration == TRUE) %>%
- ungroup() %>%
- # Create the iteration by cumsum
- mutate(Iteration = case_when(
- grepl("timste", .$Value) ~ 1,
- TRUE ~ 0)) %>%
- group_by(Rank) %>%
- mutate(Iteration = as.integer(cumsum(Iteration))) %>%
- ungroup() %>%
- # Define metadata
- mutate(EID = meta[2],
- Platform = meta[3],
- Nodes = meta[4],
- NP = meta[5],
- Partitioning = meta[6]);
-}
-TRACE <- "exp_prep-4-v1_grisou_4/prep-4_grisou_4_63_sfc.dir/scorep_prep-4_grisou_4_63_sfc/traces.csv.gz"
-df4.mpi <- alya_scorep_trace_read(TRACE);
-#+end_src
-
-#+RESULTS:
-: Parsed with column specification:
-: cols(
-: Rank = col_integer(),
-: Start = col_double(),
-: End = col_double(),
-: Value = col_character()
-: )
-:
|= | 2% 22 MB
|= | 2% 22 MB
|= | 2% 22 MB
|= | 2% 22 MB
|= | 2% 22 MB
|= | 2% 22 MB
|= | 2% 22 MB
|= | 2% 23 MB
|= | 2% 23 MB
|= | 3% 23 MB
|= | 3% 23 MB
|= | 3% 23 MB
|= | 3% 23 MB
|= | 3% 23 MB
|= | 3% 23 MB
|= | 3% 23 MB
|== | 3% 23 MB
|== | 3% 23 MB
|== | 3% 23 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 24 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 25 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 26 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 27 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 28 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 29 MB
|== | 3% 30 MB
|== | 3% 30 MB
|== | 3% 30 MB
|== | 3% 30 MB
|== | 3% 30 MB
|== | 3% 30 MB
|== | 3% 30 MB
|== | 3% 30 MB
|== | 3% 30 MB
|== | 3% 30 MB
|== | 4% 30 MB
|== | 4% 30 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 31 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 32 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 33 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 34 MB
|== | 4% 35 MB
|== | 4% 35 MB
|== | 4% 35 MB
|== | 4% 35 MB
|== | 4% 35 MB
|== | 4% 35 MB
|== | 4% 35 MB
|=== | 4% 35 MB
|=== | 4% 35 MB
|=== | 4% 35 MB
|=== | 4% 35 MB
|=== | 4% 35 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 36 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 37 MB
|=== | 4% 38 MB
|=== | 4% 38 MB
|=== | 4% 38 MB
|=== | 4% 38 MB
|=== | 4% 38 MB
|=== | 4% 38 MB
|=== | 4% 38 MB
|=== | 5% 38 MB
|=== | 5% 38 MB
|=== | 5% 38 MB
|=== | 5% 38 MB
|=== | 5% 38 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 39 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 40 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 41 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 42 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 43 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 44 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 45 MB
|=== | 5% 46 MB
|=== | 5% 46 MB
|=== | 5% 46 MB
|=== | 6% 46 MB
|=== | 6% 46 MB
|=== | 6% 46 MB
|=== | 6% 46 MB
|=== | 6% 46 MB
|=== | 6% 46 MB
|=== | 6% 46 MB
|=== | 6% 46 MB
|=== | 6% 46 MB
|=== | 6% 47 MB
|=== | 6% 47 MB
|=== | 6% 47 MB
|=== | 6% 47 MB
|=== | 6% 47 MB
|==== | 6% 47 MB
|==== | 6% 47 MB
|==== | 6% 47 MB
|==== | 6% 47 MB
|==== | 6% 47 MB
|==== | 6% 47 MB
|==== | 6% 47 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 48 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 49 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 50 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 51 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 52 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 6% 53 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 54 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 55 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 56 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 57 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 58 MB
|==== | 7% 59 MB
|==== | 7% 59 MB
|==== | 7% 59 MB
|==== | 7% 59 MB
|===== | 7% 59 MB
|===== | 7% 59 MB
|===== | 7% 59 MB
|===== | 7% 59 MB
|===== | 7% 59 MB
|===== | 7% 59 MB
|===== | 7% 59 MB
|===== | 7% 59 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 60 MB
|===== | 7% 61 MB
|===== | 7% 61 MB
|===== | 7% 61 MB
|===== | 7% 61 MB
|===== | 7% 61 MB
|===== | 7% 61 MB
|===== | 7% 61 MB
|===== | 7% 61 MB
|===== | 8% 61 MB
|===== | 8% 61 MB
|===== | 8% 61 MB
|===== | 8% 61 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 62 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 63 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 64 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 65 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 66 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 67 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 68 MB
|===== | 8% 69 MB
|===== | 8% 69 MB
|===== | 8% 69 MB
|===== | 8% 69 MB
|===== | 8% 69 MB
|===== | 9% 69 MB
|===== | 9% 69 MB
|===== | 9% 69 MB
|===== | 9% 69 MB
|===== | 9% 69 MB
|===== | 9% 69 MB
|===== | 9% 69 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 70 MB
|===== | 9% 71 MB
|===== | 9% 71 MB
|====== | 9% 71 MB
|====== | 9% 71 MB
|====== | 9% 71 MB
|====== | 9% 71 MB
|====== | 9% 71 MB
|====== | 9% 71 MB
|====== | 9% 71 MB
|====== | 9% 71 MB
|====== | 9% 71 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 72 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 73 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 74 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 75 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 76 MB
|====== | 9% 77 MB
|====== | 9% 77 MB
|====== | 10% 77 MB
|====== | 10% 77 MB
|====== | 10% 77 MB
|====== | 10% 77 MB
|====== | 10% 77 MB
|====== | 10% 77 MB
|====== | 10% 77 MB
|====== | 10% 77 MB
|====== | 10% 77 MB
|====== | 10% 77 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 78 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 79 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 80 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 81 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 82 MB
|====== | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 83 MB
|======= | 10% 84 MB
|======= | 10% 84 MB
|======= | 10% 84 MB
|======= | 10% 84 MB
|======= | 10% 84 MB
|======= | 10% 84 MB
|======= | 10% 84 MB
|======= | 10% 84 MB
|======= | 10% 84 MB
|======= | 10% 84 MB
|======= | 11% 84 MB
|======= | 11% 84 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 85 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 86 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 87 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 88 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 89 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 90 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 91 MB
|======= | 11% 92 MB
|======= | 11% 92 MB
|======= | 11% 92 MB
|======= | 11% 92 MB
|======= | 11% 92 MB
|======= | 11% 92 MB
|======= | 12% 92 MB
|======= | 12% 92 MB
|======= | 12% 92 MB
|======= | 12% 92 MB
|======= | 12% 92 MB
|======= | 12% 92 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 93 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======= | 12% 94 MB
|======== | 12% 94 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 95 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 96 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 97 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 98 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 99 MB
|======== | 12% 100 MB
|======== | 12% 100 MB
|======== | 12% 100 MB
|======== | 13% 100 MB
|======== | 13% 100 MB
|======== | 13% 100 MB
|======== | 13% 100 MB
|======== | 13% 100 MB
|======== | 13% 100 MB
|======== | 13% 100 MB
|======== | 13% 100 MB
|======== | 13% 100 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 101 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 102 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 103 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 104 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 105 MB
|======== | 13% 106 MB
|======== | 13% 106 MB
|======== | 13% 106 MB
|======== | 13% 106 MB
|======== | 13% 106 MB
|======== | 13% 106 MB
|======== | 13% 106 MB
|======== | 13% 106 MB
|======== | 13% 106 MB
|========= | 13% 106 MB
|========= | 13% 106 MB
|========= | 13% 106 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 13% 107 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 108 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 109 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 110 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 111 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 112 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 113 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 114 MB
|========= | 14% 115 MB
|========= | 14% 115 MB
|========= | 14% 115 MB
|========= | 14% 115 MB
|========= | 14% 115 MB
|========= | 14% 115 MB
|========= | 14% 115 MB
|========= | 14% 115 MB
|========= | 15% 115 MB
|========= | 15% 115 MB
|========= | 15% 115 MB
|========= | 15% 115 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 116 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 117 MB
|========= | 15% 118 MB
|========= | 15% 118 MB
|========= | 15% 118 MB
|========= | 15% 118 MB
|========= | 15% 118 MB
|========= | 15% 118 MB
|========= | 15% 118 MB
|========= | 15% 118 MB
|========== | 15% 118 MB
|========== | 15% 118 MB
|========== | 15% 118 MB
|========== | 15% 118 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 119 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 120 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 121 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 122 MB
|========== | 15% 123 MB
|========== | 15% 123 MB
|========== | 15% 123 MB
|========== | 15% 123 MB
|========== | 15% 123 MB
|========== | 16% 123 MB
|========== | 16% 123 MB
|========== | 16% 123 MB
|========== | 16% 123 MB
|========== | 16% 123 MB
|========== | 16% 123 MB
|========== | 16% 123 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 124 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 125 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 126 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 127 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 128 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 129 MB
|========== | 16% 130 MB
|========== | 16% 130 MB
|========== | 16% 130 MB
|========== | 16% 130 MB
|========== | 16% 130 MB
|========== | 16% 130 MB
|=========== | 16% 130 MB
|=========== | 16% 130 MB
|=========== | 16% 130 MB
|=========== | 16% 130 MB
|=========== | 16% 130 MB
|=========== | 16% 131 MB
|=========== | 16% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 131 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 132 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 133 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 134 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 135 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 136 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 137 MB
|=========== | 17% 138 MB
|=========== | 17% 138 MB
|=========== | 17% 138 MB
|=========== | 17% 138 MB
|=========== | 17% 138 MB
|=========== | 17% 138 MB
|=========== | 17% 138 MB
|=========== | 17% 138 MB
|=========== | 17% 138 MB
|=========== | 18% 138 MB
|=========== | 18% 138 MB
|=========== | 18% 138 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 139 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 140 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 141 MB
|=========== | 18% 142 MB
|=========== | 18% 142 MB
|=========== | 18% 142 MB
|=========== | 18% 142 MB
|============ | 18% 142 MB
|============ | 18% 142 MB
|============ | 18% 142 MB
|============ | 18% 142 MB
|============ | 18% 142 MB
|============ | 18% 142 MB
|============ | 18% 142 MB
|============ | 18% 142 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 143 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 144 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 145 MB
|============ | 18% 146 MB
|============ | 18% 146 MB
|============ | 18% 146 MB
|============ | 18% 146 MB
|============ | 18% 146 MB
|============ | 18% 146 MB
|============ | 19% 146 MB
|============ | 19% 146 MB
|============ | 19% 146 MB
|============ | 19% 146 MB
|============ | 19% 146 MB
|============ | 19% 146 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 147 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 148 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 149 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 150 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 151 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 152 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 153 MB
|============ | 19% 154 MB
|============ | 19% 154 MB
|============ | 19% 154 MB
|============= | 20% 154 MB
|============= | 20% 154 MB
|============= | 20% 154 MB
|============= | 20% 154 MB
|============= | 20% 154 MB
|============= | 20% 154 MB
|============= | 20% 154 MB
|============= | 20% 154 MB
|============= | 20% 154 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 155 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 156 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 157 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 158 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 159 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 160 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 20% 161 MB
|============= | 21% 161 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 162 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 163 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 164 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 165 MB
|============= | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 166 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 167 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 168 MB
|============== | 21% 169 MB
|============== | 21% 169 MB
|============== | 21% 169 MB
|============== | 21% 169 MB
|============== | 21% 169 MB
|============== | 21% 169 MB
|============== | 21% 169 MB
|============== | 22% 169 MB
|============== | 22% 169 MB
|============== | 22% 169 MB
|============== | 22% 169 MB
|============== | 22% 169 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 170 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 171 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 172 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 173 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 174 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 175 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 176 MB
|============== | 22% 177 MB
|============== | 22% 177 MB
|============== | 22% 177 MB
|============== | 22% 177 MB
|============== | 22% 177 MB
|============== | 23% 177 MB
|============== | 23% 177 MB
|============== | 23% 177 MB
|============== | 23% 177 MB
|============== | 23% 177 MB
|============== | 23% 177 MB
|============== | 23% 177 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 178 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 179 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 180 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 181 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 182 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 183 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 184 MB
|=============== | 23% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 185 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 186 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 187 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 188 MB
|=============== | 24% 189 MB
|=============== | 24% 189 MB
|=============== | 24% 189 MB
|=============== | 24% 189 MB
|=============== | 24% 189 MB
|=============== | 24% 189 MB
|=============== | 24% 189 MB
|=============== | 24% 189 MB
|=============== | 24% 189 MB
|=============== | 24% 189 MB
|================ | 24% 189 MB
|================ | 24% 189 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 190 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 191 MB
|================ | 24% 192 MB
|================ | 24% 192 MB
|================ | 24% 192 MB
|================ | 24% 192 MB
|================ | 24% 192 MB
|================ | 24% 192 MB
|================ | 24% 192 MB
|================ | 24% 192 MB
|================ | 24% 192 MB
|================ | 25% 192 MB
|================ | 25% 192 MB
|================ | 25% 192 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 193 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 194 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 195 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 196 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 197 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 198 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 199 MB
|================ | 25% 200 MB
|================ | 25% 200 MB
|================ | 25% 200 MB
|================ | 25% 200 MB
|================ | 25% 200 MB
|================ | 25% 200 MB
|================ | 26% 200 MB
|================ | 26% 200 MB
|================ | 26% 200 MB
|================ | 26% 200 MB
|================ | 26% 200 MB
|================ | 26% 200 MB
|================ | 26% 201 MB
|================ | 26% 201 MB
|================ | 26% 201 MB
|================ | 26% 201 MB
|================ | 26% 201 MB
|================ | 26% 201 MB
|================ | 26% 201 MB
|================ | 26% 201 MB
|================= | 26% 201 MB
|================= | 26% 201 MB
|================= | 26% 201 MB
|================= | 26% 201 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 202 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 203 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 204 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 205 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 206 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 207 MB
|================= | 26% 208 MB
|================= | 26% 208 MB
|================= | 26% 208 MB
|================= | 27% 208 MB
|================= | 27% 208 MB
|================= | 27% 208 MB
|================= | 27% 208 MB
|================= | 27% 208 MB
|================= | 27% 208 MB
|================= | 27% 208 MB
|================= | 27% 208 MB
|================= | 27% 208 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 209 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 210 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 211 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 212 MB
|================= | 27% 213 MB
|================= | 27% 213 MB
|================= | 27% 213 MB
|================= | 27% 213 MB
|================= | 27% 213 MB
|================= | 27% 213 MB
|================= | 27% 213 MB
|================== | 27% 213 MB
|================== | 27% 213 MB
|================== | 27% 213 MB
|================== | 27% 213 MB
|================== | 27% 213 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 214 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 27% 215 MB
|================== | 28% 215 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 216 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 217 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 218 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 219 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 220 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 221 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 222 MB
|================== | 28% 223 MB
|================== | 28% 223 MB
|================== | 28% 223 MB
|================== | 28% 223 MB
|================== | 28% 223 MB
|================== | 28% 223 MB
|================== | 28% 223 MB
|================== | 29% 223 MB
|================== | 29% 223 MB
|================== | 29% 223 MB
|================== | 29% 223 MB
|================== | 29% 223 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 224 MB
|================== | 29% 225 MB
|================== | 29% 225 MB
|================== | 29% 225 MB
|================== | 29% 225 MB
|=================== | 29% 225 MB
|=================== | 29% 225 MB
|=================== | 29% 225 MB
|=================== | 29% 225 MB
|=================== | 29% 225 MB
|=================== | 29% 225 MB
|=================== | 29% 225 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 226 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 227 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 228 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 229 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 230 MB
|=================== | 29% 231 MB
|=================== | 29% 231 MB
|=================== | 29% 231 MB
|=================== | 29% 231 MB
|=================== | 30% 231 MB
|=================== | 30% 231 MB
|=================== | 30% 231 MB
|=================== | 30% 231 MB
|=================== | 30% 231 MB
|=================== | 30% 231 MB
|=================== | 30% 231 MB
|=================== | 30% 231 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 232 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 233 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 234 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 235 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 236 MB
|=================== | 30% 237 MB
|=================== | 30% 237 MB
|=================== | 30% 237 MB
|==================== | 30% 237 MB
|==================== | 30% 237 MB
|==================== | 30% 237 MB
|==================== | 30% 237 MB
|==================== | 30% 237 MB
|==================== | 30% 237 MB
|==================== | 30% 237 MB
|==================== | 30% 237 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 238 MB
|==================== | 30% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 239 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 240 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 241 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 242 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 243 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 244 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 245 MB
|==================== | 31% 246 MB
|==================== | 31% 246 MB
|==================== | 31% 246 MB
|==================== | 31% 246 MB
|==================== | 31% 246 MB
|==================== | 31% 246 MB
|==================== | 31% 246 MB
|==================== | 31% 246 MB
|==================== | 31% 246 MB
|==================== | 32% 246 MB
|==================== | 32% 246 MB
|==================== | 32% 246 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 247 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 248 MB
|==================== | 32% 249 MB
|==================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 249 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 250 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 251 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 252 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 253 MB
|===================== | 32% 254 MB
|===================== | 32% 254 MB
|===================== | 32% 254 MB
|===================== | 32% 254 MB
|===================== | 32% 254 MB
|===================== | 32% 254 MB
|===================== | 33% 254 MB
|===================== | 33% 254 MB
|===================== | 33% 254 MB
|===================== | 33% 254 MB
|===================== | 33% 254 MB
|===================== | 33% 254 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 255 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 256 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 257 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 258 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 259 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|===================== | 33% 260 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 261 MB
|====================== | 33% 262 MB
|====================== | 33% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 262 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 263 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 264 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 265 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 266 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 267 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 268 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 34% 269 MB
|====================== | 35% 269 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 270 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 271 MB
|====================== | 35% 272 MB
|====================== | 35% 272 MB
|====================== | 35% 272 MB
|====================== | 35% 272 MB
|====================== | 35% 272 MB
|====================== | 35% 272 MB
|====================== | 35% 272 MB
|====================== | 35% 272 MB
|====================== | 35% 272 MB
|====================== | 35% 272 MB
|======================= | 35% 272 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 273 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 274 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 275 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 276 MB
|======================= | 35% 277 MB
|======================= | 35% 277 MB
|======================= | 35% 277 MB
|======================= | 35% 277 MB
|======================= | 35% 277 MB
|======================= | 35% 277 MB
|======================= | 35% 277 MB
|======================= | 36% 277 MB
|======================= | 36% 277 MB
|======================= | 36% 277 MB
|======================= | 36% 277 MB
|======================= | 36% 277 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 278 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 279 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 280 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 281 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 282 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 283 MB
|======================= | 36% 284 MB
|======================= | 36% 284 MB
|======================= | 36% 284 MB
|======================= | 36% 284 MB
|======================= | 36% 284 MB
|======================= | 36% 284 MB
|======================= | 36% 284 MB
|======================= | 36% 284 MB
|======================= | 36% 284 MB
|======================== | 36% 284 MB
|======================== | 36% 284 MB
|======================== | 36% 284 MB
|======================== | 36% 285 MB
|======================== | 36% 285 MB
|======================== | 36% 285 MB
|======================== | 36% 285 MB
|======================== | 37% 285 MB
|======================== | 37% 285 MB
|======================== | 37% 285 MB
|======================== | 37% 285 MB
|======================== | 37% 285 MB
|======================== | 37% 285 MB
|======================== | 37% 285 MB
|======================== | 37% 285 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 286 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 287 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 288 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 289 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 290 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 291 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 37% 292 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 293 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 294 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 295 MB
|======================== | 38% 296 MB
|======================== | 38% 296 MB
|======================== | 38% 296 MB
|======================== | 38% 296 MB
|======================== | 38% 296 MB
|======================== | 38% 296 MB
|======================== | 38% 296 MB
|========================= | 38% 296 MB
|========================= | 38% 296 MB
|========================= | 38% 296 MB
|========================= | 38% 296 MB
|========================= | 38% 296 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 297 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 298 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 299 MB
|========================= | 38% 300 MB
|========================= | 38% 300 MB
|========================= | 38% 300 MB
|========================= | 38% 300 MB
|========================= | 38% 300 MB
|========================= | 38% 300 MB
|========================= | 38% 300 MB
|========================= | 38% 300 MB
|========================= | 38% 300 MB
|========================= | 39% 300 MB
|========================= | 39% 300 MB
|========================= | 39% 300 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 301 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 302 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 303 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 304 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 305 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 306 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 307 MB
|========================= | 39% 308 MB
|========================= | 39% 308 MB
|========================= | 39% 308 MB
|========================= | 39% 308 MB
|========================= | 39% 308 MB
|========================== | 40% 308 MB
|========================== | 40% 308 MB
|========================== | 40% 308 MB
|========================== | 40% 308 MB
|========================== | 40% 308 MB
|========================== | 40% 308 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 309 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 310 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 311 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 312 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 313 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 314 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 315 MB
|========================== | 40% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 316 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 317 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 318 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 319 MB
|========================== | 41% 320 MB
|========================== | 41% 320 MB
|========================== | 41% 320 MB
|========================== | 41% 320 MB
|=========================== | 41% 320 MB
|=========================== | 41% 320 MB
|=========================== | 41% 320 MB
|=========================== | 41% 320 MB
|=========================== | 41% 320 MB
|=========================== | 41% 320 MB
|=========================== | 41% 320 MB
|=========================== | 41% 320 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 321 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 322 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 41% 323 MB
|=========================== | 42% 323 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 324 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 325 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 326 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 327 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 328 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 329 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 330 MB
|=========================== | 42% 331 MB
|=========================== | 42% 331 MB
|=========================== | 42% 331 MB
|=========================== | 42% 331 MB
|=========================== | 42% 331 MB
|=========================== | 42% 331 MB
|=========================== | 42% 331 MB
|=========================== | 43% 331 MB
|=========================== | 43% 331 MB
|=========================== | 43% 331 MB
|=========================== | 43% 331 MB
|=========================== | 43% 331 MB
|=========================== | 43% 332 MB
|=========================== | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 332 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 333 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 334 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 335 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 336 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 337 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 338 MB
|============================ | 43% 339 MB
|============================ | 43% 339 MB
|============================ | 43% 339 MB
|============================ | 43% 339 MB
|============================ | 44% 339 MB
|============================ | 44% 339 MB
|============================ | 44% 339 MB
|============================ | 44% 339 MB
|============================ | 44% 339 MB
|============================ | 44% 339 MB
|============================ | 44% 339 MB
|============================ | 44% 339 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 340 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 341 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 342 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 343 MB
|============================ | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 344 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 345 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 44% 346 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 347 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 348 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 349 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 350 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 351 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 352 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 353 MB
|============================= | 45% 354 MB
|============================= | 45% 354 MB
|============================= | 45% 354 MB
|============================= | 45% 354 MB
|============================= | 45% 354 MB
|============================= | 45% 354 MB
|============================= | 45% 354 MB
|============================= | 45% 354 MB
|============================= | 46% 354 MB
|============================= | 46% 354 MB
|============================= | 46% 354 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================= | 46% 355 MB
|============================== | 46% 355 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 356 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 357 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 358 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 359 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 360 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 361 MB
|============================== | 46% 362 MB
|============================== | 46% 362 MB
|============================== | 46% 362 MB
|============================== | 46% 362 MB
|============================== | 46% 362 MB
|============================== | 47% 362 MB
|============================== | 47% 362 MB
|============================== | 47% 362 MB
|============================== | 47% 362 MB
|============================== | 47% 362 MB
|============================== | 47% 362 MB
|============================== | 47% 362 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 363 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 364 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 365 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 366 MB
|============================== | 47% 367 MB
|============================== | 47% 367 MB
|============================== | 47% 367 MB
|============================== | 47% 367 MB
|============================== | 47% 367 MB
|============================== | 47% 367 MB
|============================== | 47% 367 MB
|============================== | 47% 367 MB
|=============================== | 47% 367 MB
|=============================== | 47% 367 MB
|=============================== | 47% 367 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 368 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 369 MB
|=============================== | 47% 370 MB
|=============================== | 47% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 370 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 371 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 372 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 373 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 374 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 375 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 376 MB
|=============================== | 48% 377 MB
|=============================== | 48% 377 MB
|=============================== | 48% 377 MB
|=============================== | 48% 377 MB
|=============================== | 48% 377 MB
|=============================== | 48% 377 MB
|=============================== | 48% 377 MB
|=============================== | 48% 377 MB
|=============================== | 48% 377 MB
|=============================== | 48% 377 MB
|=============================== | 49% 377 MB
|=============================== | 49% 377 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 378 MB
|=============================== | 49% 379 MB
|=============================== | 49% 379 MB
|=============================== | 49% 379 MB
|=============================== | 49% 379 MB
|=============================== | 49% 379 MB
|=============================== | 49% 379 MB
|=============================== | 49% 379 MB
|=============================== | 49% 379 MB
|================================ | 49% 379 MB
|================================ | 49% 379 MB
|================================ | 49% 379 MB
|================================ | 49% 379 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 380 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 381 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 382 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 383 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 384 MB
|================================ | 49% 385 MB
|================================ | 49% 385 MB
|================================ | 49% 385 MB
|================================ | 49% 385 MB
|================================ | 49% 385 MB
|================================ | 49% 385 MB
|================================ | 50% 385 MB
|================================ | 50% 385 MB
|================================ | 50% 385 MB
|================================ | 50% 385 MB
|================================ | 50% 385 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 386 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 387 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 388 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 389 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 390 MB
|================================ | 50% 391 MB
|================================ | 50% 391 MB
|================================ | 50% 391 MB
|================================ | 50% 391 MB
|================================ | 50% 391 MB
|================================ | 50% 391 MB
|================================= | 50% 391 MB
|================================= | 50% 391 MB
|================================= | 50% 391 MB
|================================= | 50% 391 MB
|================================= | 50% 391 MB
|================================= | 50% 391 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 392 MB
|================================= | 50% 393 MB
|================================= | 50% 393 MB
|================================= | 50% 393 MB
|================================= | 51% 393 MB
|================================= | 51% 393 MB
|================================= | 51% 393 MB
|================================= | 51% 393 MB
|================================= | 51% 393 MB
|================================= | 51% 393 MB
|================================= | 51% 393 MB
|================================= | 51% 393 MB
|================================= | 51% 393 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 394 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 395 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 396 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 397 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 398 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 399 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 51% 400 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 401 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 402 MB
|================================= | 52% 403 MB
|================================= | 52% 403 MB
|================================= | 52% 403 MB
|================================= | 52% 403 MB
|================================== | 52% 403 MB
|================================== | 52% 403 MB
|================================== | 52% 403 MB
|================================== | 52% 403 MB
|================================== | 52% 403 MB
|================================== | 52% 403 MB
|================================== | 52% 403 MB
|================================== | 52% 403 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 404 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 405 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 406 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 407 MB
|================================== | 52% 408 MB
|================================== | 52% 408 MB
|================================== | 52% 408 MB
|================================== | 52% 408 MB
|================================== | 52% 408 MB
|================================== | 52% 408 MB
|================================== | 52% 408 MB
|================================== | 52% 408 MB
|================================== | 53% 408 MB
|================================== | 53% 408 MB
|================================== | 53% 408 MB
|================================== | 53% 408 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 409 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 410 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 411 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 412 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 413 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 414 MB
|================================== | 53% 415 MB
|================================== | 53% 415 MB
|================================== | 53% 415 MB
|=================================== | 53% 415 MB
|=================================== | 53% 415 MB
|=================================== | 53% 415 MB
|=================================== | 53% 415 MB
|=================================== | 53% 415 MB
|=================================== | 53% 415 MB
|=================================== | 53% 415 MB
|=================================== | 53% 415 MB
|=================================== | 53% 415 MB
|=================================== | 53% 416 MB
|=================================== | 53% 416 MB
|=================================== | 53% 416 MB
|=================================== | 53% 416 MB
|=================================== | 54% 416 MB
|=================================== | 54% 416 MB
|=================================== | 54% 416 MB
|=================================== | 54% 416 MB
|=================================== | 54% 416 MB
|=================================== | 54% 416 MB
|=================================== | 54% 416 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 417 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 418 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 419 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 420 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 421 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 422 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 423 MB
|=================================== | 54% 424 MB
|=================================== | 54% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 424 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 425 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 426 MB
|=================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 427 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 428 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 429 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 430 MB
|==================================== | 55% 431 MB
|==================================== | 55% 431 MB
|==================================== | 55% 431 MB
|==================================== | 55% 431 MB
|==================================== | 55% 431 MB
|==================================== | 55% 431 MB
|==================================== | 55% 431 MB
|==================================== | 55% 431 MB
|==================================== | 55% 431 MB
|==================================== | 55% 431 MB
|==================================== | 56% 431 MB
|==================================== | 56% 431 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 432 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 433 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 434 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 435 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 436 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 437 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|==================================== | 56% 438 MB
|===================================== | 56% 438 MB
|===================================== | 56% 439 MB
|===================================== | 56% 439 MB
|===================================== | 56% 439 MB
|===================================== | 56% 439 MB
|===================================== | 56% 439 MB
|===================================== | 56% 439 MB
|===================================== | 57% 439 MB
|===================================== | 57% 439 MB
|===================================== | 57% 439 MB
|===================================== | 57% 439 MB
|===================================== | 57% 439 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 440 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 441 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 442 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 443 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 444 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 445 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 446 MB
|===================================== | 57% 447 MB
|===================================== | 57% 447 MB
|===================================== | 57% 447 MB
|===================================== | 58% 447 MB
|===================================== | 58% 447 MB
|===================================== | 58% 447 MB
|===================================== | 58% 447 MB
|===================================== | 58% 447 MB
|===================================== | 58% 447 MB
|===================================== | 58% 447 MB
|===================================== | 58% 447 MB
|===================================== | 58% 447 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 448 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 449 MB
|===================================== | 58% 450 MB
|===================================== | 58% 450 MB
|===================================== | 58% 450 MB
|===================================== | 58% 450 MB
|===================================== | 58% 450 MB
|===================================== | 58% 450 MB
|===================================== | 58% 450 MB
|===================================== | 58% 450 MB
|===================================== | 58% 450 MB
|====================================== | 58% 450 MB
|====================================== | 58% 450 MB
|====================================== | 58% 450 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 451 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 452 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 453 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 58% 454 MB
|====================================== | 59% 454 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 455 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 456 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 457 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 458 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 459 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 460 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 461 MB
|====================================== | 59% 462 MB
|====================================== | 59% 462 MB
|====================================== | 59% 462 MB
|====================================== | 59% 462 MB
|====================================== | 59% 462 MB
|====================================== | 59% 462 MB
|====================================== | 59% 462 MB
|====================================== | 59% 462 MB
|======================================= | 60% 462 MB
|======================================= | 60% 462 MB
|======================================= | 60% 462 MB
|======================================= | 60% 462 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 463 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 464 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 465 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 466 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 467 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 468 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 469 MB
|======================================= | 60% 470 MB
|======================================= | 60% 470 MB
|======================================= | 60% 470 MB
|======================================= | 60% 470 MB
|======================================= | 61% 470 MB
|======================================= | 61% 470 MB
|======================================= | 61% 470 MB
|======================================= | 61% 470 MB
|======================================= | 61% 470 MB
|======================================= | 61% 470 MB
|======================================= | 61% 470 MB
|======================================= | 61% 470 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 471 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 472 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 473 MB
|======================================= | 61% 474 MB
|======================================= | 61% 474 MB
|======================================= | 61% 474 MB
|======================================= | 61% 474 MB
|======================================= | 61% 474 MB
|======================================= | 61% 474 MB
|======================================== | 61% 474 MB
|======================================== | 61% 474 MB
|======================================== | 61% 474 MB
|======================================== | 61% 474 MB
|======================================== | 61% 474 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 475 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 476 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 477 MB
|======================================== | 61% 478 MB
|======================================== | 61% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 478 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 479 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 480 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 481 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 482 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 483 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 484 MB
|======================================== | 62% 485 MB
|======================================== | 62% 485 MB
|======================================== | 62% 485 MB
|======================================== | 62% 485 MB
|======================================== | 62% 485 MB
|======================================== | 62% 485 MB
|======================================== | 62% 485 MB
|======================================== | 62% 485 MB
|======================================== | 62% 485 MB
|======================================== | 62% 485 MB
|======================================== | 63% 485 MB
|======================================== | 63% 485 MB
|======================================== | 63% 486 MB
|======================================== | 63% 486 MB
|======================================== | 63% 486 MB
|======================================== | 63% 486 MB
|======================================== | 63% 486 MB
|========================================= | 63% 486 MB
|========================================= | 63% 486 MB
|========================================= | 63% 486 MB
|========================================= | 63% 486 MB
|========================================= | 63% 486 MB
|========================================= | 63% 486 MB
|========================================= | 63% 486 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 487 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 488 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 489 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 490 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 491 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 492 MB
|========================================= | 63% 493 MB
|========================================= | 63% 493 MB
|========================================= | 63% 493 MB
|========================================= | 63% 493 MB
|========================================= | 63% 493 MB
|========================================= | 63% 493 MB
|========================================= | 64% 493 MB
|========================================= | 64% 493 MB
|========================================= | 64% 493 MB
|========================================= | 64% 493 MB
|========================================= | 64% 493 MB
|========================================= | 64% 493 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 494 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 495 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 496 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 497 MB
|========================================= | 64% 498 MB
|========================================= | 64% 498 MB
|========================================= | 64% 498 MB
|========================================== | 64% 498 MB
|========================================== | 64% 498 MB
|========================================== | 64% 498 MB
|========================================== | 64% 498 MB
|========================================== | 64% 498 MB
|========================================== | 64% 498 MB
|========================================== | 64% 498 MB
|========================================== | 64% 498 MB
|========================================== | 64% 498 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 499 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 500 MB
|========================================== | 64% 501 MB
|========================================== | 64% 501 MB
|========================================== | 65% 501 MB
|========================================== | 65% 501 MB
|========================================== | 65% 501 MB
|========================================== | 65% 501 MB
|========================================== | 65% 501 MB
|========================================== | 65% 501 MB
|========================================== | 65% 501 MB
|========================================== | 65% 501 MB
|========================================== | 65% 501 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 502 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 503 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 504 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 505 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 506 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 507 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 65% 508 MB
|========================================== | 66% 508 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 509 MB
|========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 510 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 511 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 512 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 513 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 514 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 515 MB
|=========================================== | 66% 516 MB
|=========================================== | 66% 516 MB
|=========================================== | 66% 516 MB
|=========================================== | 66% 516 MB
|=========================================== | 66% 516 MB
|=========================================== | 66% 516 MB
|=========================================== | 66% 516 MB
|=========================================== | 66% 516 MB
|=========================================== | 67% 516 MB
|=========================================== | 67% 516 MB
|=========================================== | 67% 516 MB
|=========================================== | 67% 516 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 517 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 518 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 519 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 520 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|=========================================== | 67% 521 MB
|============================================ | 67% 521 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 522 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 523 MB
|============================================ | 67% 524 MB
|============================================ | 67% 524 MB
|============================================ | 67% 524 MB
|============================================ | 67% 524 MB
|============================================ | 68% 524 MB
|============================================ | 68% 524 MB
|============================================ | 68% 524 MB
|============================================ | 68% 524 MB
|============================================ | 68% 524 MB
|============================================ | 68% 524 MB
|============================================ | 68% 524 MB
|============================================ | 68% 524 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 525 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 526 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 527 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 528 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 529 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 530 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 531 MB
|============================================ | 68% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 532 MB
|============================================ | 69% 533 MB
|============================================ | 69% 533 MB
|============================================ | 69% 533 MB
|============================================ | 69% 533 MB
|============================================ | 69% 533 MB
|============================================ | 69% 533 MB
|============================================ | 69% 533 MB
|============================================ | 69% 533 MB
|============================================ | 69% 533 MB
|============================================ | 69% 533 MB
|============================================= | 69% 533 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 534 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 535 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 536 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 537 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 538 MB
|============================================= | 69% 539 MB
|============================================= | 69% 539 MB
|============================================= | 69% 539 MB
|============================================= | 69% 539 MB
|============================================= | 69% 539 MB
|============================================= | 69% 539 MB
|============================================= | 69% 539 MB
|============================================= | 69% 539 MB
|============================================= | 69% 539 MB
|============================================= | 70% 539 MB
|============================================= | 70% 539 MB
|============================================= | 70% 539 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 540 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 541 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 542 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 543 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 544 MB
|============================================= | 70% 545 MB
|============================================= | 70% 545 MB
|============================================= | 70% 545 MB
|============================================= | 70% 545 MB
|============================================= | 70% 545 MB
|============================================= | 70% 545 MB
|============================================= | 70% 545 MB
|============================================= | 70% 545 MB
|============================================== | 70% 545 MB
|============================================== | 70% 545 MB
|============================================== | 70% 545 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 546 MB
|============================================== | 70% 547 MB
|============================================== | 70% 547 MB
|============================================== | 70% 547 MB
|============================================== | 70% 547 MB
|============================================== | 70% 547 MB
|============================================== | 70% 547 MB
|============================================== | 71% 547 MB
|============================================== | 71% 547 MB
|============================================== | 71% 547 MB
|============================================== | 71% 547 MB
|============================================== | 71% 547 MB
|============================================== | 71% 547 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 548 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 549 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 550 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 551 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 552 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 553 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 554 MB
|============================================== | 71% 555 MB
|============================================== | 71% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 555 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 556 MB
|============================================== | 72% 557 MB
|============================================== | 72% 557 MB
|============================================== | 72% 557 MB
|============================================== | 72% 557 MB
|============================================== | 72% 557 MB
|============================================== | 72% 557 MB
|============================================== | 72% 557 MB
|=============================================== | 72% 557 MB
|=============================================== | 72% 557 MB
|=============================================== | 72% 557 MB
|=============================================== | 72% 557 MB
|=============================================== | 72% 557 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 558 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 559 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 560 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 561 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 72% 562 MB
|=============================================== | 73% 562 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 563 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 564 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 565 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 566 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 567 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 568 MB
|=============================================== | 73% 569 MB
|=============================================== | 73% 569 MB
|=============================================== | 73% 569 MB
|=============================================== | 73% 569 MB
|=============================================== | 73% 569 MB
|================================================ | 73% 569 MB
|================================================ | 73% 569 MB
|================================================ | 73% 569 MB
|================================================ | 73% 569 MB
|================================================ | 73% 569 MB
|================================================ | 73% 569 MB
|================================================ | 73% 569 MB
|================================================ | 73% 570 MB
|================================================ | 73% 570 MB
|================================================ | 73% 570 MB
|================================================ | 73% 570 MB
|================================================ | 73% 570 MB
|================================================ | 73% 570 MB
|================================================ | 73% 570 MB
|================================================ | 73% 570 MB
|================================================ | 74% 570 MB
|================================================ | 74% 570 MB
|================================================ | 74% 570 MB
|================================================ | 74% 570 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 571 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 572 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 573 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 574 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 575 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 576 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 577 MB
|================================================ | 74% 578 MB
|================================================ | 74% 578 MB
|================================================ | 74% 578 MB
|================================================ | 74% 578 MB
|================================================ | 75% 578 MB
|================================================ | 75% 578 MB
|================================================ | 75% 578 MB
|================================================ | 75% 578 MB
|================================================ | 75% 578 MB
|================================================ | 75% 578 MB
|================================================ | 75% 578 MB
|================================================ | 75% 578 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 579 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 580 MB
|================================================ | 75% 581 MB
|================================================ | 75% 581 MB
|================================================ | 75% 581 MB
|================================================ | 75% 581 MB
|================================================= | 75% 581 MB
|================================================= | 75% 581 MB
|================================================= | 75% 581 MB
|================================================= | 75% 581 MB
|================================================= | 75% 581 MB
|================================================= | 75% 581 MB
|================================================= | 75% 581 MB
|================================================= | 75% 581 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 582 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 583 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 584 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 585 MB
|================================================= | 75% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 586 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 587 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 588 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 589 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 590 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 591 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 592 MB
|================================================= | 76% 593 MB
|================================================= | 76% 593 MB
|================================================== | 76% 593 MB
|================================================== | 76% 593 MB
|================================================== | 76% 593 MB
|================================================== | 76% 593 MB
|================================================== | 76% 593 MB
|================================================== | 76% 593 MB
|================================================== | 76% 593 MB
|================================================== | 77% 593 MB
|================================================== | 77% 593 MB
|================================================== | 77% 593 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 594 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 595 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 596 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 597 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 598 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 599 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 600 MB
|================================================== | 77% 601 MB
|================================================== | 77% 601 MB
|================================================== | 77% 601 MB
|================================================== | 77% 601 MB
|================================================== | 77% 601 MB
|================================================== | 77% 601 MB
|================================================== | 78% 601 MB
|================================================== | 78% 601 MB
|================================================== | 78% 601 MB
|================================================== | 78% 601 MB
|================================================== | 78% 601 MB
|================================================== | 78% 601 MB
|================================================== | 78% 601 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 602 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 603 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|================================================== | 78% 604 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 605 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 606 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 607 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 608 MB
|=================================================== | 78% 609 MB
|=================================================== | 78% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 609 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 610 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 611 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 612 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 613 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 614 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 615 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|=================================================== | 79% 616 MB
|==================================================== | 80% 616 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 617 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 618 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 619 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 620 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 621 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 622 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 623 MB
|==================================================== | 80% 624 MB
|==================================================== | 80% 624 MB
|==================================================== | 80% 624 MB
|==================================================== | 80% 624 MB
|==================================================== | 80% 624 MB
|==================================================== | 80% 624 MB
|==================================================== | 80% 624 MB
|==================================================== | 80% 624 MB
|==================================================== | 81% 624 MB
|==================================================== | 81% 624 MB
|==================================================== | 81% 624 MB
|==================================================== | 81% 624 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 625 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 626 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 627 MB
|==================================================== | 81% 628 MB
|==================================================== | 81% 628 MB
|==================================================== | 81% 628 MB
|==================================================== | 81% 628 MB
|==================================================== | 81% 628 MB
|==================================================== | 81% 628 MB
|==================================================== | 81% 628 MB
|==================================================== | 81% 628 MB
|==================================================== | 81% 628 MB
|===================================================== | 81% 628 MB
|===================================================== | 81% 628 MB
|===================================================== | 81% 628 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 629 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 630 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 631 MB
|===================================================== | 81% 632 MB
|===================================================== | 81% 632 MB
|===================================================== | 81% 632 MB
|===================================================== | 81% 632 MB
|===================================================== | 82% 632 MB
|===================================================== | 82% 632 MB
|===================================================== | 82% 632 MB
|===================================================== | 82% 632 MB
|===================================================== | 82% 632 MB
|===================================================== | 82% 632 MB
|===================================================== | 82% 632 MB
|===================================================== | 82% 632 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 633 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 634 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 635 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 636 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 637 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 638 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 82% 639 MB
|===================================================== | 83% 640 MB
|===================================================== | 83% 640 MB
|===================================================== | 83% 640 MB
|===================================================== | 83% 640 MB
|===================================================== | 83% 640 MB
|===================================================== | 83% 640 MB
|===================================================== | 83% 640 MB
|====================================================== | 83% 640 MB
|====================================================== | 83% 640 MB
|====================================================== | 83% 640 MB
|====================================================== | 83% 640 MB
|====================================================== | 83% 640 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 641 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 642 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 643 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 644 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 645 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 646 MB
|====================================================== | 83% 647 MB
|====================================================== | 83% 647 MB
|====================================================== | 83% 647 MB
|====================================================== | 83% 647 MB
|====================================================== | 83% 647 MB
|====================================================== | 83% 647 MB
|====================================================== | 83% 647 MB
|====================================================== | 83% 647 MB
|====================================================== | 83% 647 MB
|====================================================== | 84% 647 MB
|====================================================== | 84% 647 MB
|====================================================== | 84% 647 MB
|====================================================== | 84% 647 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 648 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 649 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 650 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 651 MB
|====================================================== | 84% 652 MB
|====================================================== | 84% 652 MB
|====================================================== | 84% 652 MB
|====================================================== | 84% 652 MB
|====================================================== | 84% 652 MB
|====================================================== | 84% 652 MB
|======================================================= | 84% 652 MB
|======================================================= | 84% 652 MB
|======================================================= | 84% 652 MB
|======================================================= | 84% 652 MB
|======================================================= | 84% 652 MB
|======================================================= | 84% 652 MB
|======================================================= | 84% 652 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 653 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 654 MB
|======================================================= | 84% 655 MB
|======================================================= | 84% 655 MB
|======================================================= | 84% 655 MB
|======================================================= | 84% 655 MB
|======================================================= | 84% 655 MB
|======================================================= | 84% 655 MB
|======================================================= | 85% 655 MB
|======================================================= | 85% 655 MB
|======================================================= | 85% 655 MB
|======================================================= | 85% 655 MB
|======================================================= | 85% 655 MB
|======================================================= | 85% 655 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 656 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 657 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 658 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 659 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 660 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 661 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 662 MB
|======================================================= | 85% 663 MB
|======================================================= | 85% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 663 MB
|======================================================= | 86% 664 MB
|======================================================= | 86% 664 MB
|======================================================= | 86% 664 MB
|======================================================= | 86% 664 MB
|======================================================== | 86% 664 MB
|======================================================== | 86% 664 MB
|======================================================== | 86% 664 MB
|======================================================== | 86% 664 MB
|======================================================== | 86% 664 MB
|======================================================== | 86% 664 MB
|======================================================== | 86% 664 MB
|======================================================== | 86% 664 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 665 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 666 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 667 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 668 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 669 MB
|======================================================== | 86% 670 MB
|======================================================== | 86% 670 MB
|======================================================== | 86% 670 MB
|======================================================== | 86% 670 MB
|======================================================== | 86% 670 MB
|======================================================== | 86% 670 MB
|======================================================== | 86% 670 MB
|======================================================== | 86% 670 MB
|======================================================== | 86% 670 MB
|======================================================== | 86% 670 MB
|======================================================== | 87% 670 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 671 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 672 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 673 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 674 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 675 MB
|======================================================== | 87% 676 MB
|======================================================== | 87% 676 MB
|======================================================== | 87% 676 MB
|========================================================= | 87% 676 MB
|========================================================= | 87% 676 MB
|========================================================= | 87% 676 MB
|========================================================= | 87% 676 MB
|========================================================= | 87% 676 MB
|========================================================= | 87% 676 MB
|========================================================= | 87% 676 MB
|========================================================= | 87% 676 MB
|========================================================= | 87% 676 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 677 MB
|========================================================= | 87% 678 MB
|========================================================= | 87% 678 MB
|========================================================= | 87% 678 MB
|========================================================= | 87% 678 MB
|========================================================= | 87% 678 MB
|========================================================= | 87% 678 MB
|========================================================= | 87% 678 MB
|========================================================= | 88% 678 MB
|========================================================= | 88% 678 MB
|========================================================= | 88% 678 MB
|========================================================= | 88% 678 MB
|========================================================= | 88% 678 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 679 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 680 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 681 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 682 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 683 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 684 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 685 MB
|========================================================= | 88% 686 MB
|========================================================= | 88% 686 MB
|========================================================= | 88% 686 MB
|========================================================= | 89% 686 MB
|========================================================= | 89% 686 MB
|========================================================= | 89% 686 MB
|========================================================= | 89% 686 MB
|========================================================= | 89% 686 MB
|========================================================= | 89% 686 MB
|========================================================= | 89% 686 MB
|========================================================= | 89% 686 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 687 MB
|========================================================= | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 688 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 689 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 690 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 691 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 692 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 89% 693 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 694 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 695 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 696 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 697 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 698 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|========================================================== | 90% 699 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 700 MB
|=========================================================== | 90% 701 MB
|=========================================================== | 90% 701 MB
|=========================================================== | 90% 701 MB
|=========================================================== | 90% 701 MB
|=========================================================== | 90% 701 MB
|=========================================================== | 90% 701 MB
|=========================================================== | 90% 701 MB
|=========================================================== | 90% 701 MB
|=========================================================== | 90% 701 MB
|=========================================================== | 91% 701 MB
|=========================================================== | 91% 701 MB
|=========================================================== | 91% 701 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 702 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 703 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 704 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 705 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 706 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 707 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 708 MB
|=========================================================== | 91% 709 MB
|=========================================================== | 91% 709 MB
|=========================================================== | 91% 709 MB
|=========================================================== | 91% 709 MB
|=========================================================== | 91% 709 MB
|=========================================================== | 92% 709 MB
|=========================================================== | 92% 709 MB
|=========================================================== | 92% 709 MB
|=========================================================== | 92% 709 MB
|=========================================================== | 92% 709 MB
|=========================================================== | 92% 709 MB
|=========================================================== | 92% 709 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 710 MB
|=========================================================== | 92% 711 MB
|=========================================================== | 92% 711 MB
|=========================================================== | 92% 711 MB
|=========================================================== | 92% 711 MB
|=========================================================== | 92% 711 MB
|=========================================================== | 92% 711 MB
|=========================================================== | 92% 711 MB
|=========================================================== | 92% 711 MB
|=========================================================== | 92% 711 MB
|============================================================ | 92% 711 MB
|============================================================ | 92% 711 MB
|============================================================ | 92% 711 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 712 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 713 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 714 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 715 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 716 MB
|============================================================ | 92% 717 MB
|============================================================ | 92% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 717 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 718 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 719 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 720 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 721 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 722 MB
|============================================================ | 93% 723 MB
|============================================================ | 93% 723 MB
|============================================================ | 93% 723 MB
|============================================================ | 93% 723 MB
|============================================================ | 93% 723 MB
|============================================================ | 93% 723 MB
|============================================================ | 93% 723 MB
|============================================================= | 93% 723 MB
|============================================================= | 93% 723 MB
|============================================================= | 93% 723 MB
|============================================================= | 93% 723 MB
|============================================================= | 93% 724 MB
|============================================================= | 93% 724 MB
|============================================================= | 93% 724 MB
|============================================================= | 93% 724 MB
|============================================================= | 93% 724 MB
|============================================================= | 93% 724 MB
|============================================================= | 93% 724 MB
|============================================================= | 93% 724 MB
|============================================================= | 93% 724 MB
|============================================================= | 93% 724 MB
|============================================================= | 94% 724 MB
|============================================================= | 94% 724 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 725 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 726 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 727 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 728 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 729 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 730 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 731 MB
|============================================================= | 94% 732 MB
|============================================================= | 94% 732 MB
|============================================================= | 94% 732 MB
|============================================================= | 94% 732 MB
|============================================================= | 94% 732 MB
|============================================================= | 94% 732 MB
|============================================================= | 94% 732 MB
|============================================================= | 95% 732 MB
|============================================================= | 95% 732 MB
|============================================================= | 95% 732 MB
|============================================================= | 95% 732 MB
|============================================================= | 95% 732 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 733 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 734 MB
|============================================================= | 95% 735 MB
|============================================================= | 95% 735 MB
|============================================================= | 95% 735 MB
|============================================================= | 95% 735 MB
|============================================================= | 95% 735 MB
|============================================================= | 95% 735 MB
|============================================================== | 95% 735 MB
|============================================================== | 95% 735 MB
|============================================================== | 95% 735 MB
|============================================================== | 95% 735 MB
|============================================================== | 95% 735 MB
|============================================================== | 95% 735 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 736 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 737 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 738 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 739 MB
|============================================================== | 95% 740 MB
|============================================================== | 95% 740 MB
|============================================================== | 95% 740 MB
|============================================================== | 96% 740 MB
|============================================================== | 96% 740 MB
|============================================================== | 96% 740 MB
|============================================================== | 96% 740 MB
|============================================================== | 96% 740 MB
|============================================================== | 96% 740 MB
|============================================================== | 96% 740 MB
|============================================================== | 96% 740 MB
|============================================================== | 96% 740 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 741 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 742 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 743 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 744 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 745 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 746 MB
|============================================================== | 96% 747 MB
|============================================================== | 96% 747 MB
|============================================================== | 96% 747 MB
|============================================================== | 96% 747 MB
|============================================================== | 96% 747 MB
|=============================================================== | 96% 747 MB
|=============================================================== | 96% 747 MB
|=============================================================== | 96% 747 MB
|=============================================================== | 96% 747 MB
|=============================================================== | 96% 747 MB
|=============================================================== | 96% 747 MB
|=============================================================== | 96% 747 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 748 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 749 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 750 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 751 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 752 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 753 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 754 MB
|=============================================================== | 97% 755 MB
|=============================================================== | 97% 755 MB
|=============================================================== | 97% 755 MB
|=============================================================== | 97% 755 MB
|=============================================================== | 97% 755 MB
|=============================================================== | 97% 755 MB
|=============================================================== | 97% 755 MB
|=============================================================== | 97% 755 MB
|=============================================================== | 98% 755 MB
|=============================================================== | 98% 755 MB
|=============================================================== | 98% 755 MB
|=============================================================== | 98% 755 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 756 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 757 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 758 MB
|=============================================================== | 98% 759 MB
|=============================================================== | 98% 759 MB
|=============================================================== | 98% 759 MB
|================================================================| 98% 759 MB
|================================================================| 98% 759 MB
|================================================================| 98% 759 MB
|================================================================| 98% 759 MB
|================================================================| 98% 759 MB
|================================================================| 98% 759 MB
|================================================================| 98% 759 MB
|================================================================| 98% 759 MB
|================================================================| 98% 759 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 760 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 761 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 762 MB
|================================================================| 98% 763 MB
|================================================================| 98% 763 MB
|================================================================| 98% 763 MB
|================================================================| 98% 763 MB
|================================================================| 98% 763 MB
|================================================================| 99% 763 MB
|================================================================| 99% 763 MB
|================================================================| 99% 763 MB
|================================================================| 99% 763 MB
|================================================================| 99% 763 MB
|================================================================| 99% 763 MB
|================================================================| 99% 763 MB
|================================================================| 99% 763 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 764 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 765 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 766 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 767 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 768 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 769 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 770 MB
|================================================================| 99% 771 MB
|=================================================================| 100% 771 MB
-
-#+begin_src R :results output :session :exports both
-df4.mpi %>%
- filter(Iteration == 3) %>%
- filter(Value != "timste") %>%
- filter(grepl("MPI_", Value)) %>%
- group_by(Value) %>%
- summarize(Mean = sum(End-Start), N=n())
-#+end_src
-
-#+RESULTS:
-#+begin_example
-# A tibble: 8 x 3
- Value Mean N
-
-1 MPI_Allreduce 356.860608 304794
-2 MPI_Barrier 60.943087 1575
-3 MPI_Comm_rank 0.000148 315
-4 MPI_Comm_size 0.000251 315
-5 MPI_Irecv 2.528487 1023948
-6 MPI_Isend 12.640539 1023948
-7 MPI_Sendrecv 329.724298 47880
-8 MPI_Waitall 46.962233 92814
-#+end_example
-
-***** Plot
-
-#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 1700 :height 800 :session
-df4.mpi %>%
- filter(Rank != 0) %>%
- filter(Iteration == 7) %>%
- filter(End-Start > 0.1) -> tx;
-tx %>%
- group_by(Iteration) %>%
- mutate(End = End - min(Start),
- Start = Start - min(Start)) %>%
- ungroup() %>%
- ggplot() +
- facet_wrap(~Iteration) +
- geom_rect(aes(fill=Value,
- xmin=Start,
- xmax=End,
- ymin=Rank,
- ymax=Rank+0.9));
-#+end_src
-
-#+RESULTS:
-[[file:/tmp/babel-24763lPR/figure24763btH.png]]
-
-#+begin_src R :results output :session :exports both
-tx %>% group_by(Rank, Value, Iteration) %>% summarize(N=n())
-#+end_src
-
-#+RESULTS:
-#+begin_example
-# A tibble: 1,407 x 4
-# Groups: Rank, Value [?]
- Rank Value Iteration N
-
- 1 0 solver 150 4
- 2 1 nsi_matrix 150 1
- 3 1 solver 150 4
- 4 2 nsi_matrix 150 1
- 5 2 solver 150 4
- 6 3 nsi_matrix 150 1
- 7 3 solver 150 4
- 8 4 nsi_matrix 150 1
- 9 4 solver 150 4
-10 5 nsi_matrix 150 1
-# ... with 1,397 more rows
-#+end_example
-
-***** Calculate Comm / Comp for this prep trace
-
-#+begin_src R :results output :session :exports both
-df4.mpi %>%
- group_by(Rank, Iteration, Platform, Nodes, NP, Partitioning, EID) %>%
- filter(grepl("MPI_", Value)) %>%
- summarize(N=n(), S=min(Start), E=max(End), Comm=sum(End-Start), Comp=(E-S)-Comm) -> df4.sum;
-#+end_src
-
-#+RESULTS:
-
-#+begin_src R :results output :session :exports both
-df4.sum
-#+end_src
-
-#+RESULTS:
-#+begin_example
-# A tibble: 630 x 12
-# Groups: Rank, Iteration, Platform, Nodes, NP, Partitioning [?]
- Rank Iteration Platform Nodes NP Partitioning EID N S
-
- 1 0 1 grisou 4 63 sfc prep-4 5454 129.7846
- 2 0 2 grisou 4 63 sfc prep-4 4752 174.9951
- 3 0 3 grisou 4 63 sfc prep-4 4873 219.0275
- 4 0 4 grisou 4 63 sfc prep-4 4697 263.0684
- 5 0 5 grisou 4 63 sfc prep-4 4660 306.4143
- 6 0 6 grisou 4 63 sfc prep-4 4544 349.4879
- 7 0 7 grisou 4 63 sfc prep-4 4448 391.9349
- 8 0 8 grisou 4 63 sfc prep-4 4324 433.9185
- 9 0 9 grisou 4 63 sfc prep-4 4263 475.6839
-10 0 10 grisou 4 63 sfc prep-4 3579 517.5147
-# ... with 620 more rows, and 3 more variables: E , Comm , Comp
-#+end_example
-
-#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 500 :height 400 :session
-library(ggplot2);
-library(tidyr);
-df4.sum %>%
- filter(Rank != 0) %>%
- mutate(Total = Comm+Comp) %>%
- select(-N, -S, -E) %>%
- gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID) %>%
- ggplot(aes(x=Iteration, y=Value, group=Iteration)) +
- theme_bw(base_size=12) +
- ylim(0,NA) +
- geom_boxplot() +
- theme(legend.position="top") +
- facet_wrap(~Variable)
-#+end_src
-
-#+RESULTS:
-[[file:/tmp/babel-24763lPR/figure24763CTO.png]]
-
-** 50-node grisou :EXP5:
-** 7-node grimoire with *Infiniband* :deprecated:EXP6:
-*** First attempt
-Experiment did not work because Infiniband was not chosen by MPI.
-The message tells me this:
-#+BEGIN_EXAMPLE
-No OpenFabrics connection schemes reported that they were able to be
-used on a specific port. As such, the openib BTL (OpenFabrics
-support) will be disabled for this port.
-
- Local host: grimoire-4
- Local device: mlx4_0
- Local port: 1
- CPCs attempted: udcm
-#+END_EXAMPLE
-*** Second attempt
-JobId 1227284 @ nancy.
-**** Convert to CSV
-I need on the $PATH:
-- akypuera's otf22paje (compiled using OTF2 libraries of ScoreP 3.0)
-- pajeng's =pj_dump=
-
-#+begin_src shell :results output
-export PATH=$PATH:/home/lschnorr/akypuera/b/
-export PATH=$PATH:/home/lschnorr/pajeng/b/
-
-convert() {
- pushd $(dirname $otf2)
- otf22paje traces.otf2 | pj_dump | grep ^State | cut -d, -f2,4,5,8 | sed -e "s/ //g" -e "s/MPIRank//" | gzip > traces.csv.gz
- popd
-}
-
-EDIR=exp_6_grimoire_7
-# Files already converted (whose CSV size is not zero)
-EXISTINGFILE=$(tempfile)
-OTF2FILE=$(tempfile)
-find $EDIR -not -empty | grep csv$ | sed -e "s/.csv.gz$//" | sort > $EXISTINGFILE
-find $EDIR | grep otf2 | sed -e "s/.otf2$//" | sort > $OTF2FILE
-
-for otf2 in $(comm -3 $OTF2FILE $EXISTINGFILE | sed "s/$/.otf2/"); do
- echo $otf2
- convert $otf2
-done
-#+end_src
-**** Post-processing in R
-#+begin_src R :results output :session :exports both :tangle do-exp6.R :tangle-mode (identity #o755)
-#!/usr/bin/Rscript
-library(readr);
-library(dplyr);
-alya_scorep_trace_read <- function(filename)
-{
- meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[3], "_"));
- read_csv(filename,
- col_names=c("Rank", "Start", "End", "Value"),
- progress=TRUE) %>%
- # Transform Value to factor
- mutate(Value = as.factor(Value)) %>%
- # Detect begin and end of iterations
- mutate(Iteration = case_when(
- grepl("timste", .$Value) ~ 1,
- grepl("endste", .$Value) ~ -1,
- TRUE ~ 0)) %>%
- group_by(Rank) %>%
- # Create a logical to detect observations within iterations
- mutate(Iteration = as.logical(cumsum(Iteration))) %>%
- # Get only observations that belongs to some iteration
- filter(Iteration == TRUE) %>%
- ungroup() %>%
- # Create the iteration by cumsum
- mutate(Iteration = case_when(
- grepl("timste", .$Value) ~ 1,
- TRUE ~ 0)) %>%
- group_by(Rank) %>%
- mutate(Iteration = as.integer(cumsum(Iteration))) %>%
- ungroup() %>%
- # Define metadata
- mutate(EID = meta[2],
- Platform = meta[3],
- Nodes = meta[4],
- NP = meta[5],
- Partitioning = meta[6]);
-}
-
-alya_scorep_trace_iterations <- function(filename)
-{
- alya_scorep_trace_read(filename) %>%
- group_by(Rank, Iteration, Platform, Nodes, NP, Partitioning, EID) %>%
- filter(grepl("MPI_", Value)) %>%
- summarize(N=n(), S=min(Start), E=max(End), Comm=sum(End-Start), Comp=(E-S)-Comm);
-}
-
-args = commandArgs(trailingOnly=TRUE)
-print(args);
-df <- do.call("rbind", lapply(args, function(x) { alya_scorep_trace_iterations(x) }));
-write.csv(df, "exp6_iterations.csv");
-#+end_src
-**** Read in R :ATTACH:
-:PROPERTIES:
-:Attachments: exp6_iterations.csv.gz
-:ID: 4e48b93e-3322-4cc1-9ba9-3145d1298878
-:END:
-
-***** Reading
-#+begin_src R :results output :session :exports both
-library(readr);
-library(dplyr);
-df <- read_csv("data/4e/48b93e-3322-4cc1-9ba9-3145d1298878/exp6_iterations.csv.gz") %>%
- select(-X1) %>%
- rename(Start = S, End = E) %>%
- mutate(Duration = End - Start) %>%
- mutate(Platform = as.factor(Platform),
- Partitioning = as.factor(Partitioning),
- Nodes = as.factor(Nodes));
-df %>% summary;
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Parsed with column specification:
-cols(
- X1 = col_integer(),
- Rank = col_integer(),
- Iteration = col_integer(),
- Platform = col_character(),
- Nodes = col_integer(),
- NP = col_integer(),
- Partitioning = col_character(),
- EID = col_integer(),
- N = col_integer(),
- S = col_double(),
- E = col_double(),
- Comm = col_double(),
- Comp = col_double()
-)
-Warning message:
-Missing column names filled in: 'X1' [1]
- Rank Iteration Platform Nodes NP
- Min. : 0.00 Min. : 1.0 grimoire:4180 7:4180 Min. : 97
- 1st Qu.: 26.00 1st Qu.: 3.0 1st Qu.: 97
- Median : 52.00 Median : 5.5 Median :112
- Mean : 52.02 Mean : 5.5 Mean :105
- 3rd Qu.: 78.00 3rd Qu.: 8.0 3rd Qu.:112
- Max. :111.00 Max. :10.0 Max. :112
- Partitioning EID N Start End
- metis:2090 Min. :6 Min. : 3579 Min. :122.2 Min. :149.6
- sfc :2090 1st Qu.:6 1st Qu.:16353 1st Qu.:176.8 1st Qu.:200.4
- Median :6 Median :23759 Median :243.1 Median :267.8
- Mean :6 Mean :27314 Mean :240.3 Mean :264.6
- 3rd Qu.:6 3rd Qu.:36328 3rd Qu.:295.3 3rd Qu.:319.3
- Max. :6 Max. :85081 Max. :372.1 Max. :392.9
- Comm Comp Duration
- Min. : 0.5519 Min. : 0.0178 Min. :18.05
- 1st Qu.: 8.4911 1st Qu.:12.9418 1st Qu.:22.99
- Median :10.0122 Median :14.4247 Median :24.01
- Mean : 9.7437 Mean :14.5349 Mean :24.28
- 3rd Qu.:11.0630 3rd Qu.:15.8340 3rd Qu.:26.13
- Max. :28.4181 Max. :27.2230 Max. :28.45
-#+end_example
-***** Plot
-#+begin_src R :results output :session :exports both
-df %>% select(-Start, -End, -N, -Duration) %>% mutate(Total = Comm+Comp)
-#+end_src
-
-#+RESULTS:
-#+begin_example
-# A tibble: 4,180 × 10
- Rank Iteration Platform Nodes NP Partitioning EID Comm Comp
-
-1 0 1 grimoire 7 97 metis 6 28.41812 0.028814
-2 0 2 grimoire 7 97 metis 6 27.58274 0.024577
-3 0 3 grimoire 7 97 metis 6 27.34581 0.024256
-4 0 4 grimoire 7 97 metis 6 27.04863 0.023692
-5 0 5 grimoire 7 97 metis 6 26.84089 0.023234
-6 0 6 grimoire 7 97 metis 6 26.69917 0.022638
-7 0 7 grimoire 7 97 metis 6 26.48485 0.022195
-8 0 8 grimoire 7 97 metis 6 26.27733 0.021815
-9 0 9 grimoire 7 97 metis 6 26.06916 0.021781
-10 0 10 grimoire 7 97 metis 6 21.13847 0.017997
-# ... with 4,170 more rows, and 1 more variables: Total
-#+end_example
-
-#+begin_src R :results output :session :exports both
-library(tidyr);
-df %>%
- select(-Start, -End, -N, -Duration) %>%
- mutate(Total = Comm+Comp) %>%
- gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID);
-#+end_src
-
-#+RESULTS:
-#+begin_example
-# A tibble: 12,540 × 9
- Rank Iteration Platform Nodes NP Partitioning EID Variable Value
-
-1 0 1 grimoire 7 97 metis 6 Comm 28.41812
-2 0 2 grimoire 7 97 metis 6 Comm 27.58274
-3 0 3 grimoire 7 97 metis 6 Comm 27.34581
-4 0 4 grimoire 7 97 metis 6 Comm 27.04863
-5 0 5 grimoire 7 97 metis 6 Comm 26.84089
-6 0 6 grimoire 7 97 metis 6 Comm 26.69917
-7 0 7 grimoire 7 97 metis 6 Comm 26.48485
-8 0 8 grimoire 7 97 metis 6 Comm 26.27733
-9 0 9 grimoire 7 97 metis 6 Comm 26.06916
-10 0 10 grimoire 7 97 metis 6 Comm 21.13847
-# ... with 12,530 more rows
-#+end_example
-
-#+begin_src R :results output :session :exports both
-df %>%
- select(-Start, -End, -N, -Duration) %>%
- mutate(Total = Comm+Comp) %>%
- gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID) %>%
- group_by (Iteration, Platform, Nodes, NP, Partitioning, EID, Variable) %>%
- summarize(Mean = mean(Value), SE = 3*sd(Value)/sqrt(n()), N=n()) %>%
- # to check if it worked
- filter(NP == 97, Partitioning == "metis")
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Source: local data frame [30 x 10]
-Groups: Iteration, Platform, Nodes, NP, Partitioning, EID [10]
-
- Iteration Platform Nodes NP Partitioning EID Variable Mean
-
-1 1 grimoire 7 97 metis 6 Comm 11.77403
-2 1 grimoire 7 97 metis 6 Comp 16.53532
-3 1 grimoire 7 97 metis 6 Total 28.30936
-4 2 grimoire 7 97 metis 6 Comm 11.32483
-5 2 grimoire 7 97 metis 6 Comp 16.14359
-6 2 grimoire 7 97 metis 6 Total 27.46842
-7 3 grimoire 7 97 metis 6 Comm 11.09919
-8 3 grimoire 7 97 metis 6 Comp 16.13198
-9 3 grimoire 7 97 metis 6 Total 27.23118
-10 4 grimoire 7 97 metis 6 Comm 11.01637
-# ... with 20 more rows, and 2 more variables: SE , N
-#+end_example
-
-
-
-#+begin_src R :results output graphics :file img/exp_6-97np-metis-sfc.pdf :exports both :width 6 :height 3 :session
-library(ggplot2);
-library(tidyr);
-df %>%
- select(-Start, -End, -N, -Duration) %>%
- mutate(Total = Comm+Comp, Iteration = as.integer(Iteration)) %>%
- gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID) %>%
- group_by (Iteration, Platform, Nodes, NP, Partitioning, EID, Variable) %>%
- summarize(Mean = mean(Value), SE = 3*sd(Value)/sqrt(n()), N=n()) %>%
- ggplot(aes(x=Iteration, y=Mean, ymin=Mean-SE, ymax=Mean+SE,
- color=Partitioning)) +
- theme_bw(base_size=14) +
- ylim(0,NA) +
- xlim(0,10) +
- scale_x_continuous(breaks=seq(1:10)) +
- xlab("Iteration [count]") +
- ylab("Time [seconds]") +
- geom_point() +
- geom_errorbar(width=.3) +
- geom_line () +
- theme(legend.position=c(.78,.7),
- panel.margin = unit(.08, "lines"),
- legend.spacing = unit(.0, "lines"),
- legend.margin=margin(b = -.4, unit='cm'),
- plot.margin = unit(x = c(0, 0, 0, 0), units = "mm")
- ) +
- facet_grid(NP~Variable)
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_6-97np-metis-sfc.pdf]]
-***** With Arnaud
-#+begin_src R :results output graphics :file img/exp_6-97np_112np-metis-sfc-LB.pdf :exports both :width 6 :height 3 :session
-library(tidyr);
-library(ggplot2);
-df %>%
- select(-Start, -End, -N, -Duration) %>%
- mutate(Total = Comm+Comp) %>%
- gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID) %>%
- filter(Variable == "Comp", Rank != 0, Iteration != 11) %>%
- ggplot(aes(x=as.factor(Iteration), y=Value, color=as.factor(Rank))) +
- theme_bw(base_size=14) +
- ylim(0,30) +
- ylab("Computation Time [seconds]") +
- xlab("Iteration [count]") +
-# ggtitle("Each color is an MPI rank.") +
- geom_point(alpha=.3) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning))) +
- theme(legend.position="none",
- panel.margin = unit(.08, "lines"),
- legend.spacing = unit(.0, "lines"),
- legend.margin=margin(b = -.4, unit='cm'),
- plot.margin = unit(x = c(0, 0, 0, 0), units = "mm")
- ) +
- facet_grid(NP~Partitioning)
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_6-97np_112np-metis-sfc-LB.pdf]]
-
-#+begin_src R :results output graphics :file (org-babel-temp-file "figure" ".png") :exports both :width 600 :height 400 :session
-df %>%
- select(-Start, -End, -N, -Duration) %>%
- mutate(Total = Comm+Comp) %>%
- gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID) %>%
- filter(Variable == "Comp", Rank != 0) %>%
- filter(Iteration == 10) %>%
- ggplot(aes(x=Value, fill=Partitioning)) + theme_bw(base_size=12) + geom_histogram() + facet_grid(Partitioning~NP);
-#+end_src
-
-#+RESULTS:
-[[file:/tmp/babel-1875bQK/figure1875FAD.png]]
-
-
-Status:
-- Load balance among processes is not equal (considering only computation)
- - Iteration time is dominated by the slowest rank (numbers)
- - Such ranks also have the higher number of points as shown in
- fensap.par.log so we could have guessed in advance that the load
- balance would be a problem.
- - Actually, the load balancing is bad for both SFC and METIS
- although it is slightly better (the max load is lower) for SFC.
- - Communications are, as expected, more important for SFC than for
- METIS but when the communication network is good (e.g., ib) this
- becomes a problem only at large scale (probably above 512
- nodes as Ricard pointed out). However, when using a commodity
- network (e.g, eth0), communications are problematic at smaller
- scale (around above 100). Therefore, no need to go at high scale
- to study the load balancing problem. It is already very bad at
- medium scale.
-Questions:
-- Do you know how to control the partitioning (of Metis and SFC),
- e.g., to disable communication cost, so we have a "perfect" load
- balance.
-- Do you know how to get the mapping between the average of mesh
- points coordinates and ranks? This could help us understanding why
- Metis and SFC have difficulties computing a good load balance. A
- question we are asking ourselves is whether the heavy loaded
- processes in both METIS and SFC correspond to the same region of the
- mesh or not. (ref the fig)
-** 2-node and 4-node grimoire with Infiniband :EXP8:
-*** v1 :ATTACH:
-:PROPERTIES:
-:Attachments: exp_8-v1_grimoire_2.csv.gz
-:ID: faffa111-90b6-4e60-97f0-39475468994c
-:END:
-
-#+begin_src R :results output :session :exports both
-library(readr);
-library(dplyr);
-df <- read_csv("data/fa/ffa111-90b6-4e60-97f0-39475468994c/exp_8-v1_grimoire_2.csv.gz") %>%
- select(-X1) %>%
- rename(Start = S, End = E) %>%
- mutate(Duration = End - Start) %>%
- mutate(Platform = as.factor(Platform),
- Partitioning = as.factor(Partitioning),
- Nodes = as.factor(Nodes));
-df %>% summary;
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Parsed with column specification:
-cols(
- X1 = col_integer(),
- Rank = col_integer(),
- Iteration = col_integer(),
- Platform = col_character(),
- Nodes = col_integer(),
- NP = col_integer(),
- Partitioning = col_character(),
- EID = col_character(),
- Infiniband = col_logical(),
- N = col_integer(),
- S = col_double(),
- E = col_double(),
- Comm = col_double(),
- Comp = col_double()
-)
-Warning message:
-Missing column names filled in: 'X1' [1]
- Rank Iteration Platform Nodes NP
- Min. : 0.00 Min. : 1.0 grimoire:640 2:640 Min. :32
- 1st Qu.: 7.75 1st Qu.: 3.0 1st Qu.:32
- Median :15.50 Median : 5.5 Median :32
- Mean :15.50 Mean : 5.5 Mean :32
- 3rd Qu.:23.25 3rd Qu.: 8.0 3rd Qu.:32
- Max. :31.00 Max. :10.0 Max. :32
- Partitioning EID Infiniband N
- metis:640 Length:640 Mode :logical Min. : 3579
- Class :character FALSE:320 1st Qu.:11028
- Mode :character TRUE :320 Median :13216
- NA's :0 Mean :13343
- 3rd Qu.:15539
- Max. :22601
- Start End Comm Comp
- Min. :127.2 Min. :213.0 Min. : 0.2747 Min. : 0.02016
- 1st Qu.:298.2 1st Qu.:382.1 1st Qu.:28.2750 1st Qu.:44.78627
- Median :507.0 Median :588.7 Median :33.1652 Median :47.69400
- Mean :504.3 Mean :585.0 Mean :33.0316 Mean :47.60811
- 3rd Qu.:711.2 3rd Qu.:791.1 3rd Qu.:36.7584 3rd Qu.:52.60334
- Max. :876.5 Max. :941.3 Max. :86.2371 Max. :84.82805
- Duration
- Min. :64.25
- 1st Qu.:80.60
- Median :82.08
- Mean :80.64
- 3rd Qu.:83.44
- Max. :86.27
-#+end_example
-
-#+begin_src R :results output graphics :file img/exp_8-v1_total.png :exports both :width 500 :height 400 :session
-library(tidyr);
-library(ggplot2);
-df %>%
- select(-Start, -End, -N, -Duration) %>%
- mutate(Total = Comm+Comp) %>%
- gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID, -Infiniband) %>%
- filter(Variable == "Total", Rank != 0, Partitioning == "metis", NP == 32) %>%
- ggplot(aes(x=as.factor(Iteration), y=Value, color=Infiniband)) +
- theme_bw(base_size=12) +
- geom_point(alpha=.3) +
- ylim(0,NA) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
- theme(legend.position="top") +
- facet_grid(Partitioning~NP)
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_8-v1_total.png]]
-
-They look identical. I think both used Infiniband.
-
-I'll adapt the run alya script, run again to get v2 below.
-*** v2 :ATTACH:
-:PROPERTIES:
-:Attachments: exp_8-v2_grimoire_2.csv.gz
-:ID: 6475f529-6f29-4c6a-8d25-49c879403b6b
-:END:
-
-#+begin_src R :results output :session :exports both
-library(readr);
-library(dplyr);
-df <- read_csv("data/64/75f529-6f29-4c6a-8d25-49c879403b6b/exp_8-v2_grimoire_2.csv.gz") %>%
- select(-X1) %>%
- rename(Start = S, End = E) %>%
- mutate(Duration = End - Start) %>%
- mutate(Platform = as.factor(Platform),
- Partitioning = as.factor(Partitioning),
- Nodes = as.factor(Nodes));
-df %>% summary;
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Parsed with column specification:
-cols(
- X1 = col_integer(),
- Rank = col_integer(),
- Iteration = col_integer(),
- Platform = col_character(),
- Nodes = col_integer(),
- NP = col_integer(),
- Partitioning = col_character(),
- EID = col_character(),
- Infiniband = col_logical(),
- N = col_integer(),
- S = col_double(),
- E = col_double(),
- Comm = col_double(),
- Comp = col_double()
-)
-Warning message:
-Missing column names filled in: 'X1' [1]
- Rank Iteration Platform Nodes NP
- Min. : 0.00 Min. : 1.0 grimoire:640 2:640 Min. :32
- 1st Qu.: 7.75 1st Qu.: 3.0 1st Qu.:32
- Median :15.50 Median : 5.5 Median :32
- Mean :15.50 Mean : 5.5 Mean :32
- 3rd Qu.:23.25 3rd Qu.: 8.0 3rd Qu.:32
- Max. :31.00 Max. :10.0 Max. :32
- Partitioning EID Infiniband N
- metis:640 Length:640 Mode :logical Min. : 3579
- Class :character FALSE:320 1st Qu.:11028
- Mode :character TRUE :320 Median :13216
- NA's :0 Mean :13343
- 3rd Qu.:15539
- Max. :22601
- Start End Comm Comp
- Min. :126.5 Min. :213.5 Min. : 0.3249 Min. : 0.02036
- 1st Qu.:298.8 1st Qu.:383.4 1st Qu.:28.6923 1st Qu.:44.80856
- Median :507.5 Median :589.7 Median :33.4850 Median :47.64884
- Mean :505.2 Mean :586.0 Mean :33.2070 Mean :47.57824
- 3rd Qu.:710.9 3rd Qu.:790.4 3rd Qu.:36.8898 3rd Qu.:52.68086
- Max. :882.3 Max. :946.9 Max. :86.9266 Max. :85.24896
- Duration
- Min. :63.92
- 1st Qu.:80.31
- Median :82.00
- Mean :80.79
- 3rd Qu.:83.64
- Max. :86.96
-#+end_example
-
-#+begin_src R :results output graphics :file img/exp_8-v2_total.png :exports both :width 500 :height 400 :session
-library(tidyr);
-library(ggplot2);
-df %>%
- select(-Start, -End, -N, -Duration) %>%
- mutate(Total = Comm+Comp) %>%
- gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID, -Infiniband) %>%
- filter(Variable == "Total", Iteration != 10, Rank != 0, Partitioning == "metis", NP == 32) %>%
- ggplot(aes(x=as.factor(Iteration), y=Value, color=as.factor(Infiniband))) +
- theme_bw(base_size=12) +
- geom_point(alpha=.3) +
- ylim(0,NA) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
- theme(legend.position="top") +
- facet_grid(Partitioning~NP)
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_8-v2_total.png]]
-
-Well, it it still very similar. But now at least when Infiniband is
-TRUE, the total makespan of each iteration is smaller.
-
-*** v3 :ATTACH:
-:PROPERTIES:
-:Attachments: exp_8-v3_grimoire_4.csv.gz
-:ID: bd355764-1704-4485-bebd-5e714e76c2be
-:END:
-
-#+begin_src R :results output :session :exports both
-library(readr);
-library(dplyr);
-df <- read_csv("data/bd/355764-1704-4485-bebd-5e714e76c2be/exp_8-v3_grimoire_4.csv.gz") %>%
- select(-X1) %>%
- rename(Start = S, End = E) %>%
- mutate(Duration = End - Start) %>%
- mutate(Platform = as.factor(Platform),
- Partitioning = as.factor(Partitioning),
- Nodes = as.factor(Nodes));
-df %>% summary;
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Parsed with column specification:
-cols(
- X1 = col_integer(),
- Rank = col_integer(),
- Iteration = col_integer(),
- Platform = col_character(),
- Nodes = col_integer(),
- NP = col_integer(),
- Partitioning = col_character(),
- EID = col_character(),
- Infiniband = col_logical(),
- N = col_integer(),
- S = col_double(),
- E = col_double(),
- Comm = col_double(),
- Comp = col_double()
-)
-Warning message:
-Missing column names filled in: 'X1' [1]
- Rank Iteration Platform Nodes NP
- Min. : 0.00 Min. : 1.0 grimoire:1280 4:1280 Min. :64
- 1st Qu.:15.75 1st Qu.: 3.0 1st Qu.:64
- Median :31.50 Median : 5.5 Median :64
- Mean :31.50 Mean : 5.5 Mean :64
- 3rd Qu.:47.25 3rd Qu.: 8.0 3rd Qu.:64
- Max. :63.00 Max. :10.0 Max. :64
- Partitioning EID Infiniband N
- metis:1280 Length:1280 Mode :logical Min. : 3579
- Class :character FALSE:640 1st Qu.:11750
- Mode :character TRUE :640 Median :14600
- NA's :0 Mean :15362
- 3rd Qu.:18626
- Max. :28849
- Start End Comm Comp
- Min. :261.0 Min. :302.7 Min. : 0.3612 Min. : 0.03617
- 1st Qu.:343.8 1st Qu.:384.1 1st Qu.:13.3089 1st Qu.:21.57136
- Median :443.8 Median :483.0 Median :15.2729 Median :23.21151
- Mean :442.5 Mean :481.2 Mean :15.2131 Mean :23.46578
- 3rd Qu.:541.2 3rd Qu.:579.3 3rd Qu.:17.4194 3rd Qu.:25.69657
- Max. :621.9 Max. :653.0 Max. :41.7033 Max. :40.66069
- Duration
- Min. :30.86
- 1st Qu.:38.42
- Median :39.29
- Mean :38.68
- 3rd Qu.:39.86
- Max. :41.76
-#+end_example
-
-#+begin_src R :results output graphics :file img/exp_8-v3_total.png :exports both :width 500 :height 400 :session
-library(tidyr);
-library(ggplot2);
-df %>%
- select(-Start, -End, -N, -Duration) %>%
- mutate(Total = Comm+Comp) %>%
- gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID, -Infiniband) %>%
- filter(Variable == "Total", Iteration != 10, Rank != 0, Partitioning == "metis", NP == 64) %>%
- ggplot(aes(x=as.factor(Iteration), y=Value, color=Infiniband)) +
- theme_bw(base_size=12) +
- geom_point(alpha=.3) +
- ylim(0,NA) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
- theme(legend.position="top") +
- facet_grid(Partitioning~NP)
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_8-v3_total.png]]
-
-Same as in v3. I think Infiniband set to TRUE is already working.
-
-*** v4 :ATTACH:
-:PROPERTIES:
-:Attachments: exp_8-v4_grimoire_4.csv.gz
-:ID: 96a5786d-68c5-4d6c-9a54-6947b6fd5021
-:END:
-#+begin_src R :results output :session :exports both
-library(readr);
-library(dplyr);
-df <- read_csv("data/96/a5786d-68c5-4d6c-9a54-6947b6fd5021/exp_8-v4_grimoire_4.csv.gz") %>%
- select(-X1) %>%
- rename(Start = S, End = E) %>%
- mutate(Duration = End - Start) %>%
- mutate(Platform = as.factor(Platform),
- Partitioning = as.factor(Partitioning),
- Nodes = as.factor(Nodes));
-df %>% summary;
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Parsed with column specification:
-cols(
- X1 = col_integer(),
- Rank = col_integer(),
- Iteration = col_integer(),
- Platform = col_character(),
- Nodes = col_integer(),
- NP = col_integer(),
- Partitioning = col_character(),
- EID = col_character(),
- Infiniband = col_logical(),
- N = col_integer(),
- S = col_double(),
- E = col_double(),
- Comm = col_double(),
- Comp = col_double()
-)
-Warning message:
-Missing column names filled in: 'X1' [1]
- Rank Iteration Platform Nodes NP
- Min. : 0.00 Min. : 1.0 grimoire:2400 4:2400 Min. :60
- 1st Qu.:14.75 1st Qu.: 3.0 1st Qu.:60
- Median :29.50 Median : 5.5 Median :60
- Mean :29.50 Mean : 5.5 Mean :60
- 3rd Qu.:44.25 3rd Qu.: 8.0 3rd Qu.:60
- Max. :59.00 Max. :10.0 Max. :60
- Partitioning EID Infiniband N
- metis:1200 Length:2400 Mode :logical Min. : 3579
- sfc :1200 Class :character FALSE:1200 1st Qu.:13229
- Mode :character TRUE :1200 Median :22332
- NA's :0 Mean :25072
- 3rd Qu.:33946
- Max. :69461
- Start End Comm Comp
- Min. :238.8 Min. :278.3 Min. : 0.5323 Min. : 0.03779
- 1st Qu.:339.3 1st Qu.:380.7 1st Qu.:10.4865 1st Qu.:23.37386
- Median :431.8 Median :471.4 Median :13.5453 Median :25.22149
- Mean :431.4 Mean :470.1 Mean :13.5252 Mean :25.13902
- 3rd Qu.:523.3 3rd Qu.:562.9 3rd Qu.:16.4835 3rd Qu.:27.24979
- Max. :643.5 Max. :676.9 Max. :44.0683 Max. :42.73652
- Duration
- Min. :28.50
- 1st Qu.:36.52
- Median :38.28
- Mean :38.66
- 3rd Qu.:41.78
- Max. :44.13
-#+end_example
-
-#+begin_src R :results output graphics :file img/exp_8-v4_total.png :exports both :width 500 :height 400 :session
-library(tidyr);
-library(ggplot2);
-df %>%
- select(-Start, -End, -N, -Duration) %>%
- mutate(Total = Comm+Comp) %>%
- gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID, -Infiniband) %>%
- filter(Variable == "Total", Iteration != 10, Rank != 0, NP == 60) %>%
- ggplot(aes(x=as.factor(Iteration), y=Value, color=Infiniband)) +
- theme_bw(base_size=12) +
- geom_point(alpha=.3) +
- ylim(0,NA) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
- theme(legend.position="top") +
- facet_grid(NP~Partitioning)
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_8-v4_total.png]]
-
-Looks like we have proof that Infiniband can be desactivated.
-** 6-node grimoire infiniband check :EXP9:ATTACH:
-:PROPERTIES:
-:Attachments: exp_9-v1_grimoire_6.csv.gz
-:ID: e71e949a-1221-4b20-90b5-f8b9c6585810
-:END:
-
-#+begin_src R :results output :session :exports both
-library(readr);
-library(dplyr);
-df <- read_csv("data/e7/1e949a-1221-4b20-90b5-f8b9c6585810/exp_9-v1_grimoire_6.csv.gz") %>%
- select(-X1) %>%
- rename(Start = S, End = E) %>%
- mutate(Duration = End - Start) %>%
- mutate(Platform = as.factor(Platform),
- Partitioning = as.factor(Partitioning),
- Nodes = as.factor(Nodes));
-df %>% summary;
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Parsed with column specification:
-cols(
- X1 = col_integer(),
- Rank = col_integer(),
- Iteration = col_integer(),
- Platform = col_character(),
- Nodes = col_integer(),
- NP = col_integer(),
- Partitioning = col_character(),
- EID = col_character(),
- Infiniband = col_logical(),
- N = col_integer(),
- S = col_double(),
- E = col_double(),
- Comm = col_double(),
- Comp = col_double()
-)
-Warning message:
-Missing column names filled in: 'X1' [1]
- Rank Iteration Platform Nodes NP
- Min. : 0.00 Min. : 1.0 grimoire:3840 6:3840 Min. :96
- 1st Qu.:23.75 1st Qu.: 3.0 1st Qu.:96
- Median :47.50 Median : 5.5 Median :96
- Mean :47.50 Mean : 5.5 Mean :96
- 3rd Qu.:71.25 3rd Qu.: 8.0 3rd Qu.:96
- Max. :95.00 Max. :10.0 Max. :96
- Partitioning EID Infiniband N
- metis:1920 Length:3840 Mode :logical Min. : 3579
- sfc :1920 Class :character FALSE:1920 1st Qu.:16320
- Mode :character TRUE :1920 Median :23759
- NA's :0 Mean :27008
- 3rd Qu.:36328
- Max. :81957
- Start End Comm Comp
- Min. :123.4 Min. :150.7 Min. : 0.3444 Min. : 0.01754
- 1st Qu.:190.8 1st Qu.:217.7 1st Qu.: 8.4054 1st Qu.:14.47885
- Median :251.6 Median :277.3 Median : 9.7694 Median :15.58878
- Mean :251.8 Mean :277.3 Mean : 9.6905 Mean :15.79769
- 3rd Qu.:312.4 3rd Qu.:337.4 3rd Qu.:11.4077 3rd Qu.:17.16624
- Max. :384.8 Max. :405.8 Max. :27.9786 Max. :27.03608
- Duration
- Min. :19.22
- 1st Qu.:24.94
- Median :26.06
- Mean :25.49
- 3rd Qu.:26.76
- Max. :28.22
-#+end_example
-
-#+begin_src R :results output graphics :file img/exp_9-v1_total.png :exports both :width 500 :height 400 :session
-library(tidyr);
-library(ggplot2);
-df %>%
- select(-Start, -End, -N, -Duration) %>%
- mutate(Total = Comm+Comp) %>%
- gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID, -Infiniband) %>%
- filter(Variable == "Total", Iteration != 10, Rank != 0) %>%
- ggplot(aes(x=as.factor(Iteration), y=Value, color=Infiniband)) +
- theme_bw(base_size=12) +
- geom_point(alpha=.3) +
- ylim(0,NA) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
- theme(legend.position="top") +
- facet_grid(NP~Partitioning)
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_9-v1_total.png]]
-** 8-node grimoire infiniband-only after Alya modifications :EXP10:
-*** Read :ATTACH:
-:PROPERTIES:
-:Attachments: exp_10-v2_grimoire_8.csv.gz
-:ID: 15c9204a-2bde-4ee3-b166-242729fbc30d
-:END:
-
-#+begin_src R :results output :session :exports both
-library(readr);
-library(dplyr);
-df <- read_csv("data/15/c9204a-2bde-4ee3-b166-242729fbc30d/exp_10-v2_grimoire_8.csv.gz") %>%
- select(-X1) %>%
- rename(Start = S, End = E) %>%
- mutate(Duration = End - Start) %>%
- mutate(Platform = as.factor(Platform),
- Partitioning = as.factor(Partitioning),
- Nodes = as.factor(Nodes));
-df %>% summary;
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Parsed with column specification:
-cols(
- X1 = col_integer(),
- Rank = col_integer(),
- Iteration = col_integer(),
- Platform = col_character(),
- Nodes = col_integer(),
- NP = col_integer(),
- Partitioning = col_character(),
- EID = col_character(),
- Infiniband = col_logical(),
- N = col_integer(),
- S = col_double(),
- E = col_double(),
- Comm = col_double(),
- Comp = col_double()
-)
-Warning message:
-Missing column names filled in: 'X1' [1]
- Rank Iteration Platform Nodes NP
- Min. : 0.00 Min. : 1.0 grimoire:2560 8:2560 Min. :128
- 1st Qu.: 31.75 1st Qu.: 3.0 1st Qu.:128
- Median : 63.50 Median : 5.5 Median :128
- Mean : 63.50 Mean : 5.5 Mean :128
- 3rd Qu.: 95.25 3rd Qu.: 8.0 3rd Qu.:128
- Max. :127.00 Max. :10.0 Max. :128
- Partitioning EID Infiniband N Start
- metis:1280 Length:2560 Mode:logical Min. : 3579 Min. :121.9
- sfc :1280 Class :character TRUE:2560 1st Qu.:17937 1st Qu.:167.1
- Mode :character NA's:0 Median :25074 Median :210.0
- Mean :28662 Mean :210.3
- 3rd Qu.:37422 3rd Qu.:254.4
- Max. :91329 Max. :313.9
- End Comm Comp Duration
- Min. :139.5 Min. : 0.4235 Min. : 0.01789 Min. :13.30
- 1st Qu.:186.4 1st Qu.: 3.8991 1st Qu.:10.76531 1st Qu.:16.52
- Median :228.6 Median : 6.6020 Median :11.80869 Median :17.29
- Mean :228.5 Mean : 6.3043 Mean :11.90795 Mean :18.21
- 3rd Qu.:272.9 3rd Qu.: 8.5957 3rd Qu.:13.19428 3rd Qu.:20.24
- Max. :330.2 Max. :21.5603 Max. :20.72990 Max. :21.59
-#+end_example
-*** Plot Comp
-#+begin_src R :results output graphics :file img/exp_10-v2_comp.png :exports both :width 500 :height 400 :session
-library(tidyr);
-library(ggplot2);
-df %>%
- select(-Start, -End, -N, -Duration) %>%
- mutate(Total = Comm+Comp) %>%
- gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID, -Infiniband) %>%
- filter(Variable == "Comp", Iteration != 10, Rank != 0) %>%
- ggplot(aes(x=as.factor(Iteration), y=Value, color=Infiniband)) +
- theme_bw(base_size=12) +
- geom_point(alpha=.3) +
- ylim(0,NA) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
- theme(legend.position="top") +
- facet_grid(NP~Partitioning)
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_10-v2_comp.png]]
-*** Plot mean comp (per rank, all iterations)
-#+begin_src R :results output graphics :file img/exp_10-v2_comp_mean.png :exports both :width 600 :height 400 :session
-library(tidyr);
-library(ggplot2);
-df %>%
- select(-Start, -End, -N, -Duration) %>%
- mutate(Total = Comm+Comp) %>%
- gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID, -Infiniband) %>%
- filter(Variable == "Comp", Iteration != 10, Rank != 0) %>%
-# group_by(Rank, Platform, Nodes, NP, Partitioning, EID, Infiniband) %>%
-# summarize(MeanComp = mean(Value), SE=3*sd(Value)/sqrt(n()), N=n()) %>%
- ggplot(aes(x=Rank, y=Value, color=Partitioning)) +
- theme_bw(base_size=12) +
- geom_point(alpha=.3) +
- ylim(0,NA) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
- theme(legend.position="top") +
- facet_grid(NP~Partitioning)
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_10-v2_comp_mean.png]]
-*** Partition analysis
-**** Process log files
-#+begin_src shell :results output
-EDIR=exp_10-v2_grimoire_8
-for file in $(find $EDIR | grep results | grep log$); do
- OUTPUT=$(dirname $file)/$(basename $file .log).csv
- cat $file | tr -s " " | sed "s/^ //" | tr ' ' ',' | cut -d"," -f1,3,5,7,9,11,13 | uniq > $OUTPUT
- cat $file | tr -s " " | sed "s/^ //" | tr ' ' ',' | cut -d"," -f2,4,6,8,10,12,14 >> $OUTPUT
- head $OUTPUT
- tail $OUTPUT
- echo
-done
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Rank,NPOIN,NELEM,NELEW,NBOUN,NNEIG,NBBOU
-1,14976,72536,72796,2384,17,14976
-2,14641,72392,72575,448,12,14641
-3,22264,72647,167221,4446,16,22264
-4,28129,72464,251684,7187,11,28129
-5,23239,72488,178813,4341,25,23239
-6,26093,72525,209615,5640,14,26093
-7,22996,72391,183401,4422,10,22996
-8,21190,72393,160643,3505,7,21190
-9,25178,72612,201462,5195,10,25178
-118,24285,72483,188018,6181,14,24285
-119,26512,72604,229879,6447,15,26512
-120,24865,72374,200264,5134,10,24865
-121,22912,72446,173477,4266,15,22912
-122,14466,72460,73072,776,11,14466
-123,14046,72589,72589,944,8,14046
-124,15427,72509,73961,2155,15,15427
-125,21776,72610,153930,4907,13,21776
-126,21662,72595,161294,4243,12,21662
-127,18484,72603,111590,4500,17,18484
-
-Rank,NPOIN,NELEM,NELEW,NBOUN,NNEIG,NBBOU
-1,22405,85371,172276,3477,4,22405
-2,22923,86049,179769,3742,5,22923
-3,24316,99541,175416,3029,5,24316
-4,22907,90905,170125,3170,4,22907
-5,22636,86964,173194,3448,5,22636
-6,22774,88416,171711,3333,6,22774
-7,23564,91625,177865,3452,7,23564
-8,22211,83394,173364,3612,5,22211
-9,23427,89980,179685,3588,4,23427
-118,19344,51553,177808,5044,6,19344
-119,19050,54006,174756,4829,3,19050
-120,18742,53558,169698,4652,6,18742
-121,19158,51974,179487,5103,4,19158
-122,19820,53397,179497,5028,8,19820
-123,19357,55432,172417,4679,6,19357
-124,18855,51717,176727,5006,3,18855
-125,19473,52093,179803,5107,6,19473
-126,18948,49074,179769,5225,4,18948
-127,19037,50759,172104,4863,6,19037
-
-#+end_example
-**** Load them in R
-#+begin_src R :results output :session :exports both
-library(readr);
-library(dplyr);
-library(tidyr);
-
-read_npoin <- function(filename)
-{
- meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[2], "_"));
- meta <- gsub(".dir", "", meta);
- read_csv(filename) %>%
- gather(Variable, Value, -Rank) %>%
- mutate(EID = meta[1],
- Platform = meta[2],
- Nodes = meta[3],
- NP = meta[4],
- Partitioning = meta[5],
- Infiniband = as.logical(meta[6]));
-
-}
-files <- list.files("exp_10-v2_grimoire_8", pattern="results_NPOIN_NELEM_NELEW_NBOUN.csv", recursive=TRUE, full.names=TRUE);
-files;
-df <- do.call("rbind", lapply(files, function(x) { read_npoin(x) }))
-df %>% filter(Rank == 111, Variable == "NELEW")
-#+end_src
-
-#+RESULTS:
-#+begin_example
-[1] "exp_10-v2_grimoire_8/10-v2_grimoire_8_128_metis_true.dir/results_NPOIN_NELEM_NELEW_NBOUN.csv"
-[2] "exp_10-v2_grimoire_8/10-v2_grimoire_8_128_sfc_true.dir/results_NPOIN_NELEM_NELEW_NBOUN.csv"
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- NPOIN = col_integer(),
- NELEM = col_integer(),
- NELEW = col_integer(),
- NBOUN = col_integer(),
- NNEIG = col_integer(),
- NBBOU = col_integer()
-)
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- NPOIN = col_integer(),
- NELEM = col_integer(),
- NELEW = col_integer(),
- NBOUN = col_integer(),
- NNEIG = col_integer(),
- NBBOU = col_integer()
-)
-# A tibble: 2 × 9
- Rank Variable Value EID Platform Nodes NP Partitioning Infiniband
-
-1 111 NELEW 172833 10-v2 grimoire 8 128 metis TRUE
-2 111 NELEW 258786 10-v2 grimoire 8 128 sfc TRUE
-#+end_example
-
-#+begin_src R :results output graphics :file img/exp_10-v2_partition.png :exports both :width 600 :height 800 :session
-df %>%
- ggplot(aes(x=Rank, y=Value, color=Partitioning)) +
- theme_bw(base_size=12) +
- geom_point() +
- ylim(0,NA) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
- theme(legend.position="top") +
- facet_grid(Variable~Partitioning, scales="free_y")
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_10-v2_partition.png]]
-** 8-node grimoire infiniband-only before Alya modifs :EXP11:
-*** Read :ATTACH:
-:PROPERTIES:
-:Attachments: exp_11-v1_grimoire_8.csv.gz
-:ID: 11326040-b8e2-41f3-8479-c378f70864d9
-:END:
-
-#+begin_src R :results output :session :exports both
-library(readr);
-library(dplyr);
-df <- read_csv("data/11/326040-b8e2-41f3-8479-c378f70864d9/exp_11-v1_grimoire_8.csv.gz") %>%
- select(-X1) %>%
- rename(Start = S, End = E) %>%
- mutate(Duration = End - Start) %>%
- mutate(Platform = as.factor(Platform),
- Partitioning = as.factor(Partitioning),
- Nodes = as.factor(Nodes));
-df %>% summary;
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Parsed with column specification:
-cols(
- X1 = col_integer(),
- Rank = col_integer(),
- Iteration = col_integer(),
- Platform = col_character(),
- Nodes = col_integer(),
- NP = col_integer(),
- Partitioning = col_character(),
- EID = col_character(),
- Infiniband = col_logical(),
- N = col_integer(),
- S = col_double(),
- E = col_double(),
- Comm = col_double(),
- Comp = col_double()
-)
-Warning message:
-Missing column names filled in: 'X1' [1]
- Rank Iteration Platform Nodes NP
- Min. : 0.00 Min. : 1.0 grimoire:2560 8:2560 Min. :128
- 1st Qu.: 31.75 1st Qu.: 3.0 1st Qu.:128
- Median : 63.50 Median : 5.5 Median :128
- Mean : 63.50 Mean : 5.5 Mean :128
- 3rd Qu.: 95.25 3rd Qu.: 8.0 3rd Qu.:128
- Max. :127.00 Max. :10.0 Max. :128
- Partitioning EID Infiniband N Start
- metis:1280 Length:2560 Mode:logical Min. : 3579 Min. :121.7
- sfc :1280 Class :character TRUE:2560 1st Qu.:17937 1st Qu.:168.6
- Mode :character NA's:0 Median :24815 Median :217.9
- Mean :28185 Mean :217.4
- 3rd Qu.:37093 3rd Qu.:265.8
- Max. :81957 Max. :313.0
- End Comm Comp Duration
- Min. :142.9 Min. : 0.3322 Min. : 0.02086 Min. :15.47
- 1st Qu.:189.2 1st Qu.: 7.0911 1st Qu.:10.84112 1st Qu.:19.75
- Median :237.8 Median : 8.1973 Median :11.59094 Median :20.00
- Mean :237.1 Mean : 7.9373 Mean :11.81608 Mean :19.75
- 3rd Qu.:285.6 3rd Qu.: 9.0958 3rd Qu.:12.73405 3rd Qu.:20.39
- Max. :328.8 Max. :21.1947 Max. :20.53365 Max. :21.22
-#+end_example
-*** Plot Comp
-#+begin_src R :results output graphics :file img/exp_11-v1_comp.png :exports both :width 500 :height 400 :session
-library(tidyr);
-library(ggplot2);
-df %>%
- select(-Start, -End, -N, -Duration) %>%
- mutate(Total = Comm+Comp) %>%
- gather(Variable, Value, -Rank, -Iteration, -Platform, -Nodes, -NP, -Partitioning, -EID, -Infiniband) %>%
- filter(Variable == "Comp", Iteration != 10, Rank != 0) %>%
- ggplot(aes(x=as.factor(Iteration), y=Value, color=Infiniband)) +
- theme_bw(base_size=12) +
- geom_point(alpha=.3) +
- ylim(0,NA) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
- theme(legend.position="top") +
- facet_grid(NP~Partitioning)
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_11-v1_comp.png]]
-*** Partition analysis
-**** Process log files
-#+begin_src shell :results output
-EDIR=exp_11-v1_grimoire_8
-for file in $(find $EDIR | grep results | grep log$); do
- OUTPUT=$(dirname $file)/$(basename $file .log).csv
- cat $file | tr -s " " | sed "s/^ //" | tr ' ' ',' | cut -d"," -f1,3,5,7,9,11,13 | uniq > $OUTPUT
- cat $file | tr -s " " | sed "s/^ //" | tr ' ' ',' | cut -d"," -f2,4,6,8,10,12,14 >> $OUTPUT
- head $OUTPUT
- tail $OUTPUT
- echo
-done
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Rank,NPOIN,NELEM,NELEW,NBOUN,NNEIG,NBBOU
-1,32182,174168,174611,2973,16,32182
-2,20919,53529,174758,5381,15,20919
-3,19727,49972,175172,5020,11,19727
-4,21918,69049,174054,4277,23,21918
-5,21683,64345,174280,4439,14,21683
-6,22205,64122,175002,4582,18,22205
-7,21361,63432,174587,4436,11,21361
-8,22162,75699,174459,3914,8,22162
-9,21389,66339,174799,4332,8,21389
-118,19736,47749,174249,5084,13,19736
-119,27122,103080,174535,7852,12,27122
-120,21785,59025,175145,5304,13,21785
-121,20722,56163,175018,4869,12,20722
-122,21269,59300,174615,4657,10,21269
-123,22403,70519,174270,4135,13,22403
-124,30100,145205,174682,2874,9,30100
-125,31603,142203,174120,5271,15,31603
-126,22843,73894,175386,4418,11,22843
-127,26719,97390,174439,6359,13,26719
-
-Rank,NPOIN,NELEM,NELEW,NBOUN,NNEIG,NBBOU
-1,22405,85371,172276,3477,4,22405
-2,22923,86049,179769,3742,5,22923
-3,24316,99541,175416,3029,5,24316
-4,22907,90905,170125,3170,4,22907
-5,22636,86964,173194,3448,5,22636
-6,22774,88416,171711,3333,6,22774
-7,23564,91625,177865,3452,7,23564
-8,22211,83394,173364,3612,5,22211
-9,23427,89980,179685,3588,4,23427
-118,19344,51553,177808,5044,6,19344
-119,19050,54006,174756,4829,3,19050
-120,18742,53558,169698,4652,6,18742
-121,19158,51974,179487,5103,4,19158
-122,19820,53397,179497,5028,8,19820
-123,19357,55432,172417,4679,6,19357
-124,18855,51717,176727,5006,3,18855
-125,19473,52093,179803,5107,6,19473
-126,18948,49074,179769,5225,4,18948
-127,19037,50759,172104,4863,6,19037
-
-#+end_example
-**** Load them in R
-#+begin_src R :results output :session :exports both
-library(readr);
-library(dplyr);
-library(tidyr);
-
-read_npoin <- function(filename)
-{
- meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[2], "_"));
- meta <- gsub(".dir", "", meta);
- read_csv(filename) %>%
- gather(Variable, Value, -Rank) %>%
- mutate(EID = meta[1],
- Platform = meta[2],
- Nodes = meta[3],
- NP = meta[4],
- Partitioning = meta[5],
- Infiniband = as.logical(meta[6]));
-
-}
-files <- list.files("exp_11-v1_grimoire_8", pattern="results_NPOIN_NELEM_NELEW_NBOUN.csv", recursive=TRUE, full.names=TRUE);
-files;
-df <- do.call("rbind", lapply(files, function(x) { read_npoin(x) }))
-df %>% filter(Rank == 111, Variable == "NELEW")
-#+end_src
-
-#+RESULTS:
-#+begin_example
-[1] "exp_11-v1_grimoire_8/11-v1_grimoire_8_128_metis_true.dir/results_NPOIN_NELEM_NELEW_NBOUN.csv"
-[2] "exp_11-v1_grimoire_8/11-v1_grimoire_8_128_sfc_true.dir/results_NPOIN_NELEM_NELEW_NBOUN.csv"
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- NPOIN = col_integer(),
- NELEM = col_integer(),
- NELEW = col_integer(),
- NBOUN = col_integer(),
- NNEIG = col_integer(),
- NBBOU = col_integer()
-)
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- NPOIN = col_integer(),
- NELEM = col_integer(),
- NELEW = col_integer(),
- NBOUN = col_integer(),
- NNEIG = col_integer(),
- NBBOU = col_integer()
-)
-# A tibble: 2 × 9
- Rank Variable Value EID Platform Nodes NP Partitioning Infiniband
-
-1 111 NELEW 172833 11-v1 grimoire 8 128 metis TRUE
-2 111 NELEW 173994 11-v1 grimoire 8 128 sfc TRUE
-#+end_example
-**** Plot
-#+begin_src R :results output graphics :file img/exp_11-v1_partition.png :exports both :width 600 :height 800 :session
-df %>%
- ggplot(aes(x=Rank, y=Value, color=Partitioning)) +
- theme_bw(base_size=12) +
- geom_point() +
- ylim(0,NA) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning):as.factor(Infiniband))) +
- theme(legend.position="top") +
- facet_grid(Variable~Partitioning, scales="free_y")
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_11-v1_partition.png]]
-** 6-node grimoire tracing only application functions :EXP12:
-:PROPERTIES:
-:Attachments: scorep_12-v1_grimoire_6_48_metis_true_traces.csv.gz scorep_12-v1_grimoire_6_48_sfc_true_traces.csv.gz scorep_12-v1_grimoire_6_96_metis_true_traces.csv.gz scorep_12-v1_grimoire_6_96_sfc_true_traces.csv.gz
-:ID: 58ddccd9-95c8-4841-9c07-336a0209e264
-:END:
-
-*** Which files
-#+name: exp_12_traces
-#+begin_src shell :results output org
-find data/ | grep scorep_12-v1
-#+end_src
-
-#+RESULTS: exp_12_traces
-#+BEGIN_SRC org
-data/58/ddccd9-95c8-4841-9c07-336a0209e264/scorep_12-v1_grimoire_6_96_metis_true_traces.csv.gz
-data/58/ddccd9-95c8-4841-9c07-336a0209e264/scorep_12-v1_grimoire_6_48_metis_true_traces.csv.gz
-data/58/ddccd9-95c8-4841-9c07-336a0209e264/scorep_12-v1_grimoire_6_96_sfc_true_traces.csv.gz
-data/58/ddccd9-95c8-4841-9c07-336a0209e264/scorep_12-v1_grimoire_6_48_sfc_true_traces.csv.gz
-#+END_SRC
-*** Read in R :ATTACH:
-
-- =nsi_matrix=
-- =solver=
-- =nsi_inisol= (hardly seen)
-
-#+begin_src R :results output :session :exports both :var files=exp_12_traces
-library(readr);
-library(dplyr);
-alya_scorep_trace_read <- function(filename)
-{
- meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[4], "_"));
- read_csv(filename,
- col_names=c("Rank", "Start", "End", "Value"),
- progress=FALSE) %>%
- # Transform Value to factor
- mutate(Value = as.factor(Value)) %>%
- # Detect begin and end of iterations
- mutate(Iteration = case_when(
- (.$Value == "timste") ~ 1,
- (.$Value == "endste") ~ -1,
- TRUE ~ 0)) %>%
- group_by(Rank) %>%
- # Create a logical to detect observations within iterations
- mutate(Iteration = as.logical(cumsum(Iteration))) %>%
- # Get only observations that belongs to some iteration
- filter(Iteration == TRUE) %>%
- ungroup() %>%
- # Create the iteration by cumsum
- mutate(Iteration = case_when(
- (.$Value == "timste") ~ 1,
- TRUE ~ 0)) %>%
- group_by(Rank) %>%
- mutate(Iteration = cumsum(Iteration)) %>%
- mutate(EID = meta[2],
- Platform = meta[3],
- Nodes = meta[4],
- NP = meta[5],
- Partitioning = meta[6]);
-}
-
-files <- strsplit(files, "\n")[[1]]
-df <- do.call("rbind", lapply(files, function(x) { alya_scorep_trace_read(x) }));
-
-#
-# Keep only states that are interesting
-#
-
-df <- df %>% filter(Value %in% c("nsi_matrix", "solver", "nsi_inisol"));
-#+end_src
-
-#+RESULTS:
-#+begin_example
-
-Attaching package: ‘dplyr’
-
-The following objects are masked from ‘package:stats’:
-
- filter, lag
-
-The following objects are masked from ‘package:base’:
-
- intersect, setdiff, setequal, union
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- Start = col_double(),
- End = col_double(),
- Value = col_character()
-)
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- Start = col_double(),
- End = col_double(),
- Value = col_character()
-)
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- Start = col_double(),
- End = col_double(),
- Value = col_character()
-)
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- Start = col_double(),
- End = col_double(),
- Value = col_character()
-)
-#+end_example
-
-#+begin_src R :results output :session :exports both
-df %>%
- group_by(Rank, Iteration, EID, Platform, Nodes, NP, Partitioning, Value) %>%
- summarize(NewComp = sum(End - Start));
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Source: local data frame [86,400 x 9]
-Groups: Rank, Iteration, EID, Platform, Nodes, NP, Partitioning [?]
-
- Rank Iteration EID Platform Nodes NP Partitioning Value NewComp
-
-1 0 1 12-v1 grimoire 6 48 metis nsi_inisol 0.000013
-2 0 1 12-v1 grimoire 6 48 metis nsi_matrix 0.000075
-3 0 1 12-v1 grimoire 6 48 metis solver 42.482770
-4 0 1 12-v1 grimoire 6 48 sfc nsi_inisol 0.000012
-5 0 1 12-v1 grimoire 6 48 sfc nsi_matrix 0.000079
-6 0 1 12-v1 grimoire 6 48 sfc solver 35.042236
-7 0 1 12-v1 grimoire 6 96 metis nsi_inisol 0.000014
-8 0 1 12-v1 grimoire 6 96 metis nsi_matrix 0.000076
-9 0 1 12-v1 grimoire 6 96 metis solver 21.981336
-10 0 1 12-v1 grimoire 6 96 sfc nsi_inisol 0.000012
-# ... with 86,390 more rows
-#+end_example
-*** Plots
-#+begin_src R :results output graphics :file img/alya_nastin_solver_3_funs.png :exports both :width 600 :height 400 :session
-library(ggplot2);
-df %>%
- group_by(Rank, Iteration, EID, Platform, Nodes, NP, Partitioning) %>%
- summarize(NewComp = sum(End - Start)) %>%
- ggplot(aes(x=Iteration, y=NewComp, color=as.factor(Rank))) +
- theme_bw(base_size=12) +
- geom_point(alpha=.3) +
- ylim(0,NA) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning))) +
- theme(legend.position="none") +
- facet_grid(NP~Partitioning);
-#+end_src
-
-#+RESULTS:
-[[file:img/alya_nastin_solver_3_funs.png]]
-
-#+begin_src R :results output graphics :file img/alya_nastin_solver_solver.png :exports both :width 600 :height 400 :session
-df %>%
- filter(Value == "solver") %>%
- group_by(Rank, Iteration, EID, Platform, Nodes, NP, Value, Partitioning) %>%
- summarize(NewComp = sum(End - Start)) %>%
- ggplot(aes(x=Iteration, y=NewComp, color=as.factor(Rank))) +
- theme_bw(base_size=12) +
- geom_point(alpha=.3) +
- ylim(0,NA) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning))) +
- theme(legend.position="none") +
- facet_grid(NP~Partitioning);
-#+end_src
-
-#+RESULTS:
-[[file:img/alya_nastin_solver_solver.png]]
-
-#+begin_src R :results output graphics :file img/alya_nastin_solver_nsi_matrix.png :exports both :width 600 :height 400 :session
-df %>%
- filter(Value == "nsi_matrix") %>%
- group_by(Rank, Iteration, EID, Platform, Nodes, NP, Value, Partitioning) %>%
- summarize(NewComp = sum(End - Start)) %>%
- ggplot(aes(x=Iteration, y=NewComp, color=as.factor(Rank))) +
- theme_bw(base_size=12) +
- geom_point(alpha=.3) +
- ylim(0,NA) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning))) +
- theme(legend.position="none") +
- facet_grid(NP~Partitioning);#, scales="free_y")
-#+end_src
-
-#+RESULTS:
-[[file:img/alya_nastin_solver_nsi_matrix.png]]
-
-
-#+begin_src R :results output graphics :file img/alya_nastin_solver_nsi_inisol.png :exports both :width 600 :height 400 :session
-df %>%
- filter(Value == "nsi_inisol") %>%
- group_by(Rank, Iteration, EID, Platform, Nodes, NP, Value, Partitioning) %>%
- summarize(NewComp = sum(End - Start)) %>%
- ggplot(aes(x=Iteration, y=NewComp, color=as.factor(Rank))) +
- theme_bw(base_size=12) +
- geom_point(alpha=.3) +
- ylim(0,NA) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning))) +
- theme(legend.position="none") +
- facet_grid(NP~Partitioning);#, scales="free_y")
-#+end_src
-
-#+RESULTS:
-[[file:img/alya_nastin_solver_nsi_inisol.png]]
-** 12-node grisou (computing instrumentations, Alya modifs yes/no) :EXP13:ATTACH:
-:PROPERTIES:
-:ID: 0717e0e2-99b8-44c7-acfc-4d4ad685a2c5
-:Attachments: exp_13-v1_grisou_12
-:END:
-
-*** Which files
-#+name: exp_13_traces
-#+begin_src shell :results output
-cd data/07/17e0e2-99b8-44c7-acfc-4d4ad685a2c5/
-find exp_13-v1_grisou_12/| grep csv.gz
-#+end_src
-
-#+RESULTS: exp_13_traces
-: exp_13-v1_grisou_12/13-v1_grisou_12_96_sfc_true_Alya.x.modif.dir/scorep_13-v1_grisou_12_96_sfc_true_Alya.x.modif/traces.csv.gz
-: exp_13-v1_grisou_12/13-v1_grisou_12_96_metis_true_Alya.x.modif.dir/scorep_13-v1_grisou_12_96_metis_true_Alya.x.modif/traces.csv.gz
-: exp_13-v1_grisou_12/13-v1_grisou_12_192_metis_true_Alya.x.dir/scorep_13-v1_grisou_12_192_metis_true_Alya.x/traces.csv.gz
-: exp_13-v1_grisou_12/13-v1_grisou_12_192_sfc_true_Alya.x.dir/scorep_13-v1_grisou_12_192_sfc_true_Alya.x/traces.csv.gz
-: exp_13-v1_grisou_12/13-v1_grisou_12_192_metis_true_Alya.x.modif.dir/scorep_13-v1_grisou_12_192_metis_true_Alya.x.modif/traces.csv.gz
-: exp_13-v1_grisou_12/13-v1_grisou_12_96_metis_true_Alya.x.dir/scorep_13-v1_grisou_12_96_metis_true_Alya.x/traces.csv.gz
-: exp_13-v1_grisou_12/13-v1_grisou_12_96_sfc_true_Alya.x.dir/scorep_13-v1_grisou_12_96_sfc_true_Alya.x/traces.csv.gz
-: exp_13-v1_grisou_12/13-v1_grisou_12_192_sfc_true_Alya.x.modif.dir/scorep_13-v1_grisou_12_192_sfc_true_Alya.x.modif/traces.csv.gz
-
-#+begin_src R :results output :session :exports both :var files=exp_13_traces
-files <- strsplit(files, "\n")[[1]]
-filename <- files[[1]]
-filename;
-meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[3], "_"));
-EID = meta[2]
-Platform = meta[3]
-Nodes = meta[4]
-NP = meta[5]
-Partitioning = meta[6]
-Infiniband = meta[7]
-Alya = meta[8]
-EID;
-Platform;
-Nodes;
-NP;
-Partitioning;
-Infiniband;
-Alya;
-#+end_src
-
-#+RESULTS:
-: [1] "exp_13-v1_grisou_12/13-v1_grisou_12_96_sfc_true_Alya.x.modif.dir/scorep_13-v1_grisou_12_96_sfc_true_Alya.x.modif/traces.csv.gz"
-: [1] "13-v1"
-: [1] "grisou"
-: [1] "12"
-: [1] "96"
-: [1] "sfc"
-: [1] "true"
-: [1] "Alya.x.modif"
-*** Read in R to =dft13=
-#+begin_src R :results output :session :exports both :var files=exp_13_traces
-library(readr);
-library(dplyr);
-alya_scorep_trace_read <- function(filename, basedir = ".")
-{
- meta <- unlist(strsplit(unlist(strsplit(filename, "/"))[3], "_"));
- fullpath <- paste0(basedir, "/", filename);
- read_csv(fullpath,
- col_names=c("Rank", "Start", "End", "Value"),
- progress=FALSE) %>%
- # Transform Value to factor
- mutate(Value = as.factor(Value)) %>%
- # Detect begin and end of iterations
- mutate(Iteration = case_when(
- (.$Value == "timste") ~ 1,
- (.$Value == "endste") ~ -1,
- TRUE ~ 0)) %>%
- group_by(Rank) %>%
- # Create a logical to detect observations within iterations
- mutate(Iteration = as.logical(cumsum(Iteration))) %>%
- # Get only observations that belongs to some iteration
- filter(Iteration == TRUE) %>%
- ungroup() %>%
- # Create the iteration by cumsum
- mutate(Iteration = case_when(
- (.$Value == "timste") ~ 1,
- TRUE ~ 0)) %>%
- group_by(Rank) %>%
- mutate(Iteration = cumsum(Iteration)) %>%
- mutate(EID = as.factor(meta[2]),
- Platform = as.factor(meta[3]),
- Nodes = as.factor(meta[4]),
- NP = as.factor(meta[5]),
- Partitioning = as.factor(meta[6]),
- Infiniband = as.logical(meta[7]),
- Alya = as.factor(meta[8]));
-}
-
-files <- strsplit(files, "\n")[[1]]
-dft13 <- do.call("rbind", lapply(files, function(x) { alya_scorep_trace_read(x, basedir="data/07/17e0e2-99b8-44c7-acfc-4d4ad685a2c5") }));
-
-#
-# Keep only states that are interesting
-#
-
-dft13 <- dft13 %>% filter(Value %in% c("nsi_matrix", "solver", "nsi_inisol"));
-#+end_src
-
-#+RESULTS:
-#+begin_example
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- Start = col_double(),
- End = col_double(),
- Value = col_character()
-)
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- Start = col_double(),
- End = col_double(),
- Value = col_character()
-)
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- Start = col_double(),
- End = col_double(),
- Value = col_character()
-)
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- Start = col_double(),
- End = col_double(),
- Value = col_character()
-)
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- Start = col_double(),
- End = col_double(),
- Value = col_character()
-)
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- Start = col_double(),
- End = col_double(),
- Value = col_character()
-)
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- Start = col_double(),
- End = col_double(),
- Value = col_character()
-)
-Parsed with column specification:
-cols(
- Rank = col_integer(),
- Start = col_double(),
- End = col_double(),
- Value = col_character()
-)
-Warning messages:
-1: In bind_rows_(x, .id) : Unequal factor levels: coercing to character
-2: In bind_rows_(x, .id) : Unequal factor levels: coercing to character
-3: In bind_rows_(x, .id) : Unequal factor levels: coercing to character
-#+end_example
-
-#+begin_src R :results output :session :exports both
-dft13 %>% .$NP %>% unique
-dft13 %>% .$Alya %>% unique
-dft13;
-dft13 %>% mutate(Partitioning=as.factor(Partitioning), NP=as.factor(NP), Alya=as.factor(Alya)) %>% summary;
-#+end_src
-
-#+RESULTS:
-#+begin_example
-[1] "96" "192"
-[1] "Alya.x.modif" "Alya.x"
-Source: local data frame [2,695,680 x 12]
-Groups: Rank [192]
-
- Rank Start End Value Iteration EID Platform Nodes NP
-
-1 95 0.333969 0.333970 nsi_inisol 1 13-v1 grisou 12 96
-2 95 0.335719 1.417964 nsi_matrix 1 13-v1 grisou 12 96
-3 95 1.420657 1.978257 solver 1 13-v1 grisou 12 96
-4 95 1.982556 2.796641 solver 1 13-v1 grisou 12 96
-5 95 2.799346 2.991894 solver 1 13-v1 grisou 12 96
-6 95 3.010278 3.864155 solver 1 13-v1 grisou 12 96
-7 95 4.066358 4.066361 nsi_inisol 1 13-v1 grisou 12 96
-8 95 4.066362 4.119009 solver 1 13-v1 grisou 12 96
-9 95 4.888364 5.940038 nsi_matrix 1 13-v1 grisou 12 96
-10 95 5.942733 6.901603 solver 1 13-v1 grisou 12 96
-# ... with 2,695,670 more rows, and 3 more variables: Partitioning ,
-# Infiniband , Alya
- Rank Start End Value
- Min. : 0.00 Min. : 0.1715 Min. : 0.1715 solver :1843200
- 1st Qu.: 35.75 1st Qu.: 270.0145 1st Qu.: 270.2660 nsi_inisol: 483840
- Median : 71.50 Median : 529.9820 Median : 530.3756 nsi_matrix: 368640
- Mean : 79.50 Mean : 572.4097 Mean : 572.7664 doiter : 0
- 3rd Qu.:119.25 3rd Qu.: 785.8916 3rd Qu.: 786.1151 endste : 0
- Max. :191.00 Max. :1721.8589 Max. :1721.8598 nastin : 0
- (Other) : 0
- Iteration EID Platform Nodes NP
- Min. : 1.00 13-v1:2695680 grisou:2695680 12:2695680 192:1797120
- 1st Qu.: 21.00 96 : 898560
- Median : 47.00
- Mean : 47.81
- 3rd Qu.: 74.00
- Max. :100.00
-
- Partitioning Infiniband Alya
- metis:1347840 Mode:logical Alya.x :1347840
- sfc :1347840 TRUE:2695680 Alya.x.modif:1347840
- NA's:0
-#+end_example
-*** Plot only "computing time"
-**** nastin solver 3 functions =nsi_matrix=, =solver=, =nsi_inisol=
-#+begin_src R :results output graphics :file img/exp_13_alya_nastin_solver_3_funs.png :exports both :width 1200 :height 400 :session
-library(ggplot2);
-dft13 %>%
- filter(Rank != 0) %>%
- filter(NP == 192, Partitioning == "sfc") %>%
- group_by(Rank, Iteration, EID, Platform, Nodes, NP, Partitioning, Infiniband, Alya) %>%
- summarize(NewComp = sum(End - Start)) %>%
- ggplot(aes(x=Iteration, y=NewComp, color=as.factor(Rank))) +
- theme_bw(base_size=12) +
- geom_point() +
- ylim(0,NA) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning))) +
- theme(legend.position="none") +
- facet_grid(Partitioning~Alya);
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_13_alya_nastin_solver_3_funs.png]]
-**** nastin solver only =nsi_matrix=
-#+begin_src R :results output graphics :file img/exp_13_alya_nastin_nsi_matrix.png :exports both :width 1200 :height 400 :session
-library(ggplot2);
-dft13 %>%
- filter(Value == "nsi_matrix") %>%
- filter(Rank != 0) %>%
- filter(NP == 192, Partitioning == "sfc") %>%
- group_by(Rank, Iteration, EID, Platform, Nodes, NP, Partitioning, Infiniband, Alya) %>%
- summarize(NewComp = sum(End - Start)) %>%
- ggplot(aes(x=Iteration, y=NewComp, color=as.factor(Rank))) +
- theme_bw(base_size=12) +
- geom_point() +
- ylim(0,NA) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning))) +
- theme(legend.position="none") +
- facet_grid(Partitioning~Alya);
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_13_alya_nastin_nsi_matrix.png]]
-
-**** nastin solver only =solver=
-#+begin_src R :results output graphics :file img/exp_13_alya_nastin_solver.png :exports both :width 1200 :height 400 :session
-library(ggplot2);
-dft13 %>%
- filter(Value == "solver") %>%
- filter(Rank != 0) %>%
- filter(NP == 192, Partitioning == "sfc") %>%
- group_by(Rank, Iteration, EID, Platform, Nodes, NP, Partitioning, Infiniband, Alya) %>%
- summarize(NewComp = sum(End - Start)) %>%
- ggplot(aes(x=Iteration, y=NewComp, color=as.factor(Rank))) +
- theme_bw(base_size=12) +
- geom_point() +
- ylim(0,NA) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning))) +
- theme(legend.position="none") +
- facet_grid(Partitioning~Alya);
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_13_alya_nastin_solver.png]]
-
-**** nastin solver only =nsi_inisol=
-#+begin_src R :results output graphics :file img/exp_13_alya_nastin_nsi_inisol.png :exports both :width 1200 :height 400 :session
-library(ggplot2);
-dft13 %>%
- filter(Value == "nsi_inisol") %>%
- filter(Rank != 0) %>%
- filter(NP == 192, Partitioning == "sfc") %>%
- group_by(Rank, Iteration, EID, Platform, Nodes, NP, Partitioning, Infiniband, Alya) %>%
- summarize(NewComp = sum(End - Start)) %>%
- ggplot(aes(x=Iteration, y=NewComp, color=as.factor(Rank))) +
- theme_bw(base_size=12) +
- geom_point() +
- ylim(0,NA) +
- geom_line (aes(group=as.factor(Rank):as.factor(Partitioning))) +
- theme(legend.position="none") +
- facet_grid(Partitioning~Alya);
-#+end_src
-
-#+RESULTS:
-[[file:img/exp_13_alya_nastin_nsi_inisol.png]]
-**** Amount of calls to these functions along iterations
-#+begin_src R :results output :session :exports both
-dft13 %>%
- filter(Value == "nsi_matrix") %>%
- filter(Rank != 0) %>%
- filter(NP==192, Rank == 95, Iteration %in% c(1,100), Alya == "Alya.x") %>%
- mutate(Duration = End - Start) %>%
- select(-EID, -Platform, -Nodes, -Infiniband, -NP, -Partitioning) %>%
- group_by(Iteration) %>%
- mutate(N=n()) %>%
- ungroup() %>%
- arrange(Start, Iteration) %>%
- as.data.frame()
-#+end_src
-
-#+RESULTS:
-#+begin_example
- Rank Start End Value Iteration Alya Duration N
-1 95 0.283716 0.856840 nsi_matrix 1 Alya.x 0.573124 10
-2 95 0.378482 0.927219 nsi_matrix 1 Alya.x 0.548737 10
-3 95 3.298924 3.875023 nsi_matrix 1 Alya.x 0.576099 10
-4 95 3.320866 3.865816 nsi_matrix 1 Alya.x 0.544950 10
-5 95 6.500311 7.052999 nsi_matrix 1 Alya.x 0.552688 10
-6 95 6.595464 7.164909 nsi_matrix 1 Alya.x 0.569445 10
-7 95 9.646956 10.194573 nsi_matrix 1 Alya.x 0.547617 10
-8 95 9.782018 10.349733 nsi_matrix 1 Alya.x 0.567715 10
-9 95 12.637448 13.184027 nsi_matrix 1 Alya.x 0.546579 10
-10 95 12.929592 13.499010 nsi_matrix 1 Alya.x 0.569418 10
-11 95 902.438214 902.987738 nsi_matrix 100 Alya.x 0.549524 6
-12 95 905.091038 905.636659 nsi_matrix 100 Alya.x 0.545621 6
-13 95 907.747868 908.293808 nsi_matrix 100 Alya.x 0.545940 6
-14 95 914.429194 914.996252 nsi_matrix 100 Alya.x 0.567058 6
-15 95 917.061845 917.634066 nsi_matrix 100 Alya.x 0.572221 6
-16 95 919.813262 920.384819 nsi_matrix 100 Alya.x 0.571557 6
-#+end_example
-
-There are, for =nsi_matrix= in Rank 95:
-- 15 calls in iteration 1
-- but only 9 in iteration 100.
-
-#+begin_src R :results output :session :exports both
-dft13 %>%
- filter(Value == "solver") %>%
- filter(Rank != 0) %>%
- filter(NP == 192, Partitioning == "sfc") %>%
- filter(Rank == 3, Iteration %in% c(1,100), Alya == "Alya.x") %>%
- mutate(Duration = End-Start) %>%
- select(-EID, -Platform, -Nodes, -Infiniband, -NP, -Partitioning) %>%
- group_by(Iteration) %>%
- mutate(N=n()) %>%
- ungroup() %>%
- arrange(Start, Iteration) %>%
- as.data.frame()
-#+end_src
-
-#+RESULTS:
-#+begin_example
- Rank Start End Value Iteration Alya Duration N
-1 3 0.806091 1.397790 solver 1 Alya.x 0.591699 25
-2 3 1.400163 1.883410 solver 1 Alya.x 0.483247 25
-3 3 1.884954 1.987991 solver 1 Alya.x 0.103037 25
-4 3 1.995055 2.475562 solver 1 Alya.x 0.480507 25
-5 3 2.558144 2.693178 solver 1 Alya.x 0.135034 25
-6 3 3.821168 4.586247 solver 1 Alya.x 0.765079 25
-7 3 4.588758 5.062848 solver 1 Alya.x 0.474090 25
-8 3 5.064446 5.387481 solver 1 Alya.x 0.323035 25
-9 3 5.394513 5.799146 solver 1 Alya.x 0.404633 25
-10 3 5.881718 6.016954 solver 1 Alya.x 0.135236 25
-11 3 7.130104 7.898844 solver 1 Alya.x 0.768740 25
-12 3 7.901355 8.296631 solver 1 Alya.x 0.395276 25
-13 3 8.298181 8.584412 solver 1 Alya.x 0.286231 25
-14 3 8.591484 9.002578 solver 1 Alya.x 0.411094 25
-15 3 9.085122 9.221985 solver 1 Alya.x 0.136863 25
-16 3 10.303988 11.044705 solver 1 Alya.x 0.740717 25
-17 3 11.047239 11.510650 solver 1 Alya.x 0.463411 25
-18 3 11.512203 11.736265 solver 1 Alya.x 0.224062 25
-19 3 11.743266 12.150266 solver 1 Alya.x 0.407000 25
-20 3 12.232800 12.369987 solver 1 Alya.x 0.137187 25
-21 3 13.452176 14.151580 solver 1 Alya.x 0.699404 25
-22 3 14.154113 14.507091 solver 1 Alya.x 0.352978 25
-23 3 14.508630 14.696163 solver 1 Alya.x 0.187533 25
-24 3 14.703222 15.079027 solver 1 Alya.x 0.375805 25
-25 3 15.161596 15.297411 solver 1 Alya.x 0.135815 25
-26 3 914.954351 915.583598 solver 100 Alya.x 0.629247 15
-27 3 915.586138 915.877785 solver 100 Alya.x 0.291647 15
-28 3 915.879321 916.002794 solver 100 Alya.x 0.123473 15
-29 3 916.009884 916.294803 solver 100 Alya.x 0.284919 15
-30 3 916.381085 916.513058 solver 100 Alya.x 0.131973 15
-31 3 917.587713 918.332705 solver 100 Alya.x 0.744992 15
-32 3 918.335221 918.630677 solver 100 Alya.x 0.295456 15
-33 3 918.632221 918.755400 solver 100 Alya.x 0.123179 15
-34 3 918.762429 919.045410 solver 100 Alya.x 0.282981 15
-35 3 919.131495 919.265010 solver 100 Alya.x 0.133515 15
-36 3 920.334900 921.001416 solver 100 Alya.x 0.666516 15
-37 3 921.003959 921.290349 solver 100 Alya.x 0.286390 15
-38 3 921.291913 921.425185 solver 100 Alya.x 0.133272 15
-39 3 921.432295 921.728660 solver 100 Alya.x 0.296365 15
-40 3 921.815039 921.947326 solver 100 Alya.x 0.132287 15
-#+end_example
-
-There are, for =solver= in Rank 95:
-- 25 calls in iteration 1
-- but only 15 in iteration 100.
-
-Let's check for other ranks and kernels:
-
-#+begin_src R :results output :session :exports both
-dft13 %>%
- filter(Value == "solver") %>%
- filter(Rank != 0) %>%
- filter(NP == 192, Partitioning == "sfc") %>%
- filter(Iteration %in% c(1,100), Alya == "Alya.x") %>%
- mutate(Duration = End-Start) %>%
- select(-EID, -Platform, -Nodes, -Infiniband, -NP, -Partitioning) %>%
- group_by(Rank, Value, Iteration) %>%
- mutate(N=n()) %>%
- ungroup() %>%
- select(Rank, Iteration, N, Value) %>%
- unique %>%
- sample_n(20) %>%
- arrange(Iteration, Value, N)
-#+end_src
-
-#+RESULTS:
-#+begin_example
-# A tibble: 20 × 4
- Rank Iteration N Value
-
-1 163 1 25 solver
-2 36 1 25 solver
-3 129 1 25 solver
-4 66 1 25 solver
-5 89 1 25 solver
-6 185 1 25 solver
-7 97 1 25 solver
-8 152 1 25 solver
-9 114 1 25 solver
-10 159 1 25 solver
-11 94 1 25 solver
-12 57 1 25 solver
-13 134 1 25 solver
-14 46 100 15 solver
-15 101 100 15 solver
-16 159 100 15 solver
-17 147 100 15 solver
-18 153 100 15 solver
-19 124 100 15 solver
-20 174 100 15 solver
-#+end_example
-
-#+begin_src R :results output :session :exports both
-dft13 %>%
- filter(Value == "nsi_matrix") %>%
- filter(Rank != 0) %>%
- filter(NP == 192, Partitioning == "sfc") %>%
- filter(Iteration %in% c(1,100), Alya == "Alya.x") %>%
- mutate(Duration = End-Start) %>%
- select(-EID, -Platform, -Nodes, -Infiniband, -NP, -Partitioning) %>%
- group_by(Rank, Value, Iteration) %>%
- mutate(N=n()) %>%
- ungroup() %>%
- select(Rank, Iteration, N, Value) %>%
- unique %>%
- sample_n(20) %>%
- arrange(Iteration, Value, N)
-#+end_src
-
-#+RESULTS:
-#+begin_example
-# A tibble: 20 × 4
- Rank Iteration N Value
-