# -*- coding: utf-8 -*-
#+TITLE:  Research journal
#+LANGUAGE: EN
#+CATEGORY: INRIA
#+STARTUP: overview indent inlineimages logdrawer hidestars
#+TAGS: [ SIMGRID : SMPI(s) ]
#+TAGS: [ PROGRAMMING : C(C) CPP(c) PYTHON(p) R(r) ]
#+TAGS: [ TOOLS : ORGMODE(o) GIT(g) GDB ]
#+TAGS: [ EXPERIMENTS(x) : EXP_SETUP EXP_EXEC EXP_RESULT EXP_ANALYSIS ]
#+TAGS: TRACING(t) PERFORMANCE(X) PROFILING(R) BUG(b) PAPER(P) HPL(h) MEETING(m) G5K(G) VALIDATION(v) REPORT(V)
#+TAGS: STAMPEDE(S) CBLAS(B)
#+LOGGING: lognoterepeat
#+SEQ_TODO: TODO(t!) STARTED(s!) WAITING(w@) APPT(a!) CANCELLED(c@) DEFERRED(f@) | DONE(d!)
#+SEQ_TODO: UNREAD | READ
#+SEQ_TODO: GOOD(g!) CRITICISM INTERESTING(w!) INVESTIGATE PROPOSAL
#+SEQ_TODO: QUESTION(q!) | RESOLVED(r!)

* 2017
** 2017-02 February
*** 2017-02-06 Monday
**** DONE Read [[file:5218a011.pdf][An Evaluation of Network Architectures for Next Generation Supercomputers]] :PAPER:
Bibtex: Chen:2016:ENA:3019057.3019059
- The authors roughly do what we want to do: they use a simulator to do performance evaluation of different topologies,
  with different workloads and routing algorithms.
- In a first part, they detail what are these topologies, routing algorithms and workloads.  This could give us some
  ideas of what to test. Maybe we could try to reproduce their results?
- They focus on topologies having:
  + Full uniform bandwidth.
  + Have good partitionability and can be grown modularly.
  + Come at a lower cost than a 3-level fat tree (which is the state of the art in terms of pure performances).
- They test an adversarial traffic (task i sends to task (i+D) mod G,  tuned to “be bad”).
  + Fat tree has great performances, regardless of the routing algorithms.
  + Other topologies (Dragonfly+, Stacked all-to-all, Stacked 2D hyperX) have terrible performances with direct
    routing. For indirect or adaptive routing, performances are much better (but still a factor 2 lower than the fat
    tree).
- Then, they test neighbor traffic (the logical topology is a grid for instance).
  + Again, the fat tree has nearly full performances, regardless of the routing algorithm.
  + Other topologies have lower performances with indirect routing. Their performances are ok with direct or adaptive
    routing.
- Next, they look at AMR.
  + Here, all topologies and routing algorithms have poor performances.
  + The average throughput is high at the beginning, but decreases very quickly to nearly 0.  This long tail with low
    throughput accounts for the major part of the execution time.
  + Thus, AMR seems to be inherently bad for parallelism.
- To sum up, the best routing algorithm is the adaptive routing (except maybe for the fat tree), the best topology is
  the fat tree.
- The authors then had a look at random-mappings of the processes to the nodes (until now, the mapping was ideal). This
  could reflect what would do a scheduler which is not topology-aware.  In general, with adaptive routing, the Fat
  Tree and the Dragonfly+ are very robust to irregular placements, the completion time is not impacted too much. This is
  not the case for stacked topologies (due to a lack of path diversity). Thus, we should use a topology-aware job
  scheduler, especially for stacked topologies.  With non-adaptive routing, all the topologies suffer of performance
  degradations.
**** DONE Talk with Arnaud about the internship.                 :MEETING:
***** Two possible things to have a look at.
- Simulate the impact of network failures on the performances.
  + May need to work on Simgrid implementation, to handle failures.
  + A recent paper has shown that, in their case, removing one of the two root switches of their fat tree did not impact
    significantly the performances.
  + A reason is that jobs rarely occupy the full tree, they are localized in one of its sub-trees. Thus, nearly no
    communication go to the top switches.
- Modelize in Simgrid [[https://www.tacc.utexas.edu/stampede/][Stampede]] super computer.
  + It uses a fat tree topology.
  + We have access to real benchmark results.
  + We have access to its configuration (e.g. OS and compiler used).
***** A first step for this internship would be to run HPL on a fat tree, with Simgrid.
***** Some features of Simgrid to speedup a simulation.
- A macro to only run the first steps of a loop and infer the total time from it.
- An allocator (replacing malloc/free) to share memory between processes.
***** Some technical details about Simgrid
- For every process, we run each piece of code until we reach a MPI operation.  This gives us the execution time of this
  code block.
- We know all the communication flows of the “current step”, thanks to the routing.
  We thus have a list of linear constraints (e.g. the bandwidth of all flows going
  through a same link should not exceed the capacity of this link). We solve this
  by maximizing the minimum bandwidth of any flow (empirically, this is close to the
  reality, where flows have a fair share of the resources).
- Routing is made with an AS hierarchy. There are local routing decisions (within an
  AS) and global routing decisions (between two AS).
***** There exists other simulators.
Mainly codes/ross. Discrete event simulators, so they consider the problem at a lower level.  But being too precises has
some drawbacks:
- The exact version of every piece of code can have noticeable impact → tedious to calibrate.
- The simulation takes much more time, does not scale as much as Simgrid.
*** 2017-02-07 Tuesday
**** DONE Begin writing a journal \o/
**** DONE Read [[file:8815a909.pdf][Characterizing Parallel Scientific Applications on Commodity Clusters: An Empirical Study of a Tapered Fat-Tree]] :PAPER:
Bibtex: Leon:2016:CPS:3014904.3015009
- The authors want to characterize the behavior of applications that run on their clusters, with an emphasis on
  communication requirements.
- This should help to make more informed choices when building new clusters (should we use our budget to get more links
  or more nodes?).
- They measured the utilization of their cluster during one week. It has a fat tree topology.  The measurements show
  that the network is not used very much: the maximal link utilization is approximately 50%, the average link
  utilization is 2.4%.
- They did the same measures with a tapered fat tree (they removed one of the root switches).  Except for some outliers
  having a 90% link utilization at some point, this modification did not had a major impact on the link utilization,
  which was 3.9%.
- The authors recorded which type of jobs were submitted. A great majority of them was really small.  95% of jobs have
  at most 16 nodes, 76% have only one node. Jobs of less than 64 nodes consume 75% of the time.  Thus, if the jobs are
  well placed, the need for distant communications is very low, which explains the good performances of the tapered fat
  tree.  Of course this may change from one cluster to another, so we should reproduce these measurements and make our
  own conclusions.
- Then, the authors removed one of there two top switches.
- A first micro-benchmark shows that it only impacts the aggregate bisection bandwidth, for large messages (> 32kB).
- Then, they evaluated the impact of the tapering on the performances of several “real-life” applications.
- They found that only one of these applications was sensible to the tapering. This application does collective
  communications as well as point-to-point commulnications of large messages.
- However, the impact on the execution time of this application remains small: only 1-2% (it impacts its communication
  time by 6-7.5% which itself accounts for only 9-15%). Furthermore, this only happens for a large number of nodes (>
  512).
- Finally, the authors claim that next generation hardware (faster CPU, memory and network, accelerators...) will lead
  to some rewriting of the application to leverage this new hardware. In some applications, message sizes will be
  larger.  Thus, a tapered fat tree may have more impact with this new hardware, new experimentations will be needed to
  find out.
**** DONE Some thoughts regarding previous paper, to discuss with Arnaud :PAPER:MEETING:
:LOGBOOK:
- State "DONE"       from "TODO"       [2017-03-03 Fri 17:51]
:END:
***** Can we have data about the utilization of clusters we will work on (Stampede, Bull)?
- It would help us to find relevant hypothesis (e.g. “pruning the fat-tree will not have any impact”).
- We need this for the simulation. What kind of jobs should we run? Small ones? Large ones?
***** Can we have data about the nature of the jobs submitted on these clusters?
- What are these applications?
- What fraction of the time do they use for communications?
- Small or large messages?
- Again, it will help us to make hypothesis and perform meaningful experimentations.
\to It changes a lot from one cluster to another, or even across time. It is also hard to record (a batch scheduler does
not know the nature of the jobs that it handles).
***** How to simulate “big nodes”?
- Can we simulate MPI+OpenMP programs with Simgrid?
- The paper from Christian explains briefly how Simgrid simulates multi-core machines (with one MPI process per core, no
  threads).  Why don't they talk about it in the other paper? Both papers are from the same year.
\to It would be very hard to support OpenMP in Simgrid, the standard is quite big. Also, in OpenMP, communications are
made with shared memory, so much more difficult to track than MPI communications.
***** Are time-independent traces larger than the “classical” traces?
\to No, same thing.
***** With ScalaTrace, traces have “near-constant size”. How?
\to Compression, lossless or lossy.
***** What is “detached mode” in point-to-point communication?
- Does the OS of the sender interrupt it, to ask it to send the data?
- If so, why is large mode slower for the sender? In detached mode, the sender has to stop what it is doing,
  whereas in synchronous mode it is waiting.
\to Yes, the kernel interrupts the sender when the receiver is ready. Simgrid does not model the small messages used for
the synchronization.
***** What does the community think of closed source simulators, like xSim? Researchers behind xSim are doing strong claims that cannot be verified by independent researchers...
***** Why are there never confidence intervals in the plots of Simgrid's papers?
\to They are often not needed, because of too small variation.
***** About the paper Git/Org-mode
- Is there an implementation somewhere?  Creating custom git commands seems [[http://thediscoblog.com/blog/2014/03/29/custom-git-commands-in-3-steps/][really easy]].
  \to Yes, but not packaged yet. To test the “beta version”, ask Vincent.
- Was not convinced by the subsection 4.3.3 (“fixing code”). When terminating an experiment, they revert all the changes
  made to the source code since these may be ad hoc changes. Then the user has to cherry pick the changes (s)he wants to
  keep. Sounds really dirty...  It seems better to have generic scripts that you configure by giving command line
  arguments and/or configuration files.  Then you can simply put these arguments/files in the journal.
***** Routing in Simgrid (according to the doc)
- Routing tables are static (to achieve high performance).  → Does it mean that handling link failures and dynanmic
  re-routing will require a large code refactoring? What about the performance penalty?
- Routing algorithms are either based on short path (e.g. Floyd, Dijkstra) or manually entered. What about “classical”
  algorithms like D-mod-K?  An example is provided on [[https://github.com/simgrid/simgrid/blob/master/examples/platforms/cluster_fat_tree.xml][Github]]. The example implements a two levels fat-tree with
  D-mod-K. However, D-mod-K is not specified in the XML, it seems to be implicit. Does it mean that we are forced to use
  this routing algorithm for fat trees?
\to Read the code. Shortest path routing is a feature introduced by some Belgian researchers. For specific topologies like
fat-trees, the routing algorithm is hard-coded.
**** DONE Read [[file:hal-01415484.pdf][Simulating MPI applications: the SMPI approach]]      :PAPER:
Bibtex: degomme:hal-01415484
- This paper is about simulation of HPC systems.
- The authors claim that some research papers are based on simulation made with one-off programs with poor
  documentation, making simplifying assumptions. Worse, these programs are sometimes not public. This is a big issue for
  reproducibility.
- The whole paper consider several important aspects that a good simulator should take care of.
- Several use cases for simulation.
  + Quantitative performance evaluation (what will be the performances if we take a bigger version of our hardware?).
  + Qualitative performance evaluation (what will be the performances if we take different hardware?).
  + Detection of hardware misconfiguration (leading to unexpected performance behaviors).
  + MPI runtime tuning (e.g. choosing the algorithms of MPI collective operations).
  + Teaching (supercomputers are expensive, we cannot let the students play with them).
***** Capturing the behavior of an application.
- Off-line simulation. A trace of MPI communication events is first obtained and then replayed.
  + We measure the durations of the CPU bursts. Then, when replaying the application, we modify them to account for the
    performance differences between the target platform and the platform used to get the traces.
  + One problem is the size of the traces, which can be very large. To fix this, we may only record aggregated
    statistics. They can be enough to detect some anomalies, but we cannot do more in-depth analysis.
  + Another issue is extrapolation. Being able to extrapolate in the general case require assumptions hardly justifiable.
  + In SMPI, they use “time-independent traces”. Instead of recording time durations, they log the number of
    instructions and the number of bytes transferred by MPI primitives. These are independent of the hardware, so the
    extrapolation issue is fixed.
  + It does not solve anything for applications that adapt their behavior to the platform. But this is hopeless with
    off-line simulation.
  + There is still the issue of very large traces, they grow linearly with the problem size and the number of
    processes. It seems to be fixed by ScalaTrace, but no explanation is given.
- On-line simulation. The actual application code is executed, part of the instruction stream is intercepted and passed
  to a simulator.
  + Several challenges. Intercepting MPI calls. Interractions between the application and the simulation kernel.
    Obtaining full coverage of MPI standard. Over-subscribing resources.
  + Several possibilities to capture MPI calls. Use PMPI interface (provided by every MPI implementation), but limited
    to the high- level calls. Design a specific MPICH or OpenMPI driver, but tie the solution to a specific
    implementation. One can also develop an ad-hoc implementation of the MPI standard.
  + Many tool fold the application into a single process with several threads. This raise an issue for global variables,
    they must be protected. One can duplicate the memory area of the global variables, or use a trick based on the
    Global Offset Table (GOT).
  + SMPI is based on a complete reimplementation of MPI standard. No full-coverage yet (e.g. remote memory access or
    multithreaded MPI applications).
  + Run MPICH internal compliance tests as part of their automatic testing.
  + To protect global variables, duplicate their memory zone using mmap (smart thing, much more efficient thanks to
    COW).
***** Modeling the infrastructure (network and CPU)
- Network modeling.
  + Several solutions exist to modelize the network.
  + Packet-level simulation, here we look at individual packets. It is very precise, but it is hard to know precisely
    what we are modeling. Being precise with a wrong model is useless. Moreover, this model is very costly in terms of
    simulations.
  + Flow model. The finest grain here is the communication. Time to transfer a message of size S from i to j: L_i,j + S*B_i,j. The
    B_i,j are not constant, they need to be evaluated for every moment. This model catch some complex behaviors (e.g. RTT unfairness
    of TCP). Quite complex to implement, more costly than the delay model. Also, until recently, contentions could be neglected.
  + Delay model, we have some equations to describe the communication times (e.g. LogP, LogGPS). It is elegant and cheap
    in terms of simulations, but very unprecise. Does not take into account network topology (and eventual contentions)
    and suppose a processor can only send one message at a time (single-port model).
  + SMPI uses a hybrid network model. Point-to-point communications are divided in three modes: asynchronous, detached
    and synchronous.  Each mode has different values of bandwidth and latency, estimated by doing some benchmarks and
    then a linear regression.
  + To modelize network contentions, SMPI has three logical links for any physical link: a downlink, an uplink, and a
    limiter link.  The bandwidth of uploads (resp. downloads) must be lower than the capacity of uplinks
    (resp. downlinks). The sum of the bandwidths must be lower than the capacity of the limiter link.
- CPU modeling.
  + Like network modeling, several solutions.
  + Microscopic models, very precise, but also very costly.
  + Models with a coarser grain. For instance, we neglect the CPU load induced by communications → focus on Sequential
    Execution Blocks (SEB).
  + Most simplistic model: “CPU A is x times faster than CPU B”. Results ok for similar architectures, but not precise
    at all if too different.  For instance, number of registers, number of hypertreading cores, speed of floating point
    computations, bandwidth to memory, etc.
  + Thus, impossible to predict precisely without a perfect knowledge of the system state (and therefore a microscopic
    model).
  + Approach of SMPI: run SEB on a processor of the target architecture. Predict performances of *similar* architecrues by
    applying a constant factor.
  + Also, not all the code logic is data dependent. We can therefore greatly decrease the simulation time with two
    tricks.
    - Kernel sampling. Annotate some regions with macros. Execute them only a few times to obtain estimations, then skip
      them.
    - Memory folding. Share some data structures across processes.
***** Modeling the collective operations
- Again, several solutions for the modelization.
- More analytical ones: each collective operation has a cost equation (depending for instance on the message size and
  the number of processes). As discussed for the network modelization, such approaches do not catch the eventual network
  contention.
- Another approach is to benchmark each collective operation on the target platform, with various parameters and
  communicators.  Then, the obtained timings are reinjected in the simulation. We cannot do performance extrapolation
  with this approach. Also, the benchmarking phase may be very long.
- Some replace every collective operation by the corresponding sequence of point-to-point communications (at compile
  time).  This does not capture the logic of selecting the right algorithm.
- Others capture this decomposition into point-to-point communication during the execution, then replay it. But this is
  limited to off-line analysis.
- Simgrid implements all the collective algorithms and selection logics of both OpenMPI and MPICH. We are sure to
  capture correctly the behavior of the operations, but this is an important work. Another interesting feature is that
  the user can chose the selector or the algorithm from the command line.
***** Efficient simulation engine
- Rely on a efficient Discrete Event Simulation (DES) kernel.
- Some simulators parallelized this part (using MPI). But this results in a more complex implementations.
- In the way Simgrid works, there is not much potential parallelism. They therefore decided to keep a sequential DES.
- Simulation cost comes from the application itself (which can be greatly reduced, CPU modelization) and from the flow
  level model.
***** Evaluation
Here, the authors show that the use cases mentionned at the beginning of the paper are all realised by Simgrid.
- Simgrid is very scalable, more than xSim which is already one of the most scalable simulators (self proclaimed).
- Kernel sampling and memory folding enable simulations of non-trivial applications with a very large number of cores.
- Then, the ability to make good predictions is demonstrated with a Mont Blanc project example. Here, Simgrid is much
  closer to the reality than LogGPS model. However, no comparison is done with other simulators, so this result is hard
  to evaluate.
- A quantitative performance extrapolation is demonstrated, showing good results.
- Empirically, the largest error made by SMPI in terms of time prediction is 5%. This allow to use SMPI to detect
  hardware misconfiguration.  Indeed, it already happened to the Simgrid team.
- Similarly to the previous point, the good accuracy of SMPI allow to investigate to find which MPI parameters lead to
  the best performances.
- Finally, for obvious reasons, using a simulator is great for teaching MPI (rather than using a real cluster).
***** Conclusion
- The paper focused on MPI applications.
- But Simgrid has other use cases: formal verification of HPC applications, hybrid applications (MPI+CUDA).
**** DONE Read [[file:hal-01446134.pdf][Predicting the Performance and the Power Consumption of MPI Applications With SimGrid]] :PAPER:
Bibtex: heinrich:hal-01446134
- The paper is about using Simgrid to predict energy consumption.
- This is a challenging question, the modelization is tricky.
  + Power consumption of nodes has a static part (consumption of the node when idle) and a dynamic part.
  + The static part is very significant (~50%), so we should really do something when the core is idle.
  + A first solution is to power off the node, but the latency to power it on is large.
  + Another solution is to use Dynamic Voltage Frequency Scaling (DVFS). This is not limited to the case where the core
    is idle, it can also be used when the load is low but non-null. Performance loss is linear in the decrease of the
    frequency, but the power consumption is quadratic.
  + No other HPC simulator than Simgrid embed a power model yet.
***** Modeling multi-core architecture
- If two processes are in the same node (either a same core, or two cores of a same CPU), the simulation becomes tricky.
  + The “naive” approach is to simply give a fair share to each of htese processes. But it does not take into accoutn
    some memory effects.
  + Simgrid can be pessimistic for processes heavily exploiting the L1 cache. In the simulation, the cache will be cold
    after each MPI call, in reality the cache would be hot.
  + Simgrid can be optimistic for processes heavily exploiting the L3 cache and the memory. In the simulation, they will
    have exclusive access, in reality they will interfer between each other.
***** Modeling energy consumption
- The instantaneous power consumption is P_i,f,w(u) = Pstatic_i,f + Pdynamic_i,f,w * u, for a machine i, a frequency f,
  a computational workload w and a usage u.
- In general, we assume that Pstatic_i,f = Pstatic_i (idle state, the frequency does not matter).
- Users can specify arbitrary relation (linear in the usage) for each possible frequency (in general, they should be
  quadratic in the frequency, but it may change with new technologies).
- Each machine can have its own model, accounting for heterogeneity in the platform.
- Power consumption of each host is exposed to the application, allowing it to dynamically decide to change (or not) the
  current frequency.
***** Modeling multicore computation
- A first step is to run the target application with a small workload using all the cores of a single node, on the target platform.
- Then, re-execute the application with the same workload on top of the simulator (hence using a single core).
- From these measures, associate to each code region a speedup factor that should be applied when emulating.
- In some applications, speedups are very close to 1. In other applications, some regions have a speedup of 0.16 while
  other regions have a speedup of 14. Not taking this into accoutn can result to a large inaccuracy (~20-30%).
***** Modeling the network
- See the other paper for the details on the network model of SMPI.
- The authors also speak about local communications, within a node. They are implemented with shared memory. The model
  here is also piecewise linear, but with less variability and higher speed. However, they did not implement this model,
  they kept the classical network model since local communications were rare enough.
***** Validation
- The authors obtain a very good accuracy for performance estimations (as stated in the previous paper).
- For two of the three applications, they also have a very good accuracy for energy consumption estimations.
- With the last application, the accuracy is bad. The reason is that the application (HPL) does busy waiting on
  communications (with MPI_Probe). In the current model, they assume that it does not cost energy.
***** Experimental environment
Minor modifications to the setup can have a major impact on the performances and/or the power consumption.  The authors
therefore give a list of settings to track.
- Hardware. If we suppose that the cluster is homogeneous, it has to be the case. Two CPU having the same type can still
  exhibit different performances (e.g. if they come from two different batches/factories).
- Date of the measurements. A lot of things having an impact can change in time: temperature of the machine room,
  vibrations, BIOS and firmware version, etc.
- Operating system. The whole software stack and how it is compiled can have a huge impact. Also, always observe a delay
  between the boot and the beginning of experiments.
- Kernel configuration. For instance, its version, the scheduling algorithm, technologies like hyperthreading, etc.
- The application itself and the runtime (e.g. the algorithms used for collective operations).
***** Conclusion / future work
- The approach to simulate power consumption is accurate only if the application is regular in time. To handle
  applications with very different computation patterns, we could specify the power consumption for each code
  region. But to do so, Simgrid has to be modified, and there need to be very precise measurements to instantiate the
  model (impossible with the hardware of Grid 5000, sampling rate of only 1Hz).
- In Simgrid, we can currently not have different network models at the same time, to account for local and remote
  communications.  A refactoring of the code is underway to fix this.
*** 2017-02-08 Wednesday
**** DONE Paper reading.
- Notes have been added in the relevant section.
- One paper read today: “Simulating MPI applications: the SMPI approach”.
*** 2017-02-09 Thursday
**** TODO Read [[file:hdr.pdf][Scheduling for Large Scale Distributed Computing Systems: Approaches and Performance Evaluation Issues]] :PAPER:
Bibtex: legrand:tel-01247932
**** DONE Read [[file:SIGOPS_paper.pdf][An Effective Git And Org-Mode Based Workflow For Reproducible Research]] :ORGMODE:GIT:PAPER:
Bibtex: stanisic:hal-01112795
- A branching scheme for git, based on four types of branches.
  + One src branch, where the code to run the experiments is located. This branch is quite light.
  + One xp branch per experiment, that exists only during the period of the experiment. We can find here all the data
    specific to this experiment. Also a light branch, since limited to an experiment.
  + One data branch, in which all xp branches are merged when they are terminated. Quite an heavy branch, a lot of things.
  + One art branch per article, where only the code and data related to the article are pulled from the data branch.
  When an xp branch is merged in data and deleted, a tag is added. Then, we can easily checkout to this experiment in the future.
- Org-mode used as a laboratory notebook. All details about the experiments (what, why, how...) are written here.
  Thanks to literate programming, the command lines to execute are also contained in the notebook.
**** Presentation about org-mode by Christian.                   :ORGMODE:
- Have a per day entry in the journal.  If you work more than an our wthout writing anything in the journal, there is an
  issue.
- Put tags in the headlines, to be able to search them (e.g. :SMPI: or :PRESENTATION:).  Search with “match” key word.
  Hierarchy of tags, described in the headline.
- About papers, tags READ/UNREAD. Also the bibtex included in the file.  Attach files to the org modeo (different than a
  simple link). Use C-a, then move.
- Spacemacs: add a lot of stuff to evil mode.
- Can also use tags to have link on entries, use the CUSTOM_ID tag.
- Can use org mode to put some code.
**** DONE Paper reading.
- One paper read today: “Predicting the Performance and the Power Consumption of MPI Applications With SimGrid”.
- Notes have been added in the relevant section.
**** DONE Apply the things learnt at the org-mode presentation.
*** 2017-02-10 Friday
**** Tried to get good org-mode settings.                        :ORGMODE:
- Cloned org-mode [[/home/tom/.emacs.d/org-mode][git repository]] to have the latest version (success).
- Tried to install [[http://mescal.imag.fr/membres/arnaud.legrand/misc/init.php][Arnaud's configuration file]] (fail).
- Will try Christian's configuration file on Monday.
**** DONE Paper reading.
- One paper read today: “An Effective Git and Org-Mode Based Worflow for Reproducible Research”.
- Notes have been added in the relevant section.
**** Begin looking at the documentation.
- Documentation about the [[http://simgrid.gforge.inria.fr/simgrid/3.13/doc/platform.html][topology]].
**** Run a matrix product MPI code in a fat tree
- Code from the parallel system course.
- Tried [[https://github.com/simgrid/simgrid/blob/master/examples/platforms/cluster_fat_tree.xml][Github]] example (fat tree =2;4,4;1,2;1,2=, 2 levels and 16 nodes).
- Tried a personal example (fat tree =3;4,4,4;1,4,2;1,1,1=, 3 levels and 64 nodes).
**** DONE Find something to automatically draw a fat tree.
  + Maybe there exists some tools? Did not find one however.
  + Maybe Simgrid has a way to export a topology in a graphical way? Would be very nice.
  + Could adapt the Tikz code I wrote during 2015 internship?
*** 2017-02-13 Monday
**** Keep working on the [[file:/home/tom/Documents/Fac/2017_Stage_LIG/small_tests/matmul.c][matrix product]].                      :SMPI:C:BUG:
- Observe strange behavior.
  + Commit: 719a0fd1775340628ef8f1ec0e7aa4033470356b
  + Compilation: smpicc -O4 matmul.c -o matmul
  + Execution: smpirun --cfg=smpi/bcast:mpich --cfg=smpi/running-power:6217956542.969 -np 64 -hostfile ./hostfile_64.txt
    -platform ./cluster_fat_tree_64.xml ./matmul 2000
  Then, processes 0 and 63 behave very differently than others.
  + Processes 0 and 63 have a communication time of about 0.21 and a computation time of about 1.52.
  + Other processes have a communication time of about 0.85 and a computation time of about 0.75.
  With other topologies and/or matrix sizes, we still have this behavior (more or less accentuated).
- If we change the order of the loops of the sequential matrix product from i-j-k to k-i-j:
  + The execution time is shorter. Hypothesis: this solution has a better usage of the cache.
  + The computation times are decreased (expected), but the communication times are also decreased (unexpected).
  + Still observe the same trend than above for processes 0 and 63.
- Checked with some printf: all processes are the root of a line broadcast and of a column broadcast exactly once (expected).
- Tried several broadcast algorithms (default, mpich, ompi), still have the same behavior.
- Adding a call to MPI_Barrier at the beginning of the for loop fix the issue for the communication (all processes now
  have a communication time of about 0.22) but not for the computation (still the same differences for processes 0 and
  63).
- When using a smaller numbmer of processes (16 or 4), communication times and computation times are more consistent
  (with still some variability).
- With one process and a matrix size of 250, we have a computation time of 0.10 to 0.12. When we have 64 processes and a
  matrix size of 2000, each block has as ize of 250. Thus, we can extrapolate that the “normal” computation time in this
  case should be about 0.8 (8 iterations, so 8*0.10). Thus, processes 0 and 63 have a non-normal behavior, the others
  are ok.
- Also tried other topologies, e.g. a simple cluster. Still have the same behavior (with different times).
  + Again, normal behavior with less processes (e.g. 16).
  + We get a normal behavior if we take hostfile_1600.txt, very strange.
- Bug fixed, the problem came from the hostfile. For some unknown reason, it missed a end-of-line character at the last
  line. I suspect that two processes (0 and 63) were therefore mapped to a same host, because the last host was not
  parsed correctly by smpi. The two versions of the file have been added to the repository.
- Issue reported on [[https://github.com/simgrid/simgrid/issues/136][Github]].
**** Try to optimize the matrix product code.                     :SMPI:C:
- For the record, the following command yields communication times between 0.27 and 0.31 and computation times between
  0.78 and 0.83, for a total time of about 1.14: smpirun --cfg=smpi/bcast:mpich --cfg=smpi/running-power:6217956542.969
  -np 64 -hostfile ./hostfile_64.txt -platform ./cluster_fat_tree_64.xml ./matmul 2000
- Replaced malloc/free by SMPI_SHARED_MALLOC/SMPI_SHARED_FREE. Got similar times (approximately).
- Added SMPI_SAMPLE_GLOBAL(0.5*size, 0.01) to the outer loop of the sequential matrix product. Got similar times (approximately).
- Remark: we should verify more rigorously that these optimizations do not change the estimated time.
- Greatly reduced simulation time (from 8.2s to 0.5s).
- Other optimization: stop initializing the content of the matrices (since we do not care of their content).
**** Meeting with Arnaud.                                        :MEETING:
- There exists some visualization tools for Simgrid, to see the bandwidth that goes on some links.  May be very useful
  in the future, to have a better understanding of what is going on.
- The characteristics of the jobs (number of nodes, patterns of communication) have an important impact on performances.
  However, it is difficult for us to have access to this, we do not own a supercomputer... Maybe Matthieu can have more
  information (e.g. from Bull's clients)?
**** DONE Add supervisors on Github for the journal.
**** Some quick performance tests.                      :SMPI:EXPERIMENTS:
- Run my matrix product code, with SMPI optimizations.
- Use a 2-level fat-tree made with switches of 48 ports.
- First case: non-tapered. We use all the switches. The fat-tree is 2;24,48;1,24;1,1 (total of 1152 nodes).
  + Use 1089 processes, matrix size of 4950.
  + Time: 1.75s.
  + Communication time: 0.94s.
  + Computation time: 0.81s.
- Second case: tapered. We remove half of the root switches. The fat-tree is 2;24,48;1,12;1,1 (still 1152 nodes).
  + Still uses 1089 processes, matrix size of 4950.
  + Time: 1.78s.
  + Communication time: 0.94s.
  + Computation time: 0.82s.
- The observed difference does not seem significant, but we should check with a carefully designed experiment and
  analysis.
- For the record, running the same application on the same topology with only one process takes a time of 3607s.  Thus,
  we have a speedup of about 2026, so an efficiency of 1.86.  This is a very nice speedup (superlinear). Certainly due
  to cache effects.
- These quick tests suggest that we could remove root switches without impacting the performances, even if we use nearly
  the whole fat-tree (this is obvious if we use a small subtree).
**** DONE Run another benchmark (e.g. HPL), with more carefully designed experiments.
**** DONE The 3-level fat-tree was very long to load (aborted). Find why.
*** 2017-02-14 Tuesday
**** Work on experiment automatization.                           :PYTHON:
- Add Python functions to generate topology and host files from a given fat-tree description.
- Adapt Python script and Jupyter notebook from parallel system course to run experiments.
- The matrix size and the number of processes are fixed. We compute matrix products for various numbers of root switches
  (we test fat-trees (2;24,48;1,n;1,1) for n in [1, 24]).
- Results seem very promising. For a matrix size of 6600, we can have as few as 10 root switches without important
  impact on performances (recall that a typical 2-level fat tree with 48 port switches would have 24 root switches). If
  we keep removing switches, then performances are quickly impacted.
- Repeated the experiment with the same topology and the same matrix size, but with only 576 processes.  We observe a
  same trend, we can remove a lot of root switches without having an impact.
**** DONE Ask if it would be possible to have an SSH access to some dedicated computer.
- Does not need to have a lot of cores (Simgrid is not a parallel program), but it would be nice if it had a fast core.
- Needs to be dedicated so as to not perturbate the experiments.
**** Webinar on reproducible research: [[https://github.com/alegrand/RR_webinars/blob/master/7_publications/index.org][Publication modes favoring reproducible research]] :MEETING:
- Speakers: [[http://khinsen.net/][Konrad Hinsen]] and [[http://www.labri.fr/perso/nrougier/][Nicolas Rougier]].
- Two parts in research: dissemination (of the results/ideas) and evaluation (of the researcher).
- If we want reproducible research to become a norm, researchers should be rewarded for this (their reputation should
  also depend on the reproducibility of their research, not only the number of citations or the impact factor).
- The speaker compares reproducible research for two points of view: human part and computer part, both for
  dissemination and evaluation.
***** [[http://www.activepapers.org/][ActivePapers]]
- Not a tool that one should use (yet), neither a proposition of new standard. It is mainly an idea for computer-aided
  research.
- How to have more trusts on the software? The “ideal” one is reimplementation (e.g. ReScience). The speaker tried this
  on a dozen projects, he never got identical results. Other good ideas: good practices like verison control and
  testing, keep track of the software stack (hardware, OS, tools, etc).
- ActivePapers group scripts, software dependencies and data into a same archive.
***** [[http://rescience.github.io/][ReScience]]
- Idea: replicate science.
- For a great majority of papers, we cannot replicate their reuse their code.
- It is hard to publish replication of an original paper, most journals will reject it since not original.
- This is why ReScience was born. It is (currently) used on Github.
- To publish a new study, do a pull-request on ReScience repository. Then it is reviewed openly by reviewers selected by
  the editor.  The replication is improved until it is publishable.
*** 2017-02-15 Wednesday
**** Use Christian’s config files for org mode                   :ORGMODE:
**** Work on the experiment script
     - Parsing more generic fat-tree descriptions. For instance, our current topology description would be
       2;24,48;1,1:24;1,1.  It means that the L1 switches can have between 1 and 24 up ports.
     - Modify the script for experiments to be more generic.
       + Can give as command line arguments the fat-tree description, the (unique) matrix size, the (unique) number of processes.
       + Use Python’s argparse for a cleaner interface.
**** Re-run experiments with this new script
     - Still observe the same trend: we can afford to remove a lot of up-ports for the L1 switches.
     - Some points seem to be outliers. But we have not a lot of points, so it is difficult to say. We whould do more
       experiments to see if these points are still significantly separated from the rest.
*** 2017-02-16 Thursday
**** DONE Enhance/fix Emacs configuration                        :ORGMODE:
- Translate days and months in English.
- Increase the line length limit (120 columns?).
- Reformat the whole document with such limit.
- Add tags where relevant.
- Attach files, instead of putting a link.
**** Try to use even more SMPI optimizations                        :SMPI:
- Currently, we use the macro SMPI_SAMPLE_GLOBAL only once: for the outer for loop of the sequential matrix product.
- Maybe we can also use it for the two other loops? We could also reduce the number of iterations (currently, it is
  0.5*size). Let’s try.
- Currently, we get communication times of about 0.14s and computation times of about 0.42s, for a total time of 0.57s,
  with the following command:
  smpirun --cfg=smpi/bcast:mpich --cfg=smpi/running-power:6217956542.969 -np 64 -hostfile ./hostfile_1152.txt -platform
  ./big_tapered_fat_tree.xml ./matmul 1600
- FAIL. It seems we cannot use imbricated sample blocks. Quite strange, do not understand why...
**** Try to run HPL with Simgrid                                :SMPI:HPL:
- Copied from [[https://gitlab.inria.fr/fheinric/paper-simgrid-energy/tree/master/src/hpl-2.2][Christian’s repository]].
- Compilation fails, don’t know why. But binaries are stored in the git repository (don’t know why either), so I can use
  them to do some first tests.
  In fact, file Make.SMPI needed to be modified. Changed =mkdir= by =mkdir -p=, =ln= by =ln -f= and =cp= by =cp -f=.
  Changed top directory.
  Also, the Makefile couldn’t find the shared library atlas. It was in /usr/lib, but named =libatlas.so.3=. Added a
  symbolic link to =libatlas.so=.
- Tested my laptop (with MPI, not SMPI). With a problem size of 10000 and 12 processes, it corresponds to 16.51 Gflops.
- Tested with SMPI, with a problem size of 10000 and 4 processes. Command:
  smpirun --cfg=smpi/bcast:mpich --cfg=smpi/running-power:6217956542.969 -platform
  ../../../small_tests/cluster_fat_tree_64.xml -hostfile ../../../small_tests/hostfile_64.txt -np 4 ./xhpl
  Result: 1.849Gflops.
- Same thing, with 12 processes. Very similar: 1.847Gflops. Why is it not faster?
- Same thing, with 64 processes. Very similar: 1.858Gflops. Why is it not faster?
- Retried with a freshly compiled program. Still the same thing.
- Understood the issue: it is not enough to specify the number of processes with =-np 12=, we also have to tell it in the
  file =HPL.dat=.
- Tried with =-np 4=, P=4 and Q=1. Now, 6.6224Gflops. We have a speedup of 3.59, which seems reasonable.
- The number of processes given with =-np= must be greater or equal to P \times Q.
- Tried with =-np 4=, P=1 and Q=4. Did not have a noticeable impact on performances (in comparison with P=4, Q=1).
- Tried with =-np 4=, P=2 and Q=2. Did not have a noticeable impact on performances (in comparison with P=4, Q=1).
- Tried with =-np 64=, P=8 and Q=8. Now, 22.46Gflops. Speedup of 12, very disappointing.
- Tried with =-np 64=, P=8 and Q=8 again, but with a problem size of 20000 (it was 10000). Now 52.2Gflops (speedup of 28.3).
**** Comparison with top 500
- For the record, the order of magnitude for Intel desktop CPU of today is between 10 and 100 Gflops, according to [[https://www.pugetsystems.com/labs/hpc/Linpack-performance-Haswell-E-Core-i7-5960X-and-5930K-594/][this
  website]], [[https://setiathome.berkeley.edu/cpu_list.php][this website]] and [[https://asteroidsathome.net/boinc/cpu_list.php][this website]]. My laptop supposedly has a speed of 3.84 Gflops per core and 15.21 Gflops in
  total according to the last two websites.
- According to [[https://en.wikipedia.org/wiki/Raspberry_Pi#Performance][Wikipedia]], the first generation Raspberry Pi has a speed of 0.041 Gflops, a 64 nodes cluster made of
  those has a speed of 1.14 Gflops.
- The first supercomputer has a speed of about 93Pflops, or 93,000,000Gflops.
- The last one has a speed of about 349Tflops, or 349,000Gflops.
- In June 2005, the first one had a speed of about 136Tflops, the last one 1.2Tflops.
- In our settings with 64 nodes, each node has one core that computes at 1Gflops. Thus, our Rpeak is 64Gflops.  We have
  an efficiency of 52.2/64 = 0.81.  This is not bad, compared to the three first supercomputers of the top 500
  (respectively at 0.74, 0.61 and 0.63). But we should maybe not compare the efficiency of a 64 nodes cluster with these
  supercomputers, since it becomes harder to be efficient with a large topology.
**** DONE SMPI optimization of HPL                              :SMPI:HPL:
:LOGBOOK:
- State "DONE"       from "TODO"       [2017-03-22 Wed 17:08]
- State "TODO"       from              [2017-02-16 Thu 16:17]
:END:
- It seems that no SMPI optimization is done in the code obtained from Christian’s repository. Maybe we could speed
  things up?
- Need to check what is the algorithm behing HPL, whether it is regular (to use SMPI_SAMPLE) and data independent (to
  use SMPI_SHARED).
**** DONE Adapt the experience script to run HPL
:LOGBOOK:
- State "DONE"       from "TODO"       [2017-02-17 Fri 15:47]
- State "TODO"       from              [2017-02-16 Thu 17:03]
:END:
- Parse the output (quite ugly to parse, but easy, use methods str.split and list.index).
- Run the same kind of experiments than for the matrix product. Will be much longer if we cannot use SMPI optimizations.
*** 2017-02-17 Friday
**** Refactor the experiment script                               :PYTHON:
- Aim: reuse for HPL the code already done for the matrix product.
- Now, we have a clase =AbstractRunner=, which runs the common logic (e.g. some basic checks on the parameters, or running
  the desired number of experiments).
- We also have classes =MatrixProduct= and =HPL=, containing the piece of codes specific to the matrix product or HPL (e.g. running one experiment).
**** Some strange things with HPL                           :SMPI:BUG:HPL:
- The output has the following format:
  #+begin_example
  ================================================================================
  T/V                N    NB     P     Q               Time                 Gflops
  --------------------------------------------------------------------------------
  WR00L2L2        2000   120     1     1               3.17              1.683e+00
  #+end_example
- Sometimes, the last line is missing, so we do not have any informaiton on time and flops.
- Quite often it is present, but with wrong values: the time is 0.00 and the Gflops are absurdly high (e.g. 2.302e+03
  Gflops for a cluster made of 96 machines of 1 Gflops). It may come from an erroneous measure of the time.
- For instance, with the script of commit =dbdfeabbef3f90a3d4e2ecfbe5e8f505738cac23=, the following command line:
  =./run_measures.py --global_csv /tmp/bla --nb_runs 10 --size 5000 --nb_proc 64 --fat_tree "2;24,48;1,24;1,1"
  --experiment HPL=
  + It may get this output in one experiment:
  #+begin_example
    ================================================================================
    T/V                N    NB     P     Q               Time                 Gflops
    --------------------------------------------------------------------------------
    WR00L2L2        5000   120     8     8               0.00              1.108e+05
  #+end_example
  + And this output in another one:
  #+begin_example
    ================================================================================
    T/V                N    NB     P     Q               Time                 Gflops
    --------------------------------------------------------------------------------
    WR00L2L2        5000   120     8     8               5.35              1.560e+01
  #+end_example
  Note that, for the two experiments, *nothing* has changed. The file =HPL.dat= is the same, the number of processes given
  to the option =-np= is the same, the topology file and the host file are the same.
*** 2017-02-20 Monday
**** Keep investigating on the HPL anomaly
**** Found the issue with HPL                               :SMPI:BUG:HPL:
- Debugging with Christian, to understand what was going on.
- This was a concurrency issue. The private variables of the processes were in fact not private. This caused two
  processes to write a same variable, which led to an inconsistent value when measuring time.
- The function is =HPL_ptimer=, in file =testing/ptest/HPL_pdtest.c=.
- When using simgrid, need to use option =--cfg=smpi/privatize-global-variables:yes= to fix this.
- Used a tool to search for a word, looks nice: =cg= and =vg= (package =cgvg=).
- Another nice thing: =ctags= (command =ctags --fields=+l -R -f ./ctags src testing=).
*** 2017-02-21 Tuesday
**** Test the experiment script for HPL                  :EXPERIMENTS:HPL:
- It seems to work well, the bug is fixed.
- Scalability issue. Testing for a size of 20k already takes a lot of time, and it is still too small to have a good
  efficiency with 1000 processes (performances are worse than with 100 processes).
- Definitely need to use SMPI optimizations if we want to do anything with HPL.
**** Re-do experiments with matrix product 
- Stuck with HPL...
- We also output the speed of the computation, in Gflops (this is redondant with the time, but we can use it for
  comparison with other algorithms like HPL).
- The plot looks nice, but nothing new.
**** Work on the drawing of fat-trees
- Generate all nodes and edges of a fat-tree.
- No drawing yet.
- Will try to output Tikz code.
**** DONE Look at where to put SMPI macros in HPL, with Christian
:LOGBOOK:
- State "DONE"       from "TODO"       [2017-02-22 Thu 13:17]
- State "TODO"       from              [2017-02-21 Tue 15:03]
:END:
- Have a look at a trace, to see where most of the time is spent.
**** Keep working on the drawing of fat-trees.
- Now produce working Tikz code.
- Figure quickly becomes unreadable for large fat-trees (not surprising).
*** 2017-02-22 Wednesday
**** Terminate the work on fat-tree drawing                       :PYTHON:
- We can now do =./draw_topo.py bla.pdf "2;8,16;1,1:8;1,1" "2;4,8;1,1:4;1,1"= to draw all the fat-trees in the file
  =bla.pdf=. Very useful to visualize the differences between the trees.
- No limit on the fat-tree size, they should fit on the pdf (a very large page is generated, then cropped to the right
  dimension). However, a large fat-tree may not be very readable.
**** Tried to move the SMPI_SAMPLE of the matrix product
- Cannot use one SMPI_SAMPLE per loop (don’t know why, but it seems to be forbidden).
- It was used for the outer loop. Tried the inner loops, but performances were greatly degraded (about \times50 in simulation time).
- Reverting the change.
**** DONE Cannot use more than 1024 processes with Simgrid (need to fix) :SMPI:BUG:
:LOGBOOK:
- State "DONE"       from "TODO"       [2017-02-23 Thu 10:20]
- State "TODO"       from              [2017-02-22 Wed 14:12]
:END:
- The =open()= system call fails with =EMFILE= error code.
- It used to work, don’t understand what changed in the meantime.
**** Talk with Christian about SMPI optimizations in HPL :PERFORMANCE:HPL:
- He gave me a trace of HPL execution obtained with Simgrid.
- The parts taking most of the time are the following:
  #+begin_example
  50        /home/cheinrich/src/hpl-2.2/src/pgesv/hpl_rollt.c       242          /home/cheinrich/src/hpl-2.2/src/comm/hpl_recv.c     136 190.785263    498
  51        /home/cheinrich/src/hpl-2.2/src/pgesv/hpl_rollt.c       242          /home/cheinrich/src/hpl-2.2/src/comm/hpl_sdrv.c     180 372.272945    996
  52        /home/cheinrich/src/hpl-2.2/src/pgesv/hpl_rollt.c       242          /home/cheinrich/src/hpl-2.2/src/comm/hpl_send.c     133 179.711679    498
  #+end_example
**** Let’s track these piece of code                     :PERFORMANCE:HPL:
- =HPL_rollT.c= has only one function: =HPL_rollT=.
- This function is called only once: at the end of function =HPL_pdlaswp01T= (eponym file).
- This function is called once in function =HPL_pduptateNT= and once in function =HPL_pdupdateTT= (eponym files). There are
  very few differences between these two functions (4 line changes are relevant, which are small variations in the
  arguments of a function, =HPL_dtrsm=). These files have 443 lines: this is a huge copy-paste, very dirty.
- A candidate for the long function we are looking for is =HPL_dlaswp10N= (found by Christian). Has two nested loops.
  This function is also a good candidate for the most terrible piece of code ever written.
- Added a =SMPI_SAMPLE_GLOBAL= after the outer loop, did not reduce the simulation time. Also tried to remove the whole
  code of the function, did not reduce the simulation time either. So we can say this function is not our big consummer.
- Functions =HPL_recv= and =HPL_sdrv= are both called *only* in =HPL_pdmxswp= and =HPL_pdlaswp00N=.
- Function =HPL_pdlaswp00n= is used only in =HPL_pdupdateTN= and =HPL_pdupdateNN=, which are nearly identical. These two
  functions are then used in the =testing= folder, with something like =algo.upfun = HPL_pdupdateNN=. Might be hard to track...
- Function =HPL_pdmxswp= is used in =HPL_pdpancrT=, =HPL_pdpanllT=, =HPL_pdpanllN=, =HPL_pdpanlT=, =HPL_pdpanrlN=,
  =HPL_pdpancrN=. These functions are used in the =testing= folder, with something like =algo.pffun = HPL_pdpancrN=.
- Trying to put some printf.
  We use the command:
  #+begin_src sh
  smpirun --cfg=smpi/bcast:mpich --cfg=smpi/running-power:6217956542.969 --cfg=smpi/display-timing:yes
  --cfg=smpi/privatize-global-variables:yes -np 16 -hostfile ../../../small_tests/hostfile_64.txt -platform
  ../../../small_tests/cluster_fat_tree_64.xml ./xhpl
  #+end_src
  + Function =HPLpdupdateNN= never used.
  + Function =HPLpdupdateTN= never user.
  + Thus, function =HPL_pdlaswp00n= also never used (verified with printf in this function).
  + Function =HPL_pdmxswp= is used and takes a significant (albeit not huge) amount of time (about 2 seconds when the
    total time is 41 seconds (virtual time)).
*** 2017-02-23 Thursday
**** Try to increase the file limit                             :SMPI:BUG:
- First try, following [[http://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user][this question]] and [[http://stackoverflow.com/questions/21515463/how-to-increase-maximum-file-open-limit-ulimit-in-ubuntu][this question]] from Stackoverflow.
  + Added the following to =/etc/security/limits.conf=:
    #+begin_example
     *     soft    nofile          40000
     *     hard    nofile          40000
    #+end_example
  + Added the following to =/etc/pam.d/common-session=:
    #+begin_example
     session required pam_limits.so
    #+end_example
  + Rebooting.
- Success, =ulimit -Sn= shows =40000= and we can now run experiments with more than 1024 processes.
**** Keep tracking the time consumming piece of code in HPL :PERFORMANCE:HPL:
- Function =HPL_pdmxswp= is used in some functions which are chosen with =algo.pffun= (see above).
- They are then used (through a call to algo.pffun) in functions =HPL_pdrpancrN=, =HPL_pdrpanrlN=, =HPL_pdrpanllN=,
  =HPL_pdpanrlT=, =HPL_pdrpancrT= and =HPL_pdranllT=.
- Again, these functions are not used directly in =src=, there is something like =algo.rffun = HPL_pdrpancrT= in the =testing= folder.
- This =rffun= is used only once, in =HPL_pdfact=.
- Function =HPL_pdfact= takes between 2.5 and 2.8 seconds when the total time is 41 seconds (virtual time). This time includes
  the time spent in =HPL_pdmxswp=.
- Function =HPL_pdffact= is used in functions =HPL_pdgesvK1=, =HPL_pdgesvK2= and =HPL_pdgesv0=. These functions are then called
  in =HPL_pdgesv=.
- Function =HPL_pdgesv= takes a time of about 3 seconds when the total time is 41 seconds (virtual time).
- Strange thing. Deleting the content of this function gives a very short run-time. Maybe the way I measured time (using
  =MPI_WTIME=) is not consistent with the way HPL measure time.
- Identified the long loop in =HPL_pdgesv0=. But cannot put a =SMPI_SAMPLE= here, there are calls to MPI primitives in the block.
- Found the right function to measure time: use =HPL_timer_walltime=, not =MPI_Wtime=.
- Instrumented the code of =HPL_pdgesv0= to have an idea of what takes time. Measures are taken with =HPL_timer_walltime=.
  What takes time is the part “factor and broadcast current panel” in the loop.
  Within this part, the call to =HPL_bcast= and =HPL_pdupdate= are what take most of the (virtual) time. In an execution of
  40.96 seconds:
  #+begin_example
  pdfact = 2.907908, binit = 0.002633, bcast = 11.013843, bwait = 0.000669, pdupdate = 26.709408
  #+end_example
  Obviously there is nothing to do for the broadcast, but there may be hope for =pdupdate=.
- Several versions exist for this function:
  + =HPL_pdupdateTN=
  + =HPL_pdupdateNT=
  + =HPL_pdupdateTT=
  + =HPL_pdupdateNN=
  Only =HPL_pdupdateTT= seems to be used (with our settings).
  Removed body of function =HPL_pdupdateTT=, the simulation time becomes about 8 seconds (was 69 seconds).
- Might be tricky to optimize with SMPI macros, this function mixes computations and communications.
- Tried to insert a =return= line 208 (before comment “The panel has been forwarded at that point, finish the update”. The
  time is not impacted and the correction test are valid, so the part of the code after this point seems useless here.
  Verified by inserting a =printf=, this paprt is never executed.
- Line 143 is executed (just after comment “1 x Q case”).
- Adding a =return= statement line 136 (just before comment “Enable/disable th column panel probing mechanism”) gives a
  simulation time of 8 seconds.
  Same thing line 140, after the broadcast.
- The =if= block of lines 143-258 is never executed in our settings. Explain why acting on line 208 did not have any effect.
- Adding a =return= statement line 358 (just before comment “The panel has been forwarded at that point, finish the
  update”) gives a simulation time of 9.7 seconds.
- The =if= block of lines 360-414 seem to be always executed. The =if= block of lines 366-390 is executed sometimes, but not
  always.
  In this block, we execute the =#else= part of the =#ifdef=.
- In this block, removing the call to =HPL_dgemm= reduce a lot the simulation time (from 68s to 13s).
- Several definitions exist for =HPL_dgemm=: there is an implementation in =src/blas/HPL_dgemm.c=, but also a =#define
  HPL_dgemm cblas_dgemm= in =include/hpl_blas.h=.
- Can disable this =#define= by removing the line =HPL_OPTS = -DHPL_CALL_CBLAS= in the file =Make.SMPI=.
  Then, =HPL_dgemm= is executed, but not the others (=HPL_dgemm0=, =HPL_dgemmTT=, =HPL_dgemmTN=, =HPL_dgemmNT=, =HPL_dgemmNN=). It
  seems that =HPL_dgemm= can call =HPL_dgemm0= which can itself call the four others, but this only happens when
  =HPL_CALL_VSIPL= is defined.
- In fact, there is maybe no need to insert the =SMPI_SAMPLE= macro in =dgemm= function. We can put it inside
  =HPL_pdupdateTT=. For instance, line 360, just above the big =if= block.  However, this performs realy badly. With
  =SMPI_SAMPLE_GLOBAL(10, 0.1)=, the real time becomes about 10 seconds (speedup of \times4) but the virtual time becomes about
  90 seconds (\times2 error). If we increase one of the two numbers, the real times quickly become as large as it was before.
  Same thing with =SMPI_SAMPLE_LOCAL=.  Maybe this code is too irregular? Or we should “zoom in” and insert the SMPI
  optimizations in =dgemm= (which is in an external library, so not that easy).
*** 2017-02-27 Monday
**** Try running matrix product experiment with big fat-trees   :SMPI:BUG:
- Run a medium number of processes on a big fat-tree.
  #+begin_src sh
  ./run_measures.py --global_csv big_global.csv --local_csv big_local.csv --nb_runs 3 --size 9300 --nb_proc 961
  --fat_tree "3;24,24,48;1,24,1:24;1,1,1" --experiment matrix_product
  #+end_src sh
  Seems to work properly, one CPU core is quickly loaded at 100% and one experiment approximately takes two minutes.
- Try a larger number of processes with the same topology and the same matrix size.
  #+begin_src sh
  ./run_measures.py --global_csv big_global.csv --local_csv big_local.csv --nb_runs 3 --size 9300 --nb_proc 8649
  --fat_tree "3;24,24,48;1,24,1:24;1,1,1" --experiment matrix_product
  #+end_src
  The CPU is loaded at about 3% for quite a long time with the script =smpirun=. It finally launches =matmul= and becomes
  loaded at 100%. Then it quickly terminates with a non-null exit code: =Could not map fd 8652 with size 80000: Cannot
  allocate memory=. The memory consumption was only 3% of the total memory, this is strange.
  This happens in function =shm_map=, called by =SMPI_SHARED_MALLOC=.
- Retrying the same command, with =malloc= instead of =SMPI_SHARED_MALLOC= and =free= instead of =SMPI_SHARED_FREE=.
  As expected, larger memory consumption (10.9% of total memory). There is no error this time. The first experiment
  terminates in about 20min. For the record, it achieved 1525 Gflops, with communication time and computation time of
  approximately 0.48 seconds.
- Revert the changes to get back =SMPI_SHARED= macros. Retry to run =smpirun= with the same settings, except the option
  =--cfg=smpi/privatize-global-variables:yes= which is not passed here.  No error either this time, run for 13
  minutes. Also a large memory consumption (13.5%), maybe the 3% we observed was not the final memory consumption, since
  the process exited with an error?
- Remark: for matrix product, there is no global variable. So maybe we can safely remove this option in this case?
  This does not solve the problem since we need it for HPL.
- Try the initial command with a smaller matrix size (size=93, i.e. all processes have a sub-matrix of size 1\times1). Observed the same error.
- Also try to reproduce this with HPL, with this command:
  #+begin_src sh
  ./run_measures.py --global_csv big_global.csv --nb_runs 3 --size 5000 --nb_proc 8649 --fat_tree
  "3;24,24,48;1,24,1:24;1,1,1" --experiment HPL
  #+end_src
  Not any error, although we have a memory consumption of 71.2%.
- Try the initial command, still with a size of 93, but commenting the call to =matrix_product= in =matmul.c=. Thus, there
  is no allocation of temporary buffers, only the initial matrices (3 allocations instead of 5). No error.
- Same thing, with the call to =matrix_product= uncommented, but a =return= statement placed just after the temporary
  buffers allocations. We get the =mmap= error.
- Create a MWE from this, called =mmap_error.c=.
**** Work on a MWE for the mmap error                           :SMPI:BUG:
- File =mmap_error.c= is a MWE for the =mmap= error. It consists in 5 calls to =SMPI_SHARED_MALLOC= with a size of 1, we
  launch it with 8652 processes.
  We also get an error if we do 100k calls to =SMPI_SHARED_MALLOC= with only one process. The total number of calls to
  this macro seem to be the issue. We get the error with or without the option =smpi/privatize-global-variables:yes=.
- The following file =mmap_error.c=:
  #+begin_src c
  #include <stdio.h>
  #include <mpi.h>

  #define N 65471

  int main(int argc, char *argv[]) {

      MPI_Init(&argc, &argv);

      for(int i = 0; i < N; i++) {
          float *a = SMPI_SHARED_MALLOC(1);
      }

      MPI_Barrier(MPI_COMM_WORLD);
      printf("Success\n");
      MPI_Finalize();
      return 0;
  }
  #+end_src
  With the following command (commit =8eb0cf0b6993e174df58607e9492a134b85a4669= of Simgrid):
  #+begin_src sh
  smpicc -O4 mmap_error.c -o mmap_error
  smpirun -np 1 -hostfile hostfile_64.txt -platform cluster_fat_tree_64.xml ./mmap_error
  #+end_src
  Yields an error. Note that the host and topology files are irrelevant here.
  + For =N<65471=, we have no error (=Success= is printed).
  + For =N>65471=, we have the error =Could not map fd 3 with size 1: Cannot allocate memory=.
  + For =N=65471=, we have the error =Memory callocation of 524288 bytes failed=.
- Retried with latest version of Simgrid (commit =c8db21208f3436c35d3fdf5a875a0059719bff43=).  Now have the
  message:
  #+begin_example
  Could not map folded virtual memory (Cannot allocate memory). Do you perhaps need to increase
  the STARPU_MALLOC_SIMULATION_FOLD environment variable or the sysctl vm.max_map_count?
  #+end_example
  Found the issue:
  #+begin_src sh
  $ sysctl vm.max_map_count
  vm.max_map_count = 65530
  #+end_src
  To modify the value of a =sysctl= variable, follow [[https://www.cyberciti.biz/faq/howto-set-sysctl-variables/][this link]].
  Temporary fix:
  #+begin_src sh
  sudo sysctl -w vm.max_map_count=100000
  #+end_src
**** Run the matrix product experiment with 8649 processes
- Using the command:
  #+begin_src sh
  ./run_measures.py --global_csv big_global.csv --local_csv big_local.csv --nb_runs 3 --size 9300 --nb_proc 8649
  --fat_tree "3;24,24,48;1,24,1:24;1,1,1" --experiment matrix_product
  #+end_src
- The experiments are very long, about 30 minutes. The code is already optimized a lot (SMPI macros, no initialization
  of the matrices), a large part of this time is spent outside of the application, so there is not much hope to run it
  faster without modifying Simgrid.
- This shows that we *really* need to optimize HPL if we want to run it with a large number of processes.
- Anyway, without SMPI macros, every floating-point operation of the application is actually performed. Thus, if we are
  simulating a computation made on a 1000 Gflops cluster, using a 1 Gflops laptop, the simulation should take *at least*
  1000 times longer than the same computation on a real 1000 Gflops cluster.
- First results show no large difference in the total time for small or large number of roots. The communication time is
  about twice as large as the computation time, so maybe we should take a larger matrix. When we had 961 processes, each
  one had a sub-matrix of size 300\times300. With 8649 processes, they have a sub-matrix of size 100\times100.
  Problem: if we want to get back to the 300\times300 sub-matrices, we need to multiply the size by 3 and thus the memory
  consumption by 9. It was already about 25%, so not feasible on this laptop. But this is strange, we should have the
  memory of only one process and we successfully ran 300\times300 sub-matrices, need to check.
*** 2017-02-28 Tuesday
**** Other benchmarks on Simgrid                        :SMPI:EXPERIMENTS:
- The paper “Simulating MPI application: the SMPI approach” uses the benchmark [[https://www.nas.nasa.gov/publications/npb.html][NAS EP]] to demonstrate the scalability of
  SMPI. With SMPI optimizations, they ran it with 16384 processes in 200 to 400 seconds (depending on the topology).
  Where is the code for this?
  + Found an [[https://github.com/sbadia/simgrid/tree/master/examples/smpi/NAS][old repository]]. Not clear if it is relevant.
  + Also a (shorter) version in the [[https://github.com/simgrid/simgrid/tree/master/examples/smpi/NAS][official Simgrid repository]].
    Executable located in =simgrid/build/examples/smpi/NAS/=.
    Launch with two arguments: number of processes (don’t know what it does, we already have =-np= option given to
    =smpirun=) and the class to use (S, W, A, B, C, D, E, F).
- The NAS EP benchmark from Simgrid repository seems promising. Added a new class to have a larger problem (maybe we
  could instead give the size as an argument). With a large enough size, we can go to about 3.5 Gflops per process,
  i.e. an efficiency of 3.5 (recall that we use 1 Gflops nodes). It seems large, is it normal?
- Longer than the matrix product, 745 seconds for 1152 processes and class F (custom class with m=42). Only 93 seconds
  were spent in the application, so the code is already correctly optimized (one call to =SMPI_SAMPLE_GLOBAL=).
- Apparently not impacted by a tapered fat tree. Roughly the same speed for =2;24,48;1,24;1,1= and =2;24,48;1,1;1,1=, 1152
  processes and class F: about 3.5 Gflops. The application is made of a computation followed by three =MPI_Allreduce=
  of only one =double=, so very few communications (hence the name “embarassingly parallel”).
**** Talk with Christian about benchmarks
- Get an access to grid 5000.
- Profile the code, with something like smpirun -wrapper “valgrind <param>”.
- To use SMPI macros, run the =HPL_dgemm= implemented in HPL, not the one from the external library.
** 2017-03 March
*** 2017-03-01 Wednesday
**** Trying to use HPL without external BLAS library                 :HPL:
- Failed.
- It seems that three options are available for compilation, according to [[http://www.netlib.org/benchmark/hpl/software.html][this page]]:
  + BLAS Fortran 77 interface (the default),
  + BLAS C interface (option =-DHPL_CALL_CBLAS=),
  + VSIPL library (option =-DHPL_CALL_VSIPL=).
- We currently use the C interface, which rely on an external library (e.g. Atlas).
- There is an implementation of =HPL_dgemm= in HPL, but it seems to need either code from Fortran 77 or from VSIPL.
- According to the [[http://www.netlib.org/benchmark/hpl/][HPL homepage]]:
  #+begin_example
  The HPL software package requires the availibility on your system of an implementation of the Message Passing
  Interface MPI (1.1 compliant). An implementation of either the Basic Linear Algebra Subprograms BLAS or the Vector
  Signal Image Processing Library VSIPL is also needed. Machine-specific as well as generic implementations of MPI, the
  BLAS and VSIPL are available for a large variety of systems.
  #+end_example
  So it seems hopeless to get rid of a BLAS library.
**** Idea: trace calls to =HPL_dgemm= (Arnaud’s idea)     :SMPI:TRACING:HPL:
- To do so, surround them by calls to trivial MPI primitives (e.g. =MPI_Initialized=). For instance:
  #+begin_src c
  #define HPL_dgemm(...) ({int simgrid_test; MPI_Initialized(&simgrid_test); cblas_dgemm(__VA_ARGS__);\
  MPI_Initialized(&simgrid_test);})
  #+end_src
- Then, trace the execution (output in =/tmp/trace=):
  #+begin_src sh
  smpirun -trace -trace-file /tmp/trace --cfg=smpi/trace-call-location:1 --cfg=smpi/bcast:mpich\
  --cfg=smpi/running-power:6217956542.969 --cfg=smpi/display-timing:yes --cfg=smpi/privatize-global-variables:yes -np 16\
  -hostfile ../../../small_tests/hostfile_64.txt -platform ../../../small_tests/cluster_fat_tree_64.xml ./xhpl\
  #+end_src
- Finally, dump this trace in CSV format:
  #+begin_src sh
  pj_dump --user-defined --ignore-incomplete-links trace > trace.dump
  #+end_src
- Did not work, no =MPI_Initialized= in the trace. In fact, this primitive is currently not traced. We could modify SMPI
  to achieve this behavior, or use another MPI primitive that is already traced.
*** 2017-03-02 Thursday
**** Keep trying to trace calls to =HPL_dgemm=            :SMPI:TRACING:HPL:
- A MPI primitive is traced \Leftrightarrow the functions =new_pajePushState= and =new_pagePopState= are called (not sure, this is an
  intuition).
- This function is not called by =MPI_Initialized=, or =MPI_Wtime=.
- It is called by =MPI_Test=, but only if the =MPI_Request= object passed as argument is non-null, so we would need to do a
  fake asynchronous communication just before, which is probably not a good idea.
- Anyway, it looks dirty to use a MPI primitive like this. Wouldn’t it be better to have a custom no-op primitive that
  force the introduction of a trace entry? For instance, something like
  #+begin_src c
  SMPI_Trace {
      HPL_dgemm();
  }
  #+end_src
  or like
  #+begin_src c
  SMPI_BeginTrace();
  HPL_dgemm();
  SMPI_EndTrace();
  #+end_src
- Every MPI primitive is defined by a =#define= with a call to =smpi_trace_set_call_location= followed by a call to the
  function. For instance:
  #+begin_src c
  #define MPI_Test(...) ({ smpi_trace_set_call_location(__FILE__,__LINE__); MPI_Test(__VA_ARGS__); })
  #+end_src
  However, this only record the file name and the line number, I do not think it dumps anything in the trace.
**** Arnaud’s keynote: reproducible research                     :MEETING:
- Intro: article we had in exam, “Is everything we eat associated with cancer?”.
- In most articles, we can read formulae and trust results, but much less often reproduce the results.
- Reproducibility crisis, several scandals with falsified results (intentionnaly or not).
- Video: Brendan Gregg, shouting  in the data center.
**** Discussion with Arnaud                                      :MEETING:
- Regarding the matrix product:
  + Compare the (tapered) fat-tree with “perfect” topology (cluster with no latency and infinit bandwidth).
  + Run it with larger matrices for the same amount of processes. Do not aim at spending as much time in communication
    than computation. We want the communication time to become nearly negligible. In practices, users of a supercomputer
    try to fill the memory of their nodes.
- Regarding HPL:
  + As discussed yesterday, we want to trace the calls to =HPL_dgemm= by putting calls to a MPI primitive just before and after.
  + The short-time goal is to have an idea of the behavior of HPL regarding this function. Are there a lot of different
    calls to =HPL_dgemm= coming from different locations? Do these calls always take the same amount of time (i.e. do we
    always multiply matrices of the same size)?
  + It seems that there is some variability in the duration of =HPL_dgemm= (to be verified with the trace). If HPL really
    use the function to multiply matrices of different size, we cannot do something like
    =SMPI_SAMPLE(){HPL_dgemm()}=, it will not be precise. What we could do however is to generalize =SMPI_SAMPLE=: we could
    parametrize it by a number, representing the size of the problem that is sampled. If this size is always the same,
    then we could do what we are doing now, simply take the average. If this size changes over time, we could do
    something more elaborated for the prediction, like a linear regression.
  + Using MPI functions like =MPI_Test= is not very “clean”, but we do not want to waste time on this currently, so we stick
    with existing MPI primitives. We could try to change this in the future.
  + It is always safe to call =smpi_process_index=. Thus, we could modify =PMPI_Test= to call =TRACE_smpi_testing= functions
    even when the given request is =NULL=.
*** 2017-03-03 Friday
**** Tracing calls to =HPL_dgemm= :SMPI:C:PYTHON:R:EXPERIMENTS:TRACING:PERFORMANCE:HPL:
- Modification of the function =PMPI_Test= of Simgrid so that =MPI_Test= is traced even when the =MPI_Request= handle is
  =NULL=. To do that, we need to get the rank of the process, with =smpi_process_index=. The value returned is always 0 in
  this case. This is a problem, since we could not distinguish between calls to =MPI_Test= from different processes, thus
  it would be impossible to measure time. Reverting the changes.
- To get a non-null =MPI_Request=, did a =MPI_Isend= followed by a =MPI_Recv=:
  #+begin_src c
  #define    HPL_dgemm(...)      ({\
    int my_rank, buff=0;\
    MPI_Request request;\
    MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);\
    MPI_Isend(&buff, 1, MPI_INT, my_rank, 0, MPI_COMM_WORLD, &request);\
    MPI_Recv(&buff, 1, MPI_INT, my_rank, 0, MPI_COMM_WORLD, NULL);\
    MPI_Wait(&request, MPI_STATUS_IGNORE);\
    cblas_dgemm(__VA_ARGS__);\
    MPI_Isend(&buff, 1, MPI_INT, my_rank, 0, MPI_COMM_WORLD, &request);\
    MPI_Recv(&buff, 1, MPI_INT, my_rank, 0, MPI_COMM_WORLD, NULL);\
    MPI_Wait(&request, MPI_STATUS_IGNORE);\
  })
  #+end_src
- Forget this. HPL was executed with only one process (=-np 16= but P and Q were 1 in =HPL.dat=). This is why we only had a
  rank =0= when giving =NULL= as =MPI_Request=. Let’s revert this and use simple =MPI_Test= with =NULL=.
- Calls to =MPI_Test= seem to be correctly traced, but the post-processing of the trace with =pj_dump= crashes:
  #+begin_example
  terminate called after throwing an instance of 'std::out_of_range'
  what():  vector::_M_range_check: __n (which is 4) >= this->size() (which is 4)
  #+end_example
  It also happened with the more complex piece of code that is shown above (with =MPI_Test= instead of =MPI_Wait=).
  Reverting again, to use the bigger piece of code above.
- Now, the call to =pj_dump= succeeds, and we can see calls to =MPI_Wait= in the trace.
- The call to =smpirun= was:
#+begin_src sh
smpirun -trace -trace-file /tmp/trace --cfg=smpi/trace-call-location:1 --cfg=smpi/bcast:mpich\
--cfg=smpi/running-power:6217956542.969 --cfg=smpi/display-timing:yes --cfg=smpi/privatize-global-variables:yes -np 16\
-hostfile ../../../small_tests/hostfile_64.txt -platform ../../../small_tests/cluster_fat_tree_64.xml ./xhpl
#+end_src
- Processing of the trace.
  Clean the file:
#+begin_src sh
pj_dump --user-defined --ignore-incomplete-links /tmp/trace > /tmp/trace.csv
grep "State," /tmp/trace.csv | grep MPI_Wait | sed -e 's/()//' -e 's/MPI_STATE, //ig'  -e 's/State, //ig' -e 's/rank-//' -e\
's/PMPI_/MPI_/' | grep MPI_  | tr 'A-Z' 'a-z' > /tmp/trace_processed.csv
#+end_src

Clean the paths:
#+begin_src python
import re
reg = re.compile('((?:[^/])*)(?:/[a-zA-Z0-9_-]*)*((?:/hpl-2.2(?:/[a-zA-Z0-9_-]*)*).*)')
def process(in_file, out_file):
    with open(in_file, 'r') as in_f:
        with open(out_file, 'w') as out_f:
            for line in in_f:
                match = reg.match(line)
                out_f.write('%s%s\n' % (match.group(1), match.group(2)))
process('/tmp/trace_processed.csv', '/tmp/trace_cleaned.csv')
 #+end_src

 #+RESULTS:

#+begin_src R :results output :session *R* :exports both
df <- read.csv("/tmp/trace_cleaned.csv", header=F, strip.white=T, sep=",");
names(df) = c("rank", "start", "end", "duration", "level", "state", "Filename", "Linenumber");
head(df)
#+end_src

#+RESULTS:
#+begin_example
  rank    start      end duration level    state
1    8 2.743960 2.743960        0     0 mpi_wait
2    8 2.744005 2.744005        0     0 mpi_wait
3    8 2.744005 2.744005        0     0 mpi_wait
4    8 2.744005 2.744005        0     0 mpi_wait
5    8 2.744005 2.744005        0     0 mpi_wait
6    8 2.744005 2.744005        0     0 mpi_wait
                            Filename Linenumber
1 /hpl-2.2/src/pfact/hpl_pdrpanllt.c        222
2 /hpl-2.2/src/pfact/hpl_pdrpanllt.c        222
3 /hpl-2.2/src/pfact/hpl_pdrpanllt.c        222
4 /hpl-2.2/src/pfact/hpl_pdrpanllt.c        222
5 /hpl-2.2/src/pfact/hpl_pdrpanllt.c        222
6 /hpl-2.2/src/pfact/hpl_pdrpanllt.c        222
#+end_example

#+BEGIN_SRC R :results output :session *R* :exports both
duration_compute = function(df) {
    ndf = data.frame();
    df = df[with(df,order(rank,start)),];
    #origin = unique(df$origin)
    for(i in (sort(unique(df$rank)))) {
        start     = df[df$rank==i,]$start;
        end       = df[df$rank==i,]$end;
        l         = length(end);
        end       = c(0,end[1:(l-1)]); # Computation starts at time 0

        startline = c(0, df[df$rank==i,]$Linenumber[1:(l-1)]);
        startfile = c("", as.character(df[df$rank==i,]$Filename[1:(l-1)]));
        endline   = df[df$rank==i,]$Linenumber;
        endfile   = df[df$rank==i,]$Filename;

        ndf       = rbind(ndf, data.frame(rank=i, start=end, end=start,
            duration=start-end, state="Computing",
            startline=startline, startfile=startfile, endline=endline,
            endfile=endfile));
    }
    ndf$idx = 1:length(ndf$duration)
    ndf;
}
durations = duration_compute(df);
durations = durations[durations["startfile"] == "/hpl-2.2/src/pgesv/hpl_pdupdatett.c" & durations["endfile"] == "/hpl-2.2/src/pgesv/hpl_pdupdatett.c" &
    durations["startline"] == durations["endline"],]
#+END_SRC

#+RESULTS:

#+begin_src R :results output :session *R* :exports both
library(dplyr)
options(width=200)
group_by(durations, startfile, startline, endfile, endline) %>% summarise(duration=sum(duration), count=n()) %>% as.data.frame()
#+end_src

#+RESULTS:
:                             startfile startline                             endfile endline  duration count
: 1 /hpl-2.2/src/pgesv/hpl_pdupdatett.c       387 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     387  683.6677   659
: 2 /hpl-2.2/src/pgesv/hpl_pdupdatett.c       411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     411 2115.8129  1977

#+begin_src R :file images/trace1_16.png :results value graphics :session *R* :exports both
library(ggplot2)
ggplot(durations, aes(x=idx, y=duration, color=factor(rank))) +
    geom_point(shape=1) + ggtitle("Durations of HPL_dgemm")
#+end_src

#+RESULTS:
[[file:images/trace1_16.png]]


#+begin_src R :file images/trace2_16.png :results value graphics :session *R* :exports both
ggplot(durations, aes(x=start, y=duration, color=factor(rank))) +
    geom_point(shape=1) + ggtitle("Durations of HPL_dgemm")
#+end_src

#+RESULTS:
[[file:images/trace2_16.png]]

Same results, with four processes:

[[file:images/trace1_4.png]]

[[file:images/trace2_4.png]]

**** Seminaire                                                   :MEETING:
On the asymptotic behavior of the price of anarchy, how bad is selfish routing in highly congested networks?
- For instance, cars on a road make their own routing decisions, hence the “selfish” routing. This is not optimal (in
  comparison with a centralized routing).
**** Discussion with Arnaud & Christian                          :MEETING:
- According to the plots, it is impossible to use =SMPI_SAMPLE= as is, since there are huge variations on the duration of =HPL_dgemm=.
- The idea of a parametrized =SMPI_SAMPLE= is also not super. Every process does consecutive calls to =HPL_dgemm=, each call
  being shorter than the previous ones. So we would still have to compute expensive calls.
- A long term idea may be to have a “SimBLAS” library, that simulates the calls to =HPL_dgemm= (and other BLAS
  primitives). Christian will work on this.
- Answers to all my questions from the paper readings.
**** TODO New tasks [3/4] 
:LOGBOOK:
- State "TODO"       from              [2017-03-03 Fri 17:43]
:END:
- [X] Do the linear regression by hand, off-line. Output the sizes of the matrices given to =HPL_dgemm= (with =printf=).
- [X] Register on Grid5000. Compile HPL on one Grid5000 machine.
- [X] Try to run HPL with a very large matrix, by using =SMPI_SHARED_MALLOC= (thus look at where all the allocations of
  matrices are done).
- [ ] Have a look at the code of Simgrid, in particular the routing in fat-trees.
*** 2017-03-06 Monday
**** Output the matrix sizes                        :C:PYTHON:TRACING:HPL:
- Add the following before the relevant calls to =HPL_dgemm=:
  #+begin_src c
  printf("line=%d rank=%d m=%d n=%d k=%d\n", __LINE__+3, rank, mp, nn, jb);
  #+end_src
  Then, run HPL by redirecting =stdout= to =/tmp/output=.
- Process the output, to get a CSV file:
#+begin_src python
import re
import csv
reg = re.compile('line=([0-9]+) rank=([0-9]+) m=([0-9]+) n=([0-9]+) k=([0-9]+)')

def process(in_file, out_file):
    with open(in_file, 'r') as in_f:
        with open(out_file, 'w') as out_f:
            csv_writer = csv.writer(out_f)
            csv_writer.writerow(('line', 'rank', 'n', 'm', 'k'))
            for line in in_f:
                match = reg.match(line)
                if match is not None:
                    csv_writer.writerow(tuple(match.group(i) for i in range(1,6)))
process('/tmp/output', '/tmp/sizes.csv')
#+end_src

**** Merge the sizes with the durations        :R:EXPERIMENTS:PERFORMANCE:
- Run =smpirun= as stated above, then process the output and the trace as before.
- Process the data:
#+begin_src R :results output :session *R* :exports both
df <- read.csv("/tmp/trace_cleaned.csv", header=F, strip.white=T, sep=",");
names(df) = c("rank", "start", "end", "duration", "level", "state", "Filename", "Linenumber");
head(df)
#+end_src

#+RESULTS:
:   rank    start      end duration level    state                           Filename Linenumber
: 1    8 2.743960 2.743960        0     0 mpi_wait /hpl-2.2/src/pfact/hpl_pdrpanllt.c        222
: 2    8 2.744005 2.744005        0     0 mpi_wait /hpl-2.2/src/pfact/hpl_pdrpanllt.c        222
: 3    8 2.744005 2.744005        0     0 mpi_wait /hpl-2.2/src/pfact/hpl_pdrpanllt.c        222
: 4    8 2.744005 2.744005        0     0 mpi_wait /hpl-2.2/src/pfact/hpl_pdrpanllt.c        222
: 5    8 2.744005 2.744005        0     0 mpi_wait /hpl-2.2/src/pfact/hpl_pdrpanllt.c        222
: 6    8 2.744005 2.744005        0     0 mpi_wait /hpl-2.2/src/pfact/hpl_pdrpanllt.c        222

#+begin_src R :results output :session *R* :exports both
sizes <- read.csv("/tmp/sizes.csv");
head(sizes)
#+end_src

#+RESULTS:
:   line rank    n    m   k
: 1  411   12 4920 4920 120
: 2  387    0 4920 4920 120
: 3  411    8 5000 4920 120
: 4  411    4 5040 4920 120
: 5  411   13 4920 5040 120
: 6  387    1 4920 5040 120

#+begin_src R :results output :session *R* :exports both
durations = duration_compute(df); # same function as above
durations = durations[durations["startfile"] == "/hpl-2.2/src/pgesv/hpl_pdupdatett.c" & durations["endfile"] == "/hpl-2.2/src/pgesv/hpl_pdupdatett.c" &
    durations["startline"] == durations["endline"],]
head(durations)
#+end_src

#+RESULTS:
:     rank     start       end duration     state startline                           startfile endline                             endfile idx
: 481    0  3.153899  6.271075 3.117176 Computing       387 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     387 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 481
: 486    0  7.047247 10.063367 3.016120 Computing       411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 486
: 491    0 10.648367 13.716045 3.067678 Computing       411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 491
: 496    0 14.104534 17.155418 3.050884 Computing       411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 496
: 977    0 17.557080 20.430869 2.873789 Computing       387 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     387 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 977
: 982    0 21.104026 24.044767 2.940741 Computing       411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 982


#+begin_src R :results output :session *R* :exports both
insert_sizes = function(durations, sizes) {
    stopifnot(nrow(durations)==nrow(sizes))
    ndf = data.frame();
    for(i in (sort(unique(durations$rank)))) {
        tmp_dur = durations[durations$rank == i,]
        tmp_sizes = sizes[sizes$rank == i,]
        stopifnot(nrow(tmp_dur) == nrow(tmp_sizes))
        stopifnot(tmp_dur$startline == tmp_sizes$line)
        storage.mode(tmp_sizes$m) <- "double" # avoiding integer overflow when taking the product
        storage.mode(tmp_sizes$n) <- "double"
        storage.mode(tmp_sizes$k) <- "double"
        tmp_dur$m = tmp_sizes$m
        tmp_dur$n = tmp_sizes$n
        tmp_dur$k = tmp_sizes$k
        tmp_dur$size_product = tmp_sizes$m * tmp_sizes$n * tmp_sizes$k
        ndf = rbind(ndf, tmp_dur)
    }
    return(ndf);
}
#+end_src

#+RESULTS:

#+begin_src R :results output :session *R* :exports both
result = insert_sizes(durations, sizes)
head(result)
#+end_src

#+RESULTS:
:     rank     start       end duration     state startline                           startfile endline                             endfile idx    m    n   k size_product
: 481    0  3.153899  6.271075 3.117176 Computing       387 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     387 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 481 4920 4920 120   2904768000
: 486    0  7.047247 10.063367 3.016120 Computing       411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 486 4920 4920 120   2904768000
: 491    0 10.648367 13.716045 3.067678 Computing       411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 491 4920 4920 120   2904768000
: 496    0 14.104534 17.155418 3.050884 Computing       411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 496 4920 4920 120   2904768000
: 977    0 17.557080 20.430869 2.873789 Computing       387 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     387 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 977 4800 4800 120   2764800000
: 982    0 21.104026 24.044767 2.940741 Computing       411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 982 4800 4800 120   2764800000

**** Plot and linear regression                :R:EXPERIMENTS:PERFORMANCE:

#+begin_src R :file images/trace3_16.png :results value graphics :results output :session *R* :exports both
library(ggplot2)
ggplot(result, aes(x=size_product, y=duration, color=factor(rank))) +
    geom_point(shape=1) + ggtitle("Durations of HPL_dgemm as a function of the sizes")
#+end_src

#+RESULTS:
[[file:images/trace3_16.png]]


#+begin_src R :results output :session *R* :exports both
reg <- lm(duration~I(m*n*k), data=result)
summary(reg)
#+end_src

#+RESULTS:
#+begin_example

Call:
lm(formula = duration ~ I(m * n * k), data = result)

Residuals:
     Min       1Q   Median       3Q      Max 
-0.10066 -0.01700 -0.00085  0.00351  0.57745 

Coefficients:
               Estimate Std. Error  t value Pr(>|t|)    
(Intercept)  -2.476e-03  1.235e-03   -2.005   0.0451 *  
I(m * n * k)  1.062e-09  9.220e-13 1151.470   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.04205 on 2634 degrees of freedom
Multiple R-squared:  0.998,	Adjusted R-squared:  0.998 
F-statistic: 1.326e+06 on 1 and 2634 DF,  p-value: < 2.2e-16
#+end_example

#+begin_src R :file images/reg_16.png :results value graphics :results output :session *R* :exports both
layout(matrix(c(1,2,3,4),2,2))
plot(reg)
#+end_src

#+RESULTS:
[[file:images/reg_16.png]]
**** Comments on the linear regression                       :EXPERIMENTS:
- The plot of the duration as a function of =m*n*k= looks great. Maybe a bit of heteroscedasticity, but not so much. It is
  clearly linear.
- The linear regression however is not so good. We have a high R-squared (0.998), but the plots look bad. The
  residual-vs-fitted plot shows that the results are clearly heteroscedastic. The normal-QQ shows that they are not
  linear (in =m*n*k=) but rather exponential.
- The plot of the linear regression seems to contradict the first plot, this is strange.
**** Investigating the linear regression                               :C:
- We can print other relevant parameters of =HPL_dgemm=:
  #+begin_src c
  printf("line=%d rank=%d m=%d n=%d k=%d a=%f lead_A=%d lead_B=%d lead_C=%d\n", __LINE__+3,
    rank, mp, nn, jb, -HPL_rone, ldl2, LDU, lda);
  #+end_src
  Here, =a= is a scaling factor applied to the matrix, =lead_A=, =lead_B= and =lead_C= are the leading dimensions of matrices =A=, =B= and
  =C=.
  A sample of what we get is (only some lines are reported here):
  #+begin_example
  line=411 rank=2 m=2240 n=2160 k=120 a=-1.000000 lead_A=2480 lead_B=2160 lead_C=2480
  line=387 rank=3 m=1640 n=1641 k=120 a=-1.000000 lead_A=2480 lead_B=1641 lead_C=2480
  line=387 rank=2 m=680 n=720 k=120 a=-1.000000 lead_A=680 lead_B=720 lead_C=2480
  line=387 rank=2 m=200 n=240 k=120 a=-1.000000 lead_A=200 lead_B=240 lead_C=2480
  177 line=411 rank=1 m=480 n=441 k=120 a=-1.000000 lead_A=2520 lead_B=441 lead_C=2520
  #+end_example
  This trend seems to roughly repeat: =a= is always -1, =lead_C= is always either 2480 or 2520. For small enough values,
  =lead_A= is equal to =m= and =lead_C= is equal to =n=. For larger values, they are not equal anymore, but all are large.
  However, there are still some noticeable variations. For instance:
  #+begin_example
  line=387 rank=0 m=600 n=600 k=120 a=-1.000000 lead_A=2520 lead_B=600 lead_C=2520
  line=411 rank=0 m=600 n=600 k=120 a=-1.000000 lead_A=600 lead_B=600 lead_C=2520
  #+end_example
  In this last example, all parameters are equal, except =lead_A= which is more than four times larger in one case.
- A small leading dimension means a better locality and thus better performances. These differences in the leading
  dimensions could explain the non-linearity and the heteroscedasticity.
*** 2017-03-07 Tuesday
**** And the leading dimensions? :C:PYTHON:R:EXPERIMENTS:TRACING:PERFORMANCE:
- We have this =printf= before the calls to =HPL_dgemm= (same as before, except for the =a= that is removed):
  #+begin_src c
  printf("line=%d rank=%d m=%d n=%d k=%d lead_A=%d lead_B=%d lead_C=%d\n", __LINE__+3,
    rank, mp, nn, jb, ldl2, LDU, lda);
  #+end_src
- The trace is in the file =/tmp/trace=, we process it as before. The output is redirected in the file =/tmp/output=.
- Processing of the output:
#+begin_src python
import re
import csv
reg = re.compile('line=([0-9]+) rank=([0-9]+) m=([0-9]+) n=([0-9]+) k=([0-9]+) lead_A=([0-9]+) lead_B=([0-9]+) lead_C=([0-9]+)')

def process(in_file, out_file):
    with open(in_file, 'r') as in_f:
        with open(out_file, 'w') as out_f:
            csv_writer = csv.writer(out_f)
            csv_writer.writerow(('line', 'rank', 'n', 'm', 'k', 'lead_A', 'lead_B', 'lead_C'))
            for line in in_f:
                match = reg.match(line)
                if match is not None:
                    csv_writer.writerow(tuple(match.group(i) for i in range(1,9)))
process('/tmp/output', '/tmp/sizes.csv')
#+end_src

We have the =durations= dataframe, obtained as before:

#+begin_src R :results output :session *R* :exports both
head(durations)
#+end_src

#+RESULTS:
:     rank     start       end duration     state startline                           startfile endline                             endfile idx
: 481    0  4.111176  7.158459 3.047283 Computing       387 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     387 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 481
: 486    0  7.827329 10.848572 3.021243 Computing       411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 486
: 491    0 11.411456 14.445789 3.034333 Computing       411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 491
: 496    0 14.837377 17.868118 3.030741 Computing       411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 496
: 977    0 18.268679 21.142146 2.873467 Computing       387 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     387 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 977
: 982    0 21.809954 24.699182 2.889228 Computing       411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 982

Then we get the =sizes= dataframe:
#+begin_src R :results output :session *R* :exports both
sizes <- read.csv("/tmp/sizes.csv");
head(sizes)
#+end_src

#+RESULTS:
:   line rank    n    m   k lead_A lead_B lead_C
: 1  387    0 4920 4920 120   5040   4920   5040
: 2  411    8 5000 4920 120   5000   4920   5000
: 3  411    4 5040 4920 120   5040   4920   5040
: 4  411   12 4920 4920 120   4920   4920   4920
: 5  387    1 4920 5040 120   4920   5040   5040
: 6  411    5 5040 5040 120   5040   5040   5040

#+begin_src R :results output :session *R* :exports both
insert_sizes = function(durations, sizes) {
    stopifnot(nrow(durations)==nrow(sizes))
    ndf = data.frame();
    for(i in (sort(unique(durations$rank)))) {
        tmp_dur = durations[durations$rank == i,]
        tmp_sizes = sizes[sizes$rank == i,]
        stopifnot(nrow(tmp_dur) == nrow(tmp_sizes))
        stopifnot(tmp_dur$startline == tmp_sizes$line)
        storage.mode(tmp_sizes$m) <- "double" # avoiding integer overflow when taking the product
        storage.mode(tmp_sizes$n) <- "double"
        storage.mode(tmp_sizes$k) <- "double"
        storage.mode(tmp_sizes$lead_A) <- "double"
        storage.mode(tmp_sizes$lead_B) <- "double"
        storage.mode(tmp_sizes$lead_C) <- "double"
        tmp_dur$m = tmp_sizes$m
        tmp_dur$n = tmp_sizes$n
        tmp_dur$k = tmp_sizes$k
        tmp_dur$lead_A = tmp_sizes$lead_A
        tmp_dur$lead_B = tmp_sizes$lead_B
        tmp_dur$lead_C = tmp_sizes$lead_C
        tmp_dur$lead_product = tmp_sizes$lead_A * tmp_sizes$lead_B * tmp_sizes$lead_C
        tmp_dur$size_product = tmp_sizes$m * tmp_sizes$n * tmp_sizes$k
        tmp_dur$ratio = tmp_dur$lead_product/tmp_dur$size_product
        ndf = rbind(ndf, tmp_dur)
    }
    return(ndf);
}
#+end_src

#+begin_src R :results output :session *R* :exports both
result = insert_sizes(durations, sizes)
head(result)
#+end_src

#+RESULTS:
#+begin_example
    rank     start       end duration     state startline                           startfile endline                             endfile idx    m    n   k lead_A lead_B lead_C lead_product
481    0  4.111176  7.158459 3.047283 Computing       387 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     387 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 481 4920 4920 120   5040   4920   5040 124975872000
486    0  7.827329 10.848572 3.021243 Computing       411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 486 4920 4920 120   4920   4920   5040 122000256000
491    0 11.411456 14.445789 3.034333 Computing       411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 491 4920 4920 120   4920   4920   5040 122000256000
496    0 14.837377 17.868118 3.030741 Computing       411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 496 4920 4920 120   4920   4920   5040 122000256000
977    0 18.268679 21.142146 2.873467 Computing       387 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     387 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 977 4800 4800 120   5040   4800   5040 121927680000
982    0 21.809954 24.699182 2.889228 Computing       411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     411 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 982 4800 4800 120   4800   4800   5040 116121600000
    size_product    ratio
481   2904768000 43.02439
486   2904768000 42.00000
491   2904768000 42.00000
496   2904768000 42.00000
977   2764800000 44.10000
982   2764800000 42.00000
#+end_example

#+begin_src R :file images/trace4_16.png :results value graphics :results output :session *R* :exports both
library(ggplot2)
ggplot(result, aes(x=lead_product, y=duration, color=factor(rank))) +
    geom_point(shape=1) + ggtitle("Durations of HPL_dgemm as a function of the leading dimensions")
#+end_src

#+RESULTS:
[[file:images/trace4_16.png]]

#+begin_src R :file images/trace5_16.png :results value graphics :results output :session *R* :exports both
library(ggplot2)
ggplot(result, aes(x=lead_product, y=size_product, color=factor(rank))) +
    geom_point(shape=1) + ggtitle("Size of the matrices of HPL_dgemm as a function of the leading dimensions")
#+end_src

#+RESULTS:
[[file:images/trace5_16.png]]

#+begin_src R :file images/trace6_16.png :results value graphics :results output :session *R* :exports both
ggplot(result, aes(x=idx, y=ratio, color=factor(rank))) +
    geom_point(shape=1) + ggtitle("Ratios of the leading dimensions by the sizes over time")
#+end_src

#+RESULTS:
[[file:images/trace6_16.png]]

#+begin_src R :results output :session *R* :exports both
reg <- lm(duration~ I(m*n*k) + lead_A+lead_B+lead_C, data=result)
summary(reg)
#+end_src

#+RESULTS:
#+begin_example

Call:
lm(formula = duration ~ I(m * n * k) + lead_A + lead_B + lead_C, 
    data = result)

Residuals:
     Min       1Q   Median       3Q      Max 
-0.09477 -0.01804 -0.00439  0.00850  1.39992 

Coefficients:
               Estimate Std. Error t value Pr(>|t|)    
(Intercept)  -7.741e-01  9.915e-02  -7.807 8.37e-15 ***
I(m * n * k)  1.069e-09  4.431e-12 241.217  < 2e-16 ***
lead_A        2.965e-06  7.744e-07   3.828 0.000132 ***
lead_B       -7.048e-06  2.799e-06  -2.518 0.011863 *  
lead_C        1.547e-04  1.981e-05   7.810 8.16e-15 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.04981 on 2631 degrees of freedom
Multiple R-squared:  0.9972,	Adjusted R-squared:  0.9972 
F-statistic: 2.361e+05 on 4 and 2631 DF,  p-value: < 2.2e-16
#+end_example

#+begin_src R :file images/reg2_16.png :results value graphics :results output :session *R* :exports both
layout(matrix(c(1,2,3,4),2,2))
plot(reg)
#+end_src

#+RESULTS:
[[file:images/reg2_16.png]]

**** Discussion about the leading dimensions                 :EXPERIMENTS:
- In the three previous plots, we see that the leading dimensions have two modes, which are directly observable in the
  durations of =HPL_dgemm=.
  + One of the modes seems to be linear in the sizes, we observe a straight line.
  + The other mode is clearly non-linear. Maybe quadratic? Exponential?
- The linear regression shows that the variables =lead_A=, =lead_B= and =lead_C= have a non-negligible impact on the
  performances, albeit smaller than the sizes. We still have terrible plots, adding parameters in the model did not
  change anything.
- This could explain the “bad” plots of the linear regression.
**** Performance analysis of =dgemm= outside of HPL :C:EXPERIMENTS:PERFORMANCE:
- In the above analysis, the raw results come from a trace of HPL. Thus, we cannot control the sizes and/or leading
  dimensions. We only have observational data and not experimental data.
- To fix this, let’s write a short C code, called =dgemm_test=,  that call =cblas_dgemm= (the function to which is aliased =HPL_dgemm=).
- Currently, this code takes six parameters as arguments: the three sizes and the three leading dimensions. Be careful,
  the meaning of these sizes and leading dimensions change depending on how =dgemm= is called: =CblasColMajor= or
  =CblasRowMajor=, and =CblasNoTrans= or =CblasTrans=. In the current code, these are fixed to be the same than in HPL.
- Then, a Python script (called =runner.py=) sample random sizes and leading dimensions (taking care of the constraints
  between the sizes and dimensions) and call =dgemm_test=. It then writes the results in a CSV file.
- Quick analysis of these results in R:
  + We got plots with the same shape (both the plot of the raw results and the plot of the linear regression).
  + The call to =dgemm= is 10 times faster in =dgemm_test= than in HPL. Need to find why. Firstly, what is the time obtained in the HPL
    traces? Is it virtual or real?
  + Similarly than with HPL, the linear regression shows that the ratio has a significative impact, but lower than the
    sizes.
*** 2017-03-08 Wednesday
**** Keep looking at =dgemm= outside of HPL      :C:EXPERIMENTS:PERFORMANCE:
- Use =dgemm_test= at commit =0455edcb0af1eb673725959d216137997fc40fd2=. Run 1000 experiments.
- Here, the variable =product= is sampled randomly and uniformly in [1, 2000^3]. Then, the three sizes are set to \lfloor
  product^(1/3) \rfloor.
- The leading dimensions are equal to the sizes.
- Analysis in R:
  #+begin_src R :results output :session *R* :exports both
  result <- read.csv('~/tmp/3/result.csv')
  head(result)
  #+end_src

  #+RESULTS:
  :       time size_product lead_product ratio    m    n    k lead_A lead_B lead_C
  : 1 0.160235    843908625    843908625     1  945  945  945    945    945    945
  : 2 0.719003   4298942376   4298942376     1 1626 1626 1626   1626   1626   1626
  : 3 0.783674   4549540393   4549540393     1 1657 1657 1657   1657   1657   1657
  : 4 0.472595   2656741625   2656741625     1 1385 1385 1385   1385   1385   1385
  : 5 0.319670   1874516337   1874516337     1 1233 1233 1233   1233   1233   1233
  : 6 1.131936   6676532387   6676532387     1 1883 1883 1883   1883   1883   1883

  #+begin_src R :file images/dgemm_test_raw.png :results value graphics :results output :session *R* :exports both
  library(ggplot2)
  ggplot(result, aes(x=size_product, y=time)) +
      geom_point(shape=1) + ggtitle("Durations of cblas_dgemm as a function of the sizes product.")
  #+end_src

      #+RESULTS:
      [[file:images/dgemm_test_raw.png]]

    #+begin_src R :results output :session *R* :exports both
    reg <- lm(time ~ size_product, result)
    summary(reg)
    #+end_src

    #+RESULTS:
    #+begin_example

    Call:
    lm(formula = time ~ size_product, data = result)

    Residuals:
          Min        1Q    Median        3Q       Max 
    -0.027295 -0.008640 -0.002781  0.005900  0.229935 

    Coefficients:
                  Estimate Std. Error t value Pr(>|t|)    
    (Intercept)  1.172e-02  1.087e-03   10.78   <2e-16 ***
    size_product 1.666e-10  2.353e-13  707.87   <2e-16 ***
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    Residual standard error: 0.01716 on 998 degrees of freedom
    Multiple R-squared:  0.998,	Adjusted R-squared:  0.998 
    F-statistic: 5.011e+05 on 1 and 998 DF,  p-value: < 2.2e-16
    #+end_example


  #+begin_src R :file images/dgemm_test_lm.png :results value graphics :results output :session *R* :exports both
  layout(matrix(c(1,2,3,4),2,2))
  plot(reg)
  #+end_src

  #+RESULTS:
  [[file:images/dgemm_test_lm.png]]

- In the above plots, we can observe similar trends than with =HPL_dgemm=, albeit less important.
  The data is slightly heteroscedastic and the residuals do not follow exactly a normal distribution. It seems that
  there are several “outliers” where =dgemm= takes significantly more time, i.e. the distribution of the residuals is
  skewed to the “right”.
- For instance, the entry n°208 has been obtained with sizes of 1503. It took a time of 0.807207.
  Let’s run this experiment again 100 times (with the command =./dgemm_test 1503 1503 1503 1503 1503 1503=). The min and
  the max over all observed times are respectively 0.5813 and 0.6494. The mean is 0.5897 and the standard deviation
  is 0.0082.
- Thus, it seems that this point is a real outlier. We can suppose that this is also true for the other similar points.
- This outlier is 0.2 seconds larger than the average we got and 0.15 seconds larger than the max. It seems very
  large. Maybe the process had a “bad” context switch (e.g. if it was moved to another core, but the execution time is
  not that high, so it seems unlikely).
- There seems to be a pattern, the outliers look to happen at regular intervals.
  #+begin_src R :results output :session *R* :exports both
  x = df[abs(df$time - (1.666e-10*df$size_product + 1.172e-2)) > 5e-2, ]
  x$id = which(abs(df$time - (1.666e-10*df$size_product + 1.172e-2)) > 5e-2)
  x$prev_id = c(0, x$id[1:(length(x$id)-1)])
  x$id_diff = x$id - x$prev_id
  x
  #+end_src

  #+RESULTS:
  #+begin_example
          time size_product lead_product ratio    m    n    k lead_A lead_B
  37  0.674633   3602686437   3602686437     1 1533 1533 1533   1533   1533
  38  0.409866   2053225511   2053225511     1 1271 1271 1271   1271   1271
  207 1.295097   7055792632   7055792632     1 1918 1918 1918   1918   1918
  208 0.807207   3395290527   3395290527     1 1503 1503 1503   1503   1503
  381 1.079795   5535839609   5535839609     1 1769 1769 1769   1769   1769
  558 0.453775   1869959168   1869959168     1 1232 1232 1232   1232   1232
  657 0.917557   4699421875   4699421875     1 1675 1675 1675   1675   1675
  748 1.233466   6414120712   6414120712     1 1858 1858 1858   1858   1858
  753 0.708934   3884701248   3884701248     1 1572 1572 1572   1572   1572
  914 1.337868   7166730752   7166730752     1 1928 1928 1928   1928   1928
      lead_C  id prev_id id_diff
  37    1533  37       0      37
  38    1271  38      37       1
  207   1918 207      38     169
  208   1503 208     207       1
  381   1769 381     208     173
  558   1232 558     381     177
  657   1675 657     558      99
  748   1858 748     657      91
  753   1572 753     748       5
  914   1928 914     753     161
  #+end_example

  We see here that the differences between the ids do not seem to be uniformly random. Some of them are small (1, 5),
  others are large (161, 169, 173, 177), or in between (37, 91, 99).
- This pattern has been reproduced by runing 1000 experiments with a size of 1503. Among the results, 26 of them are
  larger than 0.7 (mean of 0.6024, standard deviation of 0.0249, min of 0.5811, max of 0.8363).
  Here is the list of the differences between the indices of these elements. The list have been sorted:
  #+begin_example
  [1, 1, 1, 1, 1, 1, 2, 4, 4, 5, 7, 7, 10, 15, 20, 25, 28, 32, 42, 42, 43, 53, 108, 200, 201]
  #+end_example
  A lot of them are small or medium, and two are much larger.
**** Time prediction in HPL       :C:PYTHON:R:EXPERIMENTS:PERFORMANCE:HPL:
- Let’s try to predict the time that will be spend in =HPL_dgemm=, and compare it with the real time.
  The aim is then to have a cheap SimBLAS: replacing calls to the function by a sleep of the predicted time.
  We have this =printf= before the calls to =HPL_dgemm=:
  #+begin_src c
  printf("line=%d rank=%d m=%d n=%d k=%d lead_A=%d lead_B=%d lead_C=%d expected_time=%f\n",
          __LINE__+3, rank, mp, nn, jb, ldl2, LDU, lda, expected_time);
  #+end_src
  We do as before: we run HPL with P=Q=4 and N=20000. The trace is dumped in =/tmp/trace= and =stdout= is redirected to
  =/tmp/output=.
- Processing of the output:
#+begin_src python
import re
import csv
reg = re.compile('line=([0-9]+) rank=([0-9]+) m=([0-9]+) n=([0-9]+) k=([0-9]+) lead_A=([0-9]+) lead_B=([0-9]+) lead_C=([0-9]+) expected_time=(-?[0-9]+.[0-9]+)')

def process(in_file, out_file):
    with open(in_file, 'r') as in_f:
        with open(out_file, 'w') as out_f:
            csv_writer = csv.writer(out_f)
            csv_writer.writerow(('line', 'rank', 'n', 'm', 'k', 'lead_A', 'lead_B', 'lead_C', 'expected_time'))
            for line in in_f:
                match = reg.match(line)
                if match is not None:
                    csv_writer.writerow(tuple(match.group(i) for i in range(1,10)))
process('/tmp/output', '/tmp/sizes.csv')
#+end_src

#+RESULTS:
: None
- We process the trace as before, we get a dataframe =durations=.
  #+begin_src R :results output :session *R* :exports both
  head(durations)
  #+end_src

  #+RESULTS:
  :     rank     start      end duration     state startline                           startfile endline                             endfile idx
  : 481    0  3.480994  6.54468 3.063686 Computing       388 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     388 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 481
  : 486    0  7.225255 10.24889 3.023633 Computing       413 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     413 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 486
  : 491    0 10.803780 13.82799 3.024215 Computing       413 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     413 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 491
  : 496    0 14.230774 17.26467 3.033897 Computing       413 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     413 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 496
  : 977    0 17.676746 20.58197 2.905229 Computing       388 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     388 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 977
  : 982    0 21.258337 24.16961 2.911277 Computing       413 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     413 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 982

  #+begin_src R :results output :session *R* :exports both
  sizes <- read.csv("/tmp/sizes.csv");
  head(sizes)
  #+end_src

  #+RESULTS:
  :   line rank    n    m   k lead_A lead_B lead_C expected_time
  : 1  413    8 5000 4920 120   5000   4920   5000      3.132548
  : 2  413   12 4920 4920 120   4920   4920   4920      3.082388
  : 3  413    4 5040 4920 120   5040   4920   5040      3.157628
  : 4  388    0 4920 4920 120   5040   4920   5040      3.082388
  : 5  413    5 5040 5040 120   5040   5040   5040      3.234704
  : 6  413    9 5000 5040 120   5000   5040   5000      3.209012

  #+begin_src R :results output :session *R* :exports both
  insert_sizes = function(durations, sizes) {
      stopifnot(nrow(durations)==nrow(sizes))
      ndf = data.frame();
      for(i in (sort(unique(durations$rank)))) {
          tmp_dur = durations[durations$rank == i,]
          tmp_sizes = sizes[sizes$rank == i,]
          stopifnot(nrow(tmp_dur) == nrow(tmp_sizes))
          stopifnot(tmp_dur$startline == tmp_sizes$line)
          storage.mode(tmp_sizes$m) <- "double" # avoiding integer overflow when taking the product
          storage.mode(tmp_sizes$n) <- "double"
          storage.mode(tmp_sizes$k) <- "double"
          storage.mode(tmp_sizes$lead_A) <- "double"
          storage.mode(tmp_sizes$lead_B) <- "double"
          storage.mode(tmp_sizes$lead_C) <- "double"
          tmp_dur$m = tmp_sizes$m
          tmp_dur$n = tmp_sizes$n
          tmp_dur$k = tmp_sizes$k
          tmp_dur$lead_A = tmp_sizes$lead_A
          tmp_dur$lead_B = tmp_sizes$lead_B
          tmp_dur$lead_C = tmp_sizes$lead_C
          tmp_dur$lead_product = tmp_sizes$lead_A * tmp_sizes$lead_B * tmp_sizes$lead_C
          tmp_dur$size_product = tmp_sizes$m * tmp_sizes$n * tmp_sizes$k
          tmp_dur$ratio = tmp_dur$lead_product/tmp_dur$size_product
          tmp_dur$expected_time = tmp_sizes$expected_time
          tmp_dur$absolute_time_diff = tmp_dur$expected_time - tmp_dur$duration
          tmp_dur$relative_time_diff = (tmp_dur$expected_time - tmp_dur$duration)/tmp_dur$expected_time
          ndf = rbind(ndf, tmp_dur)
      }
      return(ndf);
  }
  #+end_src

  #+begin_src R :results output :session *R* :exports both
  result = insert_sizes(durations, sizes)
  head(result)
  #+end_src

  #+RESULTS:
  #+begin_example
      rank     start      end duration     state startline                           startfile endline                             endfile idx    m    n   k lead_A lead_B lead_C lead_product
  481    0  3.480994  6.54468 3.063686 Computing       388 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     388 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 481 4920 4920 120   5040   4920   5040 124975872000
  486    0  7.225255 10.24889 3.023633 Computing       413 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     413 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 486 4920 4920 120   4920   4920   5040 122000256000
  491    0 10.803780 13.82799 3.024215 Computing       413 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     413 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 491 4920 4920 120   4920   4920   5040 122000256000
  496    0 14.230774 17.26467 3.033897 Computing       413 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     413 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 496 4920 4920 120   4920   4920   5040 122000256000
  977    0 17.676746 20.58197 2.905229 Computing       388 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     388 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 977 4800 4800 120   5040   4800   5040 121927680000
  982    0 21.258337 24.16961 2.911277 Computing       413 /hpl-2.2/src/pgesv/hpl_pdupdatett.c     413 /hpl-2.2/src/pgesv/hpl_pdupdatett.c 982 4800 4800 120   4800   4800   5040 116121600000
      size_product    ratio expected_time absolute_time_diff relative_time_diff
  481   2904768000 43.02439      3.082388           0.018702        0.006067374
  486   2904768000 42.00000      3.082388           0.058755        0.019061520
  491   2904768000 42.00000      3.082388           0.058173        0.018872705
  496   2904768000 42.00000      3.082388           0.048491        0.015731634
  977   2764800000 44.10000      2.933742           0.028513        0.009718987
  982   2764800000 42.00000      2.933742           0.022465        0.007657456
#+end_example

#+begin_src R :file images/trace7_16.png :results value graphics :session *R* :exports both
ggplot(result, aes(x=idx, y=absolute_time_diff, color=factor(rank))) +
    geom_point(shape=1) + ggtitle("Absolute difference between the expected time and the real time")
#+end_src

#+RESULTS:
[[file:images/trace7_16.png]]

#+begin_src R :file images/trace8_16.png :results value graphics :session *R* :exports both
ggplot(result, aes(x=start, y=absolute_time_diff, color=factor(rank))) +
    geom_point(shape=1) + ggtitle("Absolute difference between the expected time and the real time")
#+end_src

#+RESULTS:
[[file:images/trace8_16.png]]

#+begin_src R :file images/trace9_16.png :results value graphics :session *R* :exports both
ggplot(result, aes(x=start, y=relative_time_diff, color=factor(rank))) +
    geom_point(shape=1) + ggtitle("Relative difference between the expected time and the real time")
#+end_src

#+RESULTS:
[[file:images/trace9_16.png]]

#+begin_src R :file images/trace10_16.png :results value graphics :session *R* :exports both
ggplot(result[result$start < 200,], aes(x=start, y=relative_time_diff, color=factor(rank))) +
    geom_point(shape=1) + ggtitle("Relative difference between the expected time and the real time\n“Large enough” matrices")
#+end_src

#+RESULTS:
[[file:images/trace10_16.png]]

  #+begin_src R :results output :session *R* :exports both
  for(i in (sort(unique(result$rank)))) {
      print(sum(result[result$rank == i,]$absolute_time_diff))
  }
  #+end_src

  #+RESULTS:
  #+begin_example
  [1] 1.494745
  [1] 1.343339
  [1] -2.940891
  [1] -1.11672
  [1] 0.466087
  [1] 1.90049
  [1] -3.441326
  [1] -1.564635
  [1] -2.708597
  [1] -1.647053
  [1] 0.027765
  [1] -4.653833
  [1] 2.878523
  [1] 3.572304
  [1] 1.124928
  [1] 3.749203
#+end_example

- We can see several things.
  + There are very large differences between the ranks. We could already see it in the first plots (=duration= vs
    =size_product=), but it is even more obvious here. We should find why.
  + There are some outliers that may have a very significant impact on the agregated difference between prediction and
    reality.
  + The prediction ability of this is better than =SMPI_Sample=, but still far from perfect.
**** Let’s try a cheap SimBLAS                    :SMPI:C:PERFORMANCE:HPL:
- We can replace the call to =HPL_dgemm= by the following:
  #+begin_src c
  double expected_time = (1.062e-09)*(double)mp*(double)nn*(double)jb - 2.476e-03
  if(expected_time > 0)
      smpi_usleep((useconds_t)(expected_time*1e6));
  #+end_src
- First test: it works pretty well. We roughly got the same results than with the true call to =HPL_dgemm=: 2.329e+01
  Gflops, against 2.332e01, 2.305e01 and 2.315e01 Gflops. The simulation time is much shorter, about 46 seconds, against
  about 495 seconds (8 minutes and 15 seconds). Note than with or without a real call to =HPL_dgemm=, the time spent
  outside of the application is much lower: between 6 and 8 seconds. Thus, there is room for new optimizations.
**** Tracking the other expensive BLAS functions         :PERFORMANCE:HPL:
- In the file =hpl_blas.h=, several functions are defined like =HPL_dgemm=, with =#define= aliasing them to the real =cblas= function.
- We can try to replace them by a no-op, to see if it changes the simulation time significantly.
- The following table sum up the (very approximate) gain we get on simulation time if we remove each of the
  functions. We use the same parameters than above for HPL.

  | Function   | time (s) |
  |------------+----------|
  | =HPL_dswap=  |      0.5 |
  | =HPL_dcopy=  |      N/A |
  | =HPL_daxpy=  |        0 |
  | =HPL_dscal=  |      N/A |
  | =HPL_idamax= |      N/A |
  | =HPL_dgemv=  |        1 |
  | =HPL_dtrsv=  |        0 |
  | =HPL_dger=   |      0.5 |
  | =HPL_dtrsm=  |       10 |

  + The function =HPL_idamax= cannot be removed, since it returns an integer used to index an array.
  + The functions =HPL_dscal= and =HPL_dcopy= cannot be removed either, since removing them causes the following error:
    #+begin_example
    /home/tom/simgrid/src/simix/smx_global.cpp:557: [simix_kernel/CRITICAL] Oops ! Deadlock or code not perfectly clean.
    #+end_example
- It is clear that we should now focus on =HPL_dtrsm=. This function solves a triangular system of equations.
- It is also clear that the time spent in the application is not entirely spent in the BLAS functions, we should look
  for something else.
**** Forgot a call to =HPL_dgemm=                          :PERFORMANCE:HPL:
- I found out that I forgot a place where =HPL_dgemm= was used.
- If we remove all additional occurences of =HPL_dgemm=, we gain 6 seconds (in addition of the high gain we already had).
- I thought that it was used only in =HPL_pduptateTT=, but it appears that it is also used in =HPL_pdrpanllT=.
- The call to =HPL_dgemm= was correctly traced. But I filtered the results in the R script and kept only the ones of =HPL_pdupdateTT=.
- The =printf= function with the parameters was only present in =HPL_pdupdateTT=.
- Consequently, all the visualizations and linear regressions were done with missing data. We should redo them to check
  if this changes anything.
**** Looking at =HPL_dtrsm=                                :PERFORMANCE:HPL:
- This function is used in a lot of functions: =HPL_pdrpan***= and =HPL_pdupdate**= (each has several variants).
- By aliasing this function to =printf("%s\n", __FILE___)= and filtering the output with =awk '!a[$0]++'= (remove duplicates),
  we know that, in our settings, =HPL_dtrsm= is only used in =HPL_pdrpanllT= and =HPL_pdupdateTT=. By sorting with =sort= and
  then counting duplicates with =uniq -dc=, we know that =HPL_pdrpanllT= (resp. =HPL_pdupdateTT=) call our function 78664 times
  (resp. 2636 times).
*** 2017-03-09 Thursday
**** Fix =HPL_dgemm= trace                                   :C:TRACING:HPL:
- In the old version, the calls to =MPI_Wait= were done in the =#include=, so we were sure that every call to =HPL_dgemm= was
  traced by Simgrid. However, the =printf= for the parameters had to be done before every call to =HPL_dgemm=, this is why
  I missed some of them.
- Now, the =printf= is also done in the =#include=. Because we need to have the arguments given to =HPL_dgemm= here, we cannot
  anymore use variadic arguments. We have to put all the parameters.
- The code is now as follows:
  #+begin_src c
  #define  HPL_dgemm(layout, TransA, TransB, M, N, K, alpha, A, lda, B, ldb, beta, C, ldc)  ({\
      int my_rank, buff=0;\
      MPI_Request request;\
      MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);\
      double expected_time = (1.062e-09)*(double)M*(double)N*(double)K - 2.476e-03;\
      printf("file=%s line=%d rank=%d m=%d n=%d k=%d lead_A=%d lead_B=%d lead_C=%d expected_time=%f\n", __FILE__, __LINE__+3, my_rank, M, N, K, lda, ldb, ldc, expected_time);\
      MPI_Isend(&buff, 1, MPI_INT, my_rank, 0, MPI_COMM_WORLD, &request);\
      MPI_Recv(&buff, 1, MPI_INT, my_rank, 0, MPI_COMM_WORLD, NULL);\
      MPI_Wait(&request, MPI_STATUS_IGNORE);\
      cblas_dgemm(layout, TransA, TransB, M, N, K, alpha, A, lda, B, ldb, beta, C, ldc);\
      MPI_Isend(&buff, 1, MPI_INT, my_rank, 0, MPI_COMM_WORLD, &request);\
      MPI_Recv(&buff, 1, MPI_INT, my_rank, 0, MPI_COMM_WORLD, NULL);\
      MPI_Wait(&request, MPI_STATUS_IGNORE);\
  })
  #+end_src
**** Tentative of linear regression of =HPL_dgemm=: failed, there is a bug somewhere :PYTHON:R:EXPERIMENTS:PERFORMANCE:BUG:
- In the other linear regressions, some calls to =HPL_dgemm= were missing. Thus, the analysis need to be done again, just
  to check if it changes anything.
- I tried to run roughly the same process as above, but failed, there seems to be a bug somewhere.
- Everything piece of code is written here. The trace and the output have been obtained with N=5000 and P=Q=4.
Clean the file:
#+begin_src sh
pj_dump --user-defined --ignore-incomplete-links /tmp/trace > /tmp/trace.csv
grep "State," /tmp/trace.csv | grep MPI_Wait | sed -e 's/()//' -e 's/MPI_STATE, //ig'  -e 's/State, //ig' -e 's/rank-//' -e\
's/PMPI_/MPI_/' | grep MPI_  | tr 'A-Z' 'a-z' > /tmp/trace_processed.csv
#+end_src

Clean the paths:
#+begin_src python
import re
reg = re.compile('((?:[^/])*)(?:/[a-zA-Z0-9_-]*)*((?:/hpl-2.2(?:/[a-zA-Z0-9_-]*)*).*)')
def process(in_file, out_file):
    with open(in_file, 'r') as in_f:
        with open(out_file, 'w') as out_f:
            for line in in_f:
                match = reg.match(line)
                out_f.write('%s%s\n' % (match.group(1), match.group(2)))
process('/tmp/trace_processed.csv', '/tmp/trace_cleaned.csv')
 #+end_src

 #+RESULTS:
 : None

#+begin_src R :results output :session *R* :exports both
df <- read.csv("/tmp/trace_cleaned.csv", header=F, strip.white=T, sep=",");
names(df) = c("rank", "start", "end", "duration", "level", "state", "Filename", "Linenumber");
head(df)
#+end_src

#+RESULTS:
#+begin_example
  rank    start      end duration level    state
1    8 0.207257 0.207257        0     0 mpi_wait
2    8 0.207275 0.207275        0     0 mpi_wait
3    8 0.207289 0.207289        0     0 mpi_wait
4    8 0.207289 0.207289        0     0 mpi_wait
5    8 0.207309 0.207309        0     0 mpi_wait
6    8 0.207309 0.207309        0     0 mpi_wait
                            Filename Linenumber
1 /hpl-2.2/src/pfact/hpl_pdrpanllt.c        222
2 /hpl-2.2/src/pfact/hpl_pdrpanllt.c        222
3 /hpl-2.2/src/pfact/hpl_pdrpanllt.c        222
4 /hpl-2.2/src/pfact/hpl_pdrpanllt.c        222
5 /hpl-2.2/src/pfact/hpl_pdrpanllt.c        222
6 /hpl-2.2/src/pfact/hpl_pdrpanllt.c        222
#+end_example

#+BEGIN_SRC R :results output :session *R* :exports both
duration_compute = function(df) {
    ndf = data.frame();
    df = df[with(df,order(rank,start)),];
    #origin = unique(df$origin)
    for(i in (sort(unique(df$rank)))) {
        start     = df[df$rank==i,]$start;
        end       = df[df$rank==i,]$end;
        l         = length(end);
        end       = c(0,end[1:(l-1)]); # Computation starts at time 0

        startline = c(0, df[df$rank==i,]$Linenumber[1:(l-1)]);
        startfile = c("", as.character(df[df$rank==i,]$Filename[1:(l-1)]));
        endline   = df[df$rank==i,]$Linenumber;
        endfile   = df[df$rank==i,]$Filename;

        ndf       = rbind(ndf, data.frame(rank=i, start=end, end=start,
            duration=start-end, state="Computing",
            startline=startline, startfile=startfile, endline=endline,
            endfile=endfile));
    }
    ndf$idx = 1:length(ndf$duration)
    ndf;
}
durations = duration_compute(df);
durations = durations[as.character(durations$startfile) == as.character(durations$endfile) &
    durations$startline == durations$endline,]
#+END_SRC

#+BEGIN_SRC R :results output :session *R* :exports both
head(durations)
#+END_SRC

#+RESULTS:
#+begin_example
  rank    start      end duration     state startline
2    0 0.207097 0.207149  5.2e-05 Computing       222
3    0 0.207149 0.207179  3.0e-05 Computing       222
4    0 0.207179 0.207179  0.0e+00 Computing       222
5    0 0.207179 0.207194  1.5e-05 Computing       222
6    0 0.207194 0.207194  0.0e+00 Computing       222
7    0 0.207194 0.207207  1.3e-05 Computing       222
                           startfile endline                            endfile
2 /hpl-2.2/src/pfact/hpl_pdrpanllt.c     222 /hpl-2.2/src/pfact/hpl_pdrpanllt.c
3 /hpl-2.2/src/pfact/hpl_pdrpanllt.c     222 /hpl-2.2/src/pfact/hpl_pdrpanllt.c
4 /hpl-2.2/src/pfact/hpl_pdrpanllt.c     222 /hpl-2.2/src/pfact/hpl_pdrpanllt.c
5 /hpl-2.2/src/pfact/hpl_pdrpanllt.c     222 /hpl-2.2/src/pfact/hpl_pdrpanllt.c
6 /hpl-2.2/src/pfact/hpl_pdrpanllt.c     222 /hpl-2.2/src/pfact/hpl_pdrpanllt.c
7 /hpl-2.2/src/pfact/hpl_pdrpanllt.c     222 /hpl-2.2/src/pfact/hpl_pdrpanllt.c
  idx
2   2
3   3
4   4
5   5
6   6
7   7
#+end_example

#+BEGIN_SRC R :results output :session *R* :exports both
unique(durations[c("startfile", "startline")])
#+END_SRC

#+RESULTS:
:                               startfile startline
: 2    /hpl-2.2/src/pfact/hpl_pdrpanllt.c       222
: 14         /hpl-2.2/src/comm/hpl_sdrv.c       191
: 478      /hpl-2.2/src/pgesv/hpl_rollt.c       242
: 481 /hpl-2.2/src/pgesv/hpl_pdupdatett.c       384
: 486 /hpl-2.2/src/pgesv/hpl_pdupdatett.c       407

We need to check each of these to see if this is indeed a call to =HPL_dgemm=, or something else.
It appears that =HPL_rollT= and =HPL_sdrv= are not calling =HPL_dgemm=, they are just calling =MPI_Wait=. Thus, we have to
remove them.

#+BEGIN_SRC R :results output :session *R* :exports both
durations = durations[durations$startfile != "/hpl-2.2/src/comm/hpl_sdrv.c" & durations$startfile != "/hpl-2.2/src/pgesv/hpl_rollt.c",]
unique(durations[c("startfile", "startline")])
#+END_SRC


#+RESULTS:
:                               startfile startline
: 2    /hpl-2.2/src/pfact/hpl_pdrpanllt.c       222
: 481 /hpl-2.2/src/pgesv/hpl_pdupdatett.c       384
: 486 /hpl-2.2/src/pgesv/hpl_pdupdatett.c       407

Now, let us get what was output by the =printf=.

Processing the output:
#+begin_src python
import re
import csv
reg = re.compile('file=([a-zA-Z0-9/_.-]+) line=([0-9]+) rank=([0-9]+) m=([0-9]+) n=([0-9]+) k=([0-9]+) lead_A=([0-9]+) lead_B=([0-9]+) lead_C=([0-9]+) expected_time=(-?[0-9]+.[0-9]+)')

def process(in_file, out_file):
    with open(in_file, 'r') as in_f:
        with open(out_file, 'w') as out_f:
            csv_writer = csv.writer(out_f)
            csv_writer.writerow(('file', 'line', 'rank', 'n', 'm', 'k', 'lead_A', 'lead_B', 'lead_C', 'expected_time'))
            for line in in_f:
                match = reg.match(line)
                if match is not None:
                    result = list(match.group(i) for i in range(1, 11))
                    result[0] = result[0][result[0].index('/hpl'):].lower()
                    csv_writer.writerow(result)
process('/tmp/output', '/tmp/parameters.csv')
#+end_src

#+begin_src R :results output :session *R* :exports both
parameters <- read.csv("/tmp/parameters.csv");
head(parameters)
#+end_src

#+RESULTS:
#+begin_example
                                file line rank    n  m k lead_A lead_B lead_C
1 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  222    0 1320 60 0   1320    120   1320
2 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  222    8 1200 60 0   1200    120   1200
3 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  222    0 1320 30 0   1320    120   1320
4 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  222    4 1280 60 0   1280    120   1280
5 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  222    0 1320 16 0   1320    120   1320
6 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  222    0 1320  8 0   1320    120   1320
  expected_time
1     -0.002476
2     -0.002476
3     -0.002476
4     -0.002476
5     -0.002476
6     -0.002476
#+end_example

A first remark: we see that some rows have k=0, which is a bit surprising. I double-checked by adding some =printf= in the
files, this is not a bug. This only happens in =HPL_pdrpanllT= so it was unnoticed until now.

#+begin_src R :results output :session *R* :exports both
nrow(parameters)
nrow(durations)
nrow(parameters[parameters$file == "/hpl-2.2/src/pfact/hpl_pdrpanllt.c",])
nrow(durations[durations$startfile == "/hpl-2.2/src/pfact/hpl_pdrpanllt.c",])
#+end_src

#+RESULTS:
: [1] 20300
: [1] 29964
: [1] 19664
: [1] 29328

- There is obviously something wrong. We should have a one-to-one correspondance between the elements of the =parameters=
  dataframe and the elements of the =durations= dataframe. It seems here that SMPI has produced additional entries in the
  trace, or some of the =printf= I put disapeared.
- This is not an error in parsing the output (e.g. some lines not parsed because of a wrong format/regexp). The output
  file has 20359 lines.
- Tried puting a =printf("blabla\n")= just before =HPL_dgemm= in the file =HPL_pdrpanllT.c= and counted the number of times it
  appeared. Exactly the same number, so definitely not an issue with the parsing or the definition with the =#define=.
- Checked the =durations= dataframe. Nothing apparently wrong, all the entries for this file are at the same line, so I
  did not miss a hidden =MPI_Wait= somewhere else in this same file).
**** Using another way to measure durations :C:PYTHON:R:EXPERIMENTS:TRACING:PERFORMANCE:HPL:
- Let’s use something else than SMPI trace to measure durations. We will measure the time directly in the code. But
  first we need to check that this new measure is consistent with what we got with the traces.
- Now, =HPL_dgemm= is defined as:
#+begin_src c
#define  HPL_dgemm(layout, TransA, TransB, M, N, K, alpha, A, lda, B, ldb, beta, C, ldc)  ({\
  int my_rank, buff=0;\
  MPI_Request request;\
  MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);\
  double expected_time = (1.062e-09)*(double)M*(double)N*(double)K - 2.476e-03;\
  struct timeval before = {};\
  struct timeval after = {};\
  gettimeofday(&before, NULL);\
  MPI_Isend(&buff, 1, MPI_INT, my_rank, 0, MPI_COMM_WORLD, &request);\
  MPI_Recv(&buff, 1, MPI_INT, my_rank, 0, MPI_COMM_WORLD, NULL);\
  MPI_Wait(&request, MPI_STATUS_IGNORE);\
  cblas_dgemm(layout, TransA, TransB, M, N, K, alpha, A, lda, B, ldb, beta, C, ldc);\
  gettimeofday(&after, NULL);\
  MPI_Isend(&buff, 1, MPI_INT, my_rank, 0, MPI_COMM_WORLD, &request);\
  MPI_Recv(&buff, 1, MPI_INT, my_rank, 0, MPI_COMM_WORLD, NULL);\
  MPI_Wait(&request, MPI_STATUS_IGNORE);\
  double time_before = (double)(before.tv_sec) + (double)(before.tv_usec)*1e-6;\
  double time_after = (double)(after.tv_sec) + (double)(after.tv_usec)*1e-6;\
  double real_time = time_after-time_before;\
  printf("file=%s line=%d rank=%d m=%d n=%d k=%d lead_A=%d lead_B=%d lead_C=%d real_time=%f expected_time=%f\n", __FILE__, __LINE__, my_rank, M, N, K, lda, ldb, ldc, real_time, expected_time);\
})
#+end_src
- We run the same code than above to get the =durations= frame.
#+BEGIN_SRC R :results output :session *R* :exports both
head(durations)
#+END_SRC

#+RESULTS:
#+begin_example
  rank    start      end duration     state startline
2    0 0.275856 0.275896  4.0e-05 Computing       224
3    0 0.275896 0.275929  3.3e-05 Computing       224
4    0 0.275929 0.275929  0.0e+00 Computing       224
5    0 0.275929 0.275948  1.9e-05 Computing       224
6    0 0.275948 0.275948  0.0e+00 Computing       224
7    0 0.275948 0.275965  1.7e-05 Computing       224
                           startfile endline                            endfile
2 /hpl-2.2/src/pfact/hpl_pdrpanllt.c     224 /hpl-2.2/src/pfact/hpl_pdrpanllt.c
3 /hpl-2.2/src/pfact/hpl_pdrpanllt.c     224 /hpl-2.2/src/pfact/hpl_pdrpanllt.c
4 /hpl-2.2/src/pfact/hpl_pdrpanllt.c     224 /hpl-2.2/src/pfact/hpl_pdrpanllt.c
5 /hpl-2.2/src/pfact/hpl_pdrpanllt.c     224 /hpl-2.2/src/pfact/hpl_pdrpanllt.c
6 /hpl-2.2/src/pfact/hpl_pdrpanllt.c     224 /hpl-2.2/src/pfact/hpl_pdrpanllt.c
7 /hpl-2.2/src/pfact/hpl_pdrpanllt.c     224 /hpl-2.2/src/pfact/hpl_pdrpanllt.c
  idx
2   2
3   3
4   4
5   5
6   6
7   7
#+end_example

Now, we process the parameters:
#+begin_src python
import re
import csv
reg = re.compile('file=([a-zA-Z0-9/_.-]+) line=([0-9]+) rank=([0-9]+) m=([0-9]+) n=([0-9]+) k=([0-9]+) lead_A=([0-9]+) lead_B=([0-9]+) lead_C=([0-9]+) real_time=(-?[0-9]+.[0-9]+) expected_time=(-?[0-9]+.[0-9]+)')

def process(in_file, out_file):
    with open(in_file, 'r') as in_f:
        with open(out_file, 'w') as out_f:
            csv_writer = csv.writer(out_f)
            csv_writer.writerow(('file', 'line', 'rank', 'n', 'm', 'k', 'lead_A', 'lead_B', 'lead_C', 'real_time', 'expected_time'))
            for line in in_f:
                match = reg.match(line)
                if match is not None:
                    result = list(match.group(i) for i in range(1, 12))
                    result[0] = result[0][result[0].index('/hpl'):].lower()
                    csv_writer.writerow(result)
process('/tmp/output', '/tmp/parameters.csv')
#+end_src

#+begin_src R :results output :session *R* :exports both
parameters <- read.csv("/tmp/parameters.csv");
head(parameters)
#+end_src

#+RESULTS:
#+begin_example
                                file line rank    n  m k lead_A lead_B lead_C
1 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  224    0 1320 60 0   1320    120   1320
2 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  224    0 1320 30 0   1320    120   1320
3 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  224    0 1320 16 0   1320    120   1320
4 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  224    0 1320  8 0   1320    120   1320
5 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  224    0 1320  4 0   1320    120   1320
6 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  224    0 1320  2 0   1320    120   1320
  real_time expected_time
1   8.1e-05     -0.002476
2   0.0e+00     -0.002476
3   0.0e+00     -0.002476
4   0.0e+00     -0.002476
5   1.0e-06     -0.002476
6   0.0e+00     -0.002476
#+end_example

We merge the =durations= and =parameters= dataframes, but only the entries for the file =hpl_pdupdatett.c= (we cannot do it
for the other file since we have a mismatch).
#+begin_src R :results output :session *R* :exports both
insert_sizes = function(durations, sizes) {
    stopifnot(nrow(durations)==nrow(sizes))
    ndf = data.frame();
    for(i in (sort(unique(durations$rank)))) {
        tmp_dur = durations[durations$rank == i,]
        tmp_sizes = sizes[sizes$rank == i,]
        stopifnot(nrow(tmp_dur) == nrow(tmp_sizes))
        stopifnot(tmp_dur$startline == tmp_sizes$line)
        storage.mode(tmp_sizes$m) <- "double" # avoiding integer overflow when taking the product
        storage.mode(tmp_sizes$n) <- "double"
        storage.mode(tmp_sizes$k) <- "double"
        storage.mode(tmp_sizes$lead_A) <- "double"
        storage.mode(tmp_sizes$lead_B) <- "double"
        storage.mode(tmp_sizes$lead_C) <- "double"
        tmp_dur$m = tmp_sizes$m
        tmp_dur$n = tmp_sizes$n
        tmp_dur$k = tmp_sizes$k
        tmp_dur$lead_A = tmp_sizes$lead_A
        tmp_dur$lead_B = tmp_sizes$lead_B
        tmp_dur$lead_C = tmp_sizes$lead_C
        tmp_dur$lead_product = tmp_sizes$lead_A * tmp_sizes$lead_B * tmp_sizes$lead_C
        tmp_dur$size_product = tmp_sizes$m * tmp_sizes$n * tmp_sizes$k
        tmp_dur$ratio = tmp_dur$lead_product/tmp_dur$size_product
        tmp_dur$real_time = tmp_sizes$real_time
        tmp_dur$expected_time = tmp_sizes$expected_time
        tmp_dur$absolute_time_diff = tmp_dur$expected_time - tmp_dur$duration
        tmp_dur$relative_time_diff = (tmp_dur$expected_time - tmp_dur$duration)/tmp_dur$expected_time
        ndf = rbind(ndf, tmp_dur)
    }
    return(ndf);
}
#+end_src

#+begin_src R :results output :session *R* :exports both
result = insert_sizes(durations[durations$startfile == "/hpl-2.2/src/pgesv/hpl_pdupdatett.c",], parameters[parameters$file == "/hpl-2.2/src/pgesv/hpl_pdupdatett.c",])
#+end_src

Now we plot the time measured by SMPI traces against the time measured by =gettimeofday=.

#+begin_src R :file images/gettimeofday.png :results value graphics :session *R* :exports both
library(ggplot2)
ggplot(result, aes(x=duration, y=real_time)) +
    geom_point(shape=1) + ggtitle("Time measured by SMPI against time measured by gettimeofday")
#+end_src

#+RESULTS:
[[file:images/gettimeofday.png]]

Checking with a linear regression, just to be sure:
#+begin_src R :results output :session *R* :exports both
summary(lm(duration~real_time, data=result))
#+end_src

#+RESULTS:
#+begin_example

Call:
lm(formula = duration ~ real_time, data = result)

Residuals:
       Min         1Q     Median         3Q        Max 
-4.917e-05 -4.088e-06  1.075e-06  5.261e-06  6.181e-05 

Coefficients:
              Estimate Std. Error    t value Pr(>|t|)    
(Intercept) -2.617e-06  6.285e-07     -4.163 3.57e-05 ***
real_time    9.999e-01  7.058e-06 141678.252  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.034e-05 on 634 degrees of freedom
Multiple R-squared:      1,	Adjusted R-squared:      1 
F-statistic: 2.007e+10 on 1 and 634 DF,  p-value: < 2.2e-16
#+end_example

It is not perfect, but it looks pretty great. So, let’s use this to measure time.

**** Now we can finally re-do the analysis of =HPL_dgemm= :R:EXPERIMENTS:PERFORMANCE:HPL:
- There are less things to do, since all the data come from the output file.
- Recall the aim of doing this again: in the previous analysis, some calls to =HPL_dgemm= were missing. Thus, it needs to
  be done again, just to check if it changes anything.
- Generate the CSV file by runing the same Python script as in the previous section (the output format did not change).
- Then, analysis in R:
#+begin_src R :results output :session *R* :exports both
results <- read.csv("/tmp/parameters.csv");
head(results)
#+end_src

#+RESULTS:
#+begin_example
                                file line rank    n  m k lead_A lead_B lead_C
1 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  222    0 5040 60 0   5040    120   5040
2 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  222    0 5040 30 0   5040    120   5040
3 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  222    0 5040 16 0   5040    120   5040
4 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  222    0 5040  8 0   5040    120   5040
5 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  222    0 5040  4 0   5040    120   5040
6 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  222    8 5000 60 0   5000    120   5000
  real_time expected_time
1   5.7e-05     -0.002476
2   7.0e-06     -0.002476
3   0.0e+00     -0.002476
4   0.0e+00     -0.002476
5   0.0e+00     -0.002476
6   9.0e-06     -0.002476
#+end_example

#+begin_src R :results output :session *R* :exports both
process_results = function(results) {
    storage.mode(results$m) <- "double" # avoiding integer overflow when taking the product
    storage.mode(results$n) <- "double"
    storage.mode(results$k) <- "double"
    storage.mode(results$lead_A) <- "double"
    storage.mode(results$lead_B) <- "double"
    storage.mode(results$lead_C) <- "double"
    results$lead_product = results$lead_A * results$lead_B * results$lead_C
    results$size_product = results$m * results$n * results$k
    results$ratio = results$lead_product/results$size_product
    results$absolute_time_diff = results$expected_time - results$real_time
    results$relative_time_diff = (results$expected_time - results$real_time)/results$expected_time
    results$idx = 1:length(results$rank)
    return(results);
}
#+end_src

#+begin_src R :results output :session *R* :exports both
results = process_results(results)
head(results)
#+end_src

#+RESULTS:
#+begin_example
                                file line rank    n  m k lead_A lead_B lead_C
1 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  222    0 5040 60 0   5040    120   5040
2 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  222    0 5040 30 0   5040    120   5040
3 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  222    0 5040 16 0   5040    120   5040
4 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  222    0 5040  8 0   5040    120   5040
5 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  222    0 5040  4 0   5040    120   5040
6 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  222    8 5000 60 0   5000    120   5000
  real_time expected_time lead_product size_product ratio absolute_time_diff
1   5.7e-05     -0.002476   3048192000            0   Inf          -0.002533
2   7.0e-06     -0.002476   3048192000            0   Inf          -0.002483
3   0.0e+00     -0.002476   3048192000            0   Inf          -0.002476
4   0.0e+00     -0.002476   3048192000            0   Inf          -0.002476
5   0.0e+00     -0.002476   3048192000            0   Inf          -0.002476
6   9.0e-06     -0.002476   3000000000            0   Inf          -0.002485
  relative_time_diff idx
1           1.023021   1
2           1.002827   2
3           1.000000   3
4           1.000000   4
5           1.000000   5
6           1.003635   6
#+end_example

#+begin_src R :file images/trace_gettimeofday1_16.png :results value graphics :session *R* :exports both
library(ggplot2)
ggplot(results, aes(x=idx, y=real_time, color=factor(file))) +
    geom_point(shape=1) + ggtitle("Durations of HPL_dgemm")
#+end_src

#+RESULTS:
[[file:images/trace_gettimeofday1_16.png]]


This is the plot of the duration of =HPL_dgemm= over time (analogous to the plot =duration= vs =start= that we had). The part
for =hpl_pduptatett= looks exactly as before. We see that the calls to =HPL_dgemm= in =hpl_pdrpanllt= are always very short.

#+begin_src R :file images/trace_gettimeofday2_16.png :results value graphics :session *R* :exports both
library(ggplot2)
ggplot(results, aes(x=size_product, y=real_time, color=factor(rank))) +
    geom_point(shape=1) + ggtitle("Durations of HPL_dgemm")
#+end_src

#+RESULTS:
[[file:images/trace_gettimeofday2_16.png]]

Without surprise, we find exactly the same kind of plot as before, since all the new calls to =HPL_dgemm= are very short
and thus hidden in the left part of the graph.


#+begin_src R :results output :session *R* :exports both
reg <- lm(duration~I(m*n*k), data=result)
summary(reg)
#+end_src

#+RESULTS:
#+begin_example

Call:
lm(formula = duration ~ I(m * n * k), data = result)

Residuals:
      Min        1Q    Median        3Q       Max 
-0.004843 -0.001337 -0.000024  0.000280  0.055746 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept)  2.393e-04  2.182e-04   1.097    0.273    
I(m * n * k) 1.064e-09  2.615e-12 406.932   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.003594 on 634 degrees of freedom
Multiple R-squared:  0.9962,	Adjusted R-squared:  0.9962 
F-statistic: 1.656e+05 on 1 and 634 DF,  p-value: < 2.2e-16
#+end_example

#+begin_src R :file images/reg_gettimeofday_16.png :results value graphics :results output :session *R* :exports both
layout(matrix(c(1,2,3,4),2,2))
plot(reg)
#+end_src

#+RESULTS:
[[file:images/reg_gettimeofday_16.png]]

The summary of the linear regression shows that the factor =m*n*k= barely changed. The intercept is very different, but
its t-value is too low, so it is not meaning-full.
The residuals vs fitted plot seems to look better, with no more heteroscedasticity. My guess is that we added a lot of
points with very low values, so their weight hide the problem.
The QQ-plot still looks problematic.
**** Replacing =HPL_dgemm= by =smpi_usleep= again       :SMPI:PERFORMANCE:HPL:
- As for the =printf=, we will put the =smpi_usleep= in the =#define=. We take the coefficients of the latest linear regression.
- Testing: we still get the same number of Gflops (about 23 Gflops) but the simulation runs in 41 seconds now.
*** 2017-03-10 Friday
**** Tracing =HPL_dtrsm=        :C:PYTHON:R:EXPERIMENTS:TRACING:PERFORMANCE:
- The goal is to do something similar for =HPL_dtrsm=. In a first time, we will trace the parameters used to call it and
  its durations, then we will do a linear regression, to finally replace it by a =smpi_usleep=.
- Recall that this function solves a triangular set of equations. It takes as input two m \times n matrices. We expect the
  complexity to be O(m*n).
- Replace the definition of =HPL_dtrsm= in =hpl_blas.h= by the following:
#+begin_src c
#define HPL_dtrsm(layout, Side, Uplo, TransA, Diag, M, N, alpha, A, lda, B, ldb) ({\
    int my_rank, buff=0;\
    MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);\
    struct timeval before = {};\
    struct timeval after = {};\
    gettimeofday(&before, NULL);\
    cblas_dtrsm(layout, Side, Uplo, TransA, Diag, M, N, alpha, A, lda, B, ldb);\
    gettimeofday(&after, NULL);\
    double time_before = (double)(before.tv_sec) + (double)(before.tv_usec)*1e-6;\
    double time_after = (double)(after.tv_sec) + (double)(after.tv_usec)*1e-6;\
    double real_time = time_after-time_before;\
    printf("file=%s line=%d rank=%d m=%d n=%d lead_A=%d lead_B=%d real_time=%f\n", __FILE__, __LINE__, my_rank, M, N, lda, ldb, real_time);\
})
#+end_src
- Run the simulation:
#+begin_src sh
smpirun --cfg=smpi/bcast:mpich --cfg=smpi/running-power:6217956542.969 --cfg=smpi/display-timing:yes\
--cfg=smpi/privatize-global-variables:yes -np 16 -hostfile ../../../small_tests/hostfile_64.txt -platform\
../../../small_tests/cluster_fat_tree_64.xml ./xhpl > /tmp/output
#+end_src
- Process the output file:
#+begin_src python
import re
import csv
reg = re.compile('file=([a-zA-Z0-9/_.-]+) line=([0-9]+) rank=([0-9]+) m=([0-9]+) n=([0-9]+) lead_A=([0-9]+) lead_B=([0-9]+) real_time=(-?[0-9]+.[0-9]+)')

def process(in_file, out_file):
    with open(in_file, 'r') as in_f:
        with open(out_file, 'w') as out_f:
            csv_writer = csv.writer(out_f)
            csv_writer.writerow(('file', 'line', 'rank', 'n', 'm', 'lead_A', 'lead_B', 'real_time'))
            for line in in_f:
                match = reg.match(line)
                if match is not None:
                    result = list(match.group(i) for i in range(1, 9))
                    result[0] = result[0][result[0].index('/hpl'):].lower()
                    csv_writer.writerow(result)
process('/tmp/output', '/tmp/parameters.csv')
#+end_src

- Analysis in R:
#+begin_src R :results output :session *R* :exports both
results <- read.csv("/tmp/parameters.csv");
head(results)
#+end_src

#+RESULTS:
:                                 file line rank  n m lead_A lead_B real_time
: 1 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  171    8 60 0    120    120  0.000102
: 2 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  171    8 30 0    120    120  0.000013
: 3 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  171    8 16 0    120    120  0.000000
: 4 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  171    8  8 0    120    120  0.000000
: 5 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  171    8  4 0    120    120  0.000000
: 6 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  171    8  2 0    120    120  0.000000

#+begin_src R :results output :session *R* :exports both
process_results = function(results) {
    storage.mode(results$m) <- "double" # avoiding integer overflow when taking the product
    storage.mode(results$n) <- "double"
    storage.mode(results$lead_A) <- "double"
    storage.mode(results$lead_B) <- "double"
    results$lead_product = results$lead_A * results$lead_B
    results$size_product = results$m * results$n
    results$ratio = results$lead_product/results$size_product
 #  results$absolute_time_diff = results$expected_time - results$real_time
 #  results$relative_time_diff = (results$expected_time - results$real_time)/results$expected_time
    results$idx = 1:length(results$rank)
    return(results);
}
#+end_src

#+begin_src R :results output :session *R* :exports both
results = process_results(results)
head(results)
#+end_src

#+RESULTS:
#+begin_example
                                file line rank  n m lead_A lead_B real_time
1 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  171    8 60 0    120    120  0.000102
2 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  171    8 30 0    120    120  0.000013
3 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  171    8 16 0    120    120  0.000000
4 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  171    8  8 0    120    120  0.000000
5 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  171    8  4 0    120    120  0.000000
6 /hpl-2.2/src/pfact/hpl_pdrpanllt.c  171    8  2 0    120    120  0.000000
  lead_product size_product ratio idx
1        14400            0   Inf   1
2        14400            0   Inf   2
3        14400            0   Inf   3
4        14400            0   Inf   4
5        14400            0   Inf   5
6        14400            0   Inf   6
#+end_example

#+begin_src R :file images/trace_dtrsm1_16.png :results value graphics :session *R* :exports both
library(ggplot2)
ggplot(results, aes(x=idx, y=real_time, color=factor(file))) +
    geom_point(shape=1) + ggtitle("Durations of HPL_dgemm")
#+end_src

#+RESULTS:
[[file:images/trace_dtrsm1_16.png]]

We can observe a trend similar to =HPL_dgemm=. The function is only used in two places, =HPL_pdrpanllT= and
=HPL_pdupdateTT=. In the former, all the calls are very short, whereas in the later, the calls are long at the beginning
and become shorter throughout the execution. We also have some outliers.

#+begin_src R :file images/trace_dtrsm2_16.png :results value graphics :session *R* :exports both
library(ggplot2)
ggplot(results, aes(x=size_product, y=real_time, color=factor(rank))) +
    geom_point(shape=1) + ggtitle("Durations of HPL_dgemm")
#+end_src

#+RESULTS:
[[file:images/trace_dtrsm2_16.png]]

As expected, the duration looks proportional to the product of the sizes.


#+begin_src R :results output :session *R* :exports both
reg <- lm(real_time~I(m*n), data=results)
summary(reg)
#+end_src

#+RESULTS:
#+begin_example

Call:
lm(formula = real_time ~ I(m * n), data = results)

Residuals:
      Min        1Q    Median        3Q       Max 
-0.002999  0.000010  0.000010  0.000010  0.043651 

Coefficients:
              Estimate Std. Error  t value Pr(>|t|)    
(Intercept) -1.042e-05  2.445e-06   -4.263 2.02e-05 ***
I(m * n)     9.246e-08  3.915e-11 2361.957  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.0006885 on 81298 degrees of freedom
Multiple R-squared:  0.9856,	Adjusted R-squared:  0.9856 
F-statistic: 5.579e+06 on 1 and 81298 DF,  p-value: < 2.2e-16
#+end_example

#+begin_src R :file images/reg_dtrsm_16.png :results value graphics :results output :session *R* :exports both
layout(matrix(c(1,2,3,4),2,2))
plot(reg)
#+end_src

#+RESULTS:
[[file:images/reg_dtrsm_16.png]]

The R-squared is high and both the intercept and sizes have a significant impact.
However, the outliers are even more concerning than with =HPL_dgemm=. The Q-Q plot shows a large tail, and the residual vs
leverage shows that these outliers are non-negligible in the linear regression (i.e. if we removed them, the
coefficients would change significantly).
**** Replacing =HPL_dtrsm= by =smpi_sleep=              :SMPI:PERFORMANCE:HPL:
- Similarly to what have been done with =HPL_dgemm=, we use the coefficients found with the linear regression to replace
  the function by a sleep.
#+begin_src c
#define HPL_dtrsm(layout, Side, Uplo, TransA, Diag, M, N, alpha, A, lda, B, ldb) ({\
    double expected_time = (9.246e-08)*(double)M*(double)N - 1.024e-05;\
    if(expected_time > 0)\
        smpi_usleep((useconds_t)(expected_time*1e6));\
})
#+end_src
- Running HPL again. We get the expected speed (about 23 Gflops) and a simulation time of 29 seconds (gain of 12 seconds).
**** Having a look at =malloc=                    :PYTHON:R:PERFORMANCE:HPL:
- To run HPL with larger matrices, we need to replace some calls to =malloc= (resp. =free=) by =SMPI_SHARED_MALLOC=
  (resp. =SMPI_SHARED_FREE=).
- Firstly, let’s see where the big allocations are.
- Define =MY_MALLOC= in =hpl.h= as follows:
#+begin_src c
#define MY_MALLOC(n) ({\
    int my_rank;\
    MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);\
    printf("file=%s line=%d rank=%d size=%lu\n", __FILE__, __LINE__, my_rank, n);\
    malloc(n);\
})
#+end_src
- Replace all the calls to =malloc= in the files by =MY_MALLOC=:
#+begin_src sh
grep -l malloc testing/**/*.c src/**/*.c | xargs sed -i 's/malloc/MY_MALLOC/g'
#+end_src
- Run =smpirun= (N=20000, P=Q=4) and redirect the output to =/tmp/output=.
- Process the output file:
#+begin_src python
import re
import csv
reg = re.compile('file=([a-zA-Z0-9/_.-]+) line=([0-9]+) rank=([0-9]+) size=([0-9]+)')

def process(in_file, out_file):
    with open(in_file, 'r') as in_f:
        with open(out_file, 'w') as out_f:
            csv_writer = csv.writer(out_f)
            csv_writer.writerow(('file', 'line', 'rank', 'size'))
            for line in in_f:
                match = reg.match(line)
                if match is not None:
                    result = list(match.group(i) for i in range(1, 5))
                    result[0] = result[0][result[0].index('/hpl'):].lower()
                    csv_writer.writerow(result)
process('/tmp/output', '/tmp/malloc.csv')
#+end_src
- Analysis in R:
#+begin_src R :results output :session *R* :exports both
results <- read.csv("/tmp/malloc.csv");
head(results)
#+end_src

#+RESULTS:
:                             file line rank size
: 1 /hpl-2.2/src/grid/hpl_reduce.c  127    0    4
: 2 /hpl-2.2/src/grid/hpl_reduce.c  127    1    4
: 3 /hpl-2.2/src/grid/hpl_reduce.c  127    2    4
: 4 /hpl-2.2/src/grid/hpl_reduce.c  127    3    4
: 5 /hpl-2.2/src/grid/hpl_reduce.c  127    4    4
: 6 /hpl-2.2/src/grid/hpl_reduce.c  127    5    4

#+begin_src R :file images/trace_malloc1_16.png :results value graphics :session *R* :exports both
library(ggplot2)
ggplot(results, aes(x=file, y=size)) +
    geom_boxplot() + ggtitle("Sizes of malloc") + theme(axis.text.x = element_text(angle = 90, hjust = 1))
#+end_src

#+RESULTS:
[[file:images/trace_malloc1_16.png]]

#+begin_src R :results output :session *R* :exports both
storage.mode(results$size) <- "double" # avoiding integer overflow when taking the product
aggregated_results = aggregate(results$size, by=list(file=results$file), FUN=sum)
head(aggregated_results)
#+end_src

#+RESULTS:
:                                    file           x
: 1         /hpl-2.2/src/comm/hpl_packl.c     9034816
: 2        /hpl-2.2/src/grid/hpl_reduce.c     3200736
: 3 /hpl-2.2/src/panel/hpl_pdpanel_init.c 11592866048
: 4  /hpl-2.2/src/panel/hpl_pdpanel_new.c        3456
: 5     /hpl-2.2/src/pauxil/hpl_pdlange.c     2560032
: 6       /hpl-2.2/src/pfact/hpl_pdfact.c     2645504

#+begin_src R :file images/trace_malloc2_16.png :results value graphics :session *R* :exports both
library(ggplot2)
ggplot(aggregated_results, aes(x=file, y=x)) +
    geom_boxplot() + ggtitle("Sizes of malloc") + theme(axis.text.x = element_text(angle = 90, hjust = 1))
#+end_src

#+RESULTS:
[[file:images/trace_malloc2_16.png]]


There are several things to notice:
- The biggest chunks are allocated in =HPL_pdtest=. These are the local matrices of each process.
- However, regarding the total quantity of allocated memory, =HPL_pdpanel_init= is the clear winner.
- In these tests, =htop= reported that about 20% of the 16GB of my laptop’s memory were used, i.e. about 3.2GB. We use a
  matrix of size 20000, each element is of type =double= (8 bytes), so the total amount of memory for the whole matrix is
  20000^2/8 = 3.2GB.
- Thus, it seems that the =malloc= used in =HPL_pdpanel_init= are in fact negligible. An hypothesis is that they are quickly
  followed by a =free=.
- Verifying that every process allocates the same thing:

#+begin_src R :file images/trace_malloc3_16.png :results value graphics :session *R* :exports both
library(ggplot2)
ggplot(results[results$file == "/hpl-2.2/testing/ptest/hpl_pdtest.c",], aes(x="", y=size, fill=factor(rank))) +
    coord_polar("y", start=0) +
    geom_bar(width=1, stat="identity") +
    ggtitle("Sizes of malloc in HPL_pdtest")
#+end_src

#+RESULTS:
[[file:images/trace_malloc3_16.png]]

#+begin_src R :results output :session *R* :exports both
res_pdtest = results[results$file == "/hpl-2.2/testing/ptest/hpl_pdtest.c",]
unique(res_pdtest[order(res_pdtest$size),]$size)
#+end_src

#+RESULTS:
: [1] 193729992 196879432 198454152 200080072 201680392 203293512

- The different calls to =malloc= in =HPL_pdtest= have approximately the same size, but not exactly. This understandable, P
  and Q may not divide the matrix sizes. Maybe this could cause =SMPI_SHARED_MALLOC= to not work properly?
**** Tentative to use =SMIP_SHARED_MALLOC= and =SMPI_SHARED_FREE= in HPL :SMPI:PERFORMANCE:BUG:HPL:
- Revert the previous changes regarding =malloc=.
- In file =hpl_pdtest.c=, replace =malloc= by =SMPI_SHARED_MALLOC= and =free= by =SMPI_SHARED_FREE=.
- Run HPL with Simgrid. Two issues:
  + The memory consumption stays the same, about 20% of my laptop’s memory. A first guess would be that the
    =SHARED_MALLOC= did not work, a new allocation was made for every process. Maybe because different sizes were given?
  + The execution time (both virutal and real) decreased significantly. The virtual time dropped from 233 to 223 seconds,
    the real time from 28 to 15 seconds. If we forget the first point, a guess could be that =SHARED_MALLOC= worked
    properly and resulted in a lower number of cache misses (since all processes share the same sub-matrix) and thus
    improved performances. It is an experimental bias, we should avoid it.
  The fact that we have these two issues combined is very surprising.
- Let’s try to see if the =SHARED_MALLOC= makes only one allocation or not, by adding some =printf= in its implementation.
  + The path =shmalloc_global= is taken.
  + The =bogusfile= is created only once, as expected.
  + Then, every process maps the file in memory, chunk by chunk. The base adress is not the same for every process, but
    this is not an issue (we are speaking of virtual memory here).
- Tested my matrix product program. Got 34% memory utilization, 44 virtual seconds and 8 real seconds with =SMPI_SHARED_MALLOC=, but 11% memory utilization, 81
  virtual seconds and 7 real seconds with =malloc=. Very strange.
- Hypothesis: either the measure of the memory consumption is broken, or =SHARED_MALLOC= is broken.
- Try to use something else than =htop=:
  #+begin_src sh
  watch -n 0,1 cat /proc/meminfo
  #+end_src
  + With =malloc= and =free=, the available memory drop from 14.4 GB to 11.0 GB.
  + With =SMPI_SHARED_MALLOC= and =SMPI_SHARED_FREE=, the available memory drop from 14.4 GB to 14.1 GB.
  This seems more coherent, so =htop= would be a bad tool to measure memory consumption when using =SMPI_SHARED_MALLOC=.
  But this does not solve the time issue.
*** 2017-03-12 Sunday
**** Experiment with SMPI macros in the matrix product code :C:R:EXPERIMENTS:PERFORMANCE:
- Use the matrix product code, at commit =91633ea99463109736b900c92f2eacc84630e5b5=. Run 10 tests with or without
  =SMPI_SHARED_MALLOC= and =SMPI_SAMPLE= with a matrix size of 4000 and 64 processes, by running the command:
  #+begin_src sh
  ./smpi_macros.py 10 /tmp/results.csv
  #+end_src
- Analysis, in R:
#+begin_src R :results output :session *R* :exports both
results <- read.csv("/tmp/results.csv");
head(results)
#+end_src

#+RESULTS:
:       time size smpi_sample smpi_malloc
: 1 2.134820 4000           1           1
: 2 2.608971 4000           0           0
: 3 3.767625 4000           1           0
: 4 2.412387 4000           0           1
: 5 3.767162 4000           1           0
: 6 2.497480 4000           0           0

We already see that the case where we use =SMPI_SAMPLE= but not =SMPI_SHARED_MALLOC= seems to be different than the others.

#+begin_src R :results output :session *R* :exports both
res_aov = aov(time~(smpi_sample + smpi_malloc)^2, data=results)
summary(res_aov)
#+end_src

#+RESULTS:
:                         Df Sum Sq Mean Sq F value   Pr(>F)    
: smpi_sample              1  1.202   1.202   9.227  0.00442 ** 
: smpi_malloc              1  4.579   4.579  35.163 8.62e-07 ***
: smpi_sample:smpi_malloc  1  8.332   8.332  63.981 1.68e-09 ***
: Residuals               36  4.688   0.130                     
: ---
: Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

#+begin_src R :file images/smpi_macros_1.png :results value graphics :results output :session *R* :exports both
suppressWarnings(suppressMessages(library(FrF2))) # FrF2 outputs a bunch of useless messages...
MEPlot(res_aov, abbrev=4, select=c(1, 2), response="time")
#+end_src

#+RESULTS:
[[file:images/smpi_macros_1.png]]

#+begin_src R :file images/smpi_macros_2.png :results value graphics :results output :session *R* :exports both
IAPlot(res_aov, abbrev=4, show.alias=FALSE, select=c(1, 2))
#+end_src

#+RESULTS:
[[file:images/smpi_macros_2.png]]

#+begin_src R :results output :session *R* :exports both
mean(results[results$smpi_sample == 0 & results$smpi_malloc == 0,]$time)
mean(results[results$smpi_sample == 0 & results$smpi_malloc == 1,]$time)
mean(results[results$smpi_sample == 1 & results$smpi_malloc == 0,]$time)
mean(results[results$smpi_sample == 1 & results$smpi_malloc == 1,]$time)
#+end_src

#+RESULTS:
: [1] 2.513953
: [1] 2.750056
: [1] 3.773385
: [1] 2.183901

- In this small experiment, we see that both macros have a non-negligible impact on the time estimated by SMPI. When
  none of the optimizations is used, adding one of them will decreases the application’s performances. When one of the
  optimizations is already used, adding the other one increases the application’s performances.
- When I have added the SMPI macros in =matmul.c=, I have firstly added =SMPI_SHARED_MALLOC= and then =SMPI_SAMPLE_GLOBAL=
  (see the entry for 13/02/2017). According to the tests above, here the variation is not huge (I did not try the
  configuration with =SMPI_SAMPLE_GLOBAL= and without =SMPI_SHARED_MALLOC=). Furthermore, I did not perform extensive
  tests. This may explain why I did not notice this sooner.
*** 2017-03-13 Monday
**** Let’s play with Grid 5000                                       :G5K:
- Connect to Grenoble’s site:
  #+begin_src sh
  ssh tocornebize@access.grid5000.fr
  ssh grenoble
  #+end_src
- Reserve a node and deploy:
  #+begin_src sh
  oarsub -I -l nodes=1,walltime=7 -t deploy
  kadeploy3 -f $OAR_NODE_FILE -e jessie-x64-big -k
  #+end_src
- Connect as root on the new node:
  #+begin_src sh
  ssh root@genepi-33.grenoble.grid5000.fr
  #+end_src
- Install Simgrid:
  #+begin_src sh
  wget https://github.com/simgrid/simgrid/archive/c8db21208f3436c35d3fdf5a875a0059719bff43.zip -O simgrid.zip
  unzip simgrid.zip
  cd simgrid-*
  mkdir build
  cd build
  cmake -Denable_documentation=OFF ..
  make -j 8
  make install
  #+end_src
- Copy HPL on the machine, with =scp=.
- Change the variable =TOPDIR= in the file =Make.SMPI=.
- Do not forget to clean HPL directory when copying it, otherwise the modification of the variable =TOPDIR= will not be
  applied on the sub-makefiles.
- Success of compilation and execution of HPL with Simgrid on one Grid5000 node.
- Strange thing: the virtual time did not change much (228 seconds, or 23.3 Gflops), although the simulation time
  changed a lot (50 seconds, against 15 seconds on my laptop) and I used the same value for the option =running-power=.
**** Scrit for automatic installation                          :SHELL:G5K:
- A small bash script to install Simgrid and compile HPL. Store it in file =deploy.sh=. It assume that archives for
  Simgrid and HPL are located in =/home/tocornebize=.
  #+begin_src sh
function abort {
    echo -e "\e[1;31m Error:" $1 "\e[0m"
    exit 1
}

rm -rf hpl* simgrid*
cp /home/tocornebize/{hpl,simgrid}.zip . &&\
unzip hpl.zip
unzip simgrid.zip
if [ $? -ne 0 ]
then
    abort "Could not copy or extract the archives."
fi

echo ""
echo -e "\e[1;34m Installing Simgrid\e[0m"
cd simgrid* &&\
mkdir build &&\
cd build &&\
cmake -Denable_documentation=OFF .. &&\
make -j 8 &&\
make install &&\
cd ../..
if [ $? -ne 0 ]
then
    abort "Could not install Simgrid."
fi

echo ""
echo -e "\e[1;34m Installing HPL\e[0m"
cd hpl* &&\
sed -ri "s|TOPdir\s+=.+|TOPdir="`pwd`"|g" Make.SMPI &&\ # fixing TOPdir variable
make startup -j 8 arch=SMPI &&\
make -j 8 arch=SMPI &&\
cd ..
if [ $? -ne 0 ]
then
    abort "Could not compile HPL."
fi

echo ""
echo -e "\e[1;32m Everything was ok\e[0m"
  #+end_src
- Given a node obtained with =oarsub= and =kadeploy3=, connect in ssh to it. Then, just run:
  #+begin_src sh
  /home/tocornebize/deploy.sh
  #+end_src
**** Recurrent failure in HPL with =SMPI_SHARED_MALLOC=       :SMPI:BUG:HPL:
- The following error often happens when runing HPL with =SMPI_SHARED_MALLOC=:
  #+begin_example
  src/simix/smx_global.cpp:557: [simix_kernel/CRITICAL] Oops ! Deadlock or code not perfectly clean.
  #+end_example
- It does not seem to happen without =SMPI_SHARED_MALLOC=.
- It does not always happen with =SMPI_SHARED_MALLOC=.
- I do not understand what is happening.
**** Another failure in HPL with =SMPI_SHARED_MALLOC=         :SMPI:BUG:HPL:
- Similarly, the tests on the matrix at the end of HPL are never computed when we use =SMPI_SHARED_MALLOC, because of an
  error. For instance:
  #+begin_example
  HPL ERROR from process # 0, on line 331 of function HPL_pdtest:
  >>> Error code returned by solve is 1021, skip <<<
  #+end_example
- Example of error code: 1021, 1322, 1324, 1575... These values appear nowhere in the code.
**** Tracking the error in HPL
- Put some =printf= to track the error.
*** 2017-03-14 Tuesday
**** Keep tracking the error.                       :SMPI:GIT:TRACING:BUG:
- Add the option =--cfg=smpi/simulate-computation:0= to have a deterministic execution.
- The error code is the field =info= of the matrix. It is modified in the execution path =HPL_pdgesv= \to =HPL_pdgesv0= \to
  =HPL_pdpanel_free=, by the following line:
  #+begin_src c
  if( PANEL->pmat->info == 0 ) PANEL->pmat->info = *(PANEL->DINFO);
  #+end_src
  Thus, we now have to track the values of the =DINFO= field in the panel.
- Strange thing, the field =DINFO= is a pointer to a =float=.
- To track this, use this function:
  #+begin_src c
  void print_info(HPL_T_panel *PANEL, int line) {
     if(PANEL->grid->myrow == 0 && PANEL->grid->mycol == 0) {
          printf("info = %f, line = %d\n", *PANEL->DINFO, line);
     }
  }
  #+end_src
  Put some calls to it at nearly every line of the target file (when you are done with a file, remove these calls).
- Field =DINFO= is modified in the execution path =HPL_pdgesv0= \to =HPL_pdfact= \to =panel->algo->rffun=. The pointer =rffun= is
  one of the functions =HPL_pdrpan***=. In our settings, =HPL_pdrpanllT= is used.
- Field =DINFO= is modified by =PANEL->algo->pffun=, which is one of the functions =HPL_pdpan***=. In our settings,
  =HPL_pdpanllT= is used.
- Then it is modified by the first call to =HPL_dlocswpT=. This function directly modifies the value of =DINFO= with the
  line:
  #+begin_src c
  if( *(PANEL->DINFO) == 0.0 )
     *(PANEL->DINFO) = (double)(PANEL->ia + JJ + 1);
  #+end_src
- If we remove this line, as expected the message about the error code disappears. So it confirms the error code come
  from here.
- Looking at =HPL_pdpanel_init.c=,
  + =DINFO= is a pointer to a part of =DPIV=:
    #+begin_src c
    PANEL->DINFO = PANEL->DPIV + JB;
    #+end_src
  + =DPIV= is a pointer to a part of =L1=:
    #+begin_src c
    PANEL->DPIV  = PANEL->L1    + JB * JB;
    #+end_src
  + =L1= is an (aligned) alias for =WORK=, which is itself a block of memory allocated with =malloc=:
    #+begin_src c
    PANEL->WORK = (void*) malloc((size_t)(lwork) * sizeof(double));
    // [...]
    PANEL->L1    = (double *)HPL_PTR( PANEL->WORK, dalign );
    #+end_src
  L1 is the jb \times jb upper block of the local matrix. It is used for computations. Thus, it seems that HPL expects a
  particular cell of this local matrix to have the value 0. This cell is not always the same.
  Interpretation: HPL is checking that the matrix is correctly factorized (it uses LU factorization, so it computes L
  and U such that A=LU, L is lower-triangular and U is upper-triangular). Since we use shared memory, it is not
  surprising that the correctness check do not pass anymore.
  What is more surprising is that this particular check was still passing when the two BLAS functions were replaced by
  =smpi_usleep=. A guess: the fact that the resulting matrices are triangular only depends on the correctness of the
  swapping of rows.
- Thus, it seems that the error code is explained. This is a normal behavior, considered what we are doing.
- The deadlock happening in some executions is not explained however.
**** Webinar                                                     :MEETING:
- Enabling open and reproducible research at computer system’s conferences: good, bad and ugly
- Grigori Fursin
- The speaker created an [[http://ctuning.org/][organization]] about reproducible research.
- Artifact evaluation is about peer review of experiments.
- How it works: papers accepted to a conference can ask for an artifact evaluation. If they pass it, they would get a
  nice stamp on the paper. If they fail it, nobody will know. View this as a bonus for a paper. For the evaluation of
  the artifacts, the conference nominates several reviewers.
- ACM conferences also start using this kind of things, with several different stamps.
- But artifact evaluation is not easy to do. Firstly, there is a lot of artifacts to evaluate, hard to scale. Some
  artifact evaluations require proprietary software and/or rare hardware (e.g. supercomputers). Also, hard to find a
  reviewer with suitable skills for some cases.
- Also, it is difficult to reproduce empirical results (changing software and hardware). Everyone has its own scripts,
  so hard to standardize a universal workflow.
- Other [[http://cknowledge.org][website]].
*** 2017-03-15 Wednesday
**** Hunting the deadlock                  :SMPI:PYTHON:R:TRACING:BUG:HPL:
- With N=40000, P=Q=4 and the option =--cfg=smpi/simulate-computation:0=, it seems we always have a deadlock.
- Let’s trace it, with option =-trace -trace-file /tmp/trace --cfg=smpi/trace-call-location:1=.
- Processing the trace file:
#+begin_src sh
pj_dump --user-defined --ignore-incomplete-links /tmp/trace > /tmp/trace.csv
grep "State," /tmp/trace.csv | sed -e 's/()//' -e 's/MPI_STATE, //ig'  -e 's/State, //ig' -e 's/rank-//' -e\
's/PMPI_/MPI_/' | grep MPI_  | tr 'A-Z' 'a-z' > /tmp/trace_processed.csv
#+end_src

Clean the paths:
#+begin_src python
import re
reg = re.compile('((?:[^/])*)(?:/[a-zA-Z0-9_-]*)*((?:/hpl-2.2(?:/[a-zA-Z0-9_-]*)*).*)')
def process(in_file, out_file):
    with open(in_file, 'r') as in_f:
        with open(out_file, 'w') as out_f:
            for line in in_f:
                match = reg.match(line)
                out_f.write('%s%s\n' % (match.group(1), match.group(2)))
process('/tmp/trace_processed.csv', '/tmp/trace_cleaned.csv')
 #+end_src

Analysis:
#+begin_src R :results output :session *R* :exports both
trace <- read.csv("/tmp/trace_cleaned.csv", header=F, strip.white=T, sep=",");
names(trace) = c("rank", "start", "end", "duration", "level", "state", "Filename", "Linenumber");
trace$idx = 1:length(trace$rank)
head(trace)
#+end_src

#+RESULTS:
#+begin_example
  rank    start      end duration level    state
1    8 0.000000 0.000000 0.000000     0 mpi_init
2    8 0.000000 0.000202 0.000202     0 mpi_recv
3    8 0.000202 0.000403 0.000201     0 mpi_recv
4    8 0.000403 0.000806 0.000403     0 mpi_recv
5    8 0.000806 0.000806 0.000000     0 mpi_send
6    8 0.000806 0.001612 0.000806     0 mpi_recv
                               Filename Linenumber idx
1 /hpl-2.2/testing/ptest/hpl_pddriver.c        109   1
2        /hpl-2.2/src/grid/hpl_reduce.c        165   2
3        /hpl-2.2/src/grid/hpl_reduce.c        165   3
4        /hpl-2.2/src/grid/hpl_reduce.c        165   4
5        /hpl-2.2/src/grid/hpl_reduce.c        159   5
6     /hpl-2.2/src/grid/hpl_broadcast.c        130   6
#+end_example

#+begin_src R :results output :session *R* :exports both
get_last_event = function(df) {
    result = data.frame() 
    for(rank in (sort(unique(trace$rank)))) {
        tmp_trace = trace[trace$rank == rank,]
        result = rbind(result, tmp_trace[which.max(tmp_trace$idx),])
    }
    return(result)
}
get_last_event(trace)[c(1, 2, 3, 6, 7, 8)]
#+end_src

#+RESULTS:
#+begin_example
      rank    start      end    state                         Filename
18756    0 67.01313 67.10575 mpi_recv /hpl-2.2/src/pgesv/hpl_spreadt.c
9391     1 66.84201 67.10575 mpi_recv /hpl-2.2/src/pgesv/hpl_spreadt.c
7865     2 66.92821 67.10575 mpi_recv /hpl-2.2/src/pgesv/hpl_spreadt.c
7048     3 67.01313 67.10575 mpi_recv /hpl-2.2/src/pgesv/hpl_spreadt.c
6242     4 67.08334 67.10575 mpi_send   /hpl-2.2/src/pgesv/hpl_rollt.c
4699     5 66.93228 67.10575 mpi_wait   /hpl-2.2/src/pgesv/hpl_rollt.c
3174     6 67.02313 67.10575 mpi_wait   /hpl-2.2/src/pgesv/hpl_rollt.c
2358     7 67.08334 67.10575 mpi_send   /hpl-2.2/src/pgesv/hpl_rollt.c
1554     8 67.08334 67.10575 mpi_recv /hpl-2.2/src/pgesv/hpl_spreadt.c
17201    9 66.93228 67.10575 mpi_send /hpl-2.2/src/pgesv/hpl_spreadt.c
15675   10 67.02313 67.10575 mpi_send /hpl-2.2/src/pgesv/hpl_spreadt.c
14858   11 67.08334 67.10575 mpi_recv /hpl-2.2/src/pgesv/hpl_spreadt.c
14053   12 67.06093 67.10575 mpi_recv /hpl-2.2/src/pgesv/hpl_spreadt.c
12516   13 66.88778 67.10575 mpi_recv /hpl-2.2/src/pgesv/hpl_spreadt.c
10998   14 66.97831 67.10575 mpi_recv /hpl-2.2/src/pgesv/hpl_spreadt.c
10189   15 67.06093 67.10575 mpi_recv /hpl-2.2/src/pgesv/hpl_spreadt.c
      Linenumber
18756        321
9391         321
7865         321
7048         321
6242         235
4699         242
3174         242
2358         235
1554         321
17201        351
15675        351
14858        321
14053        321
12516        321
10998        321
10189        321
#+end_example

If the trace is correct, the deadlock happens in functions =HPL_rollT= and =HPL_spreadT=.
Some =printf= confirm that the deadlock is indeed happening in these places.
**** Found the deadlock                                   :SMPI:C:BUG:HPL:
- Let’s add some =printf= in files =HPL_spreadT.c= and =HPL_rollT.c=.
  First, add the functions:
  #+begin_src c
  int local_rank_to_global(int local_rank, MPI_Comm local_communicator) {
      int result;
      MPI_Group local_group, world_group;
      MPI_Comm_group(local_communicator, &local_group);
      MPI_Comm_group(MPI_COMM_WORLD, &world_group);
      MPI_Group_translate_ranks(local_group, 1, &local_rank, world_group, &result);
      return result;
  }
  void print_info(int src_rank, int dst_rank, char *function, int line, char *file) {
      printf("src=%d dst=%d function=%s line=%d file=%s\n", src_rank, dst_rank, function,
  line, file);
  }
  #+end_src
  Then, add a call to =print_info= before each of the four lines we found:
  + =HPL_spreadT.c=, line 321:
    #+begin_src c
    int local_rank = local_rank_to_global(IPMAP[SRCDIST+partner], comm);
    print_info(my_rank, local_rank, "mpi_recv", __LINE__, __FILE__);
    #+end_src
  + =HPL_spreadT.c=, line 351:
    #+begin_src c
    int local_rank = local_rank_to_global(IPMAP[SRCDIST+partner], comm);
    print_info(my_rank, local_rank, "mpi_send", __LINE__, __FILE__);
    #+end_src
  + =HPL_rollT.c=, line 235:
    #+begin_src c
    int local_rank = local_rank_to_global(partner, comm);
    print_info(my_rank, local_rank, "mpi_send", __LINE__, __FILE__);
    #+end_src
  + =HPL_rollT.c=, line 242:
    #+begin_src c
    int local_rank = local_rank_to_global(partner, comm);
    print_info(my_rank, local_rank, "mpi_wait", __LINE__, __FILE__);
    #+end_src
- Then, run HPL with =stdout= redirected to a file =/tmp/output=.
- For each rank, look for the last time this rank was the caller of a blocking MPI primitive. For instance, for rank =15=:
  #+begin_src sh
  RANK="15 " && grep "src="$RANK /tmp/output | tail -n 1
  #+end_src
  Observe the destination and the function.
  With P=Q=4, we had these dependencies:
  #+begin_example
           12
            |
  mpi_recv  |
            |
            v     mpi_recv
            4 <——————————————+
            |                |
            |                |
  mpi_wait  |                |
            |                |
            v                |
            8 —————————————> 0
                 mpi_send
  #+end_example
  There is the same pattern for {1, 5, 9, 13}, {2, 6, 10, 14} and {3, 7, 11, 15}.
- This exact deadlock has been reproduced on Grid 5000, with the same parameters.
*** 2017-03-16 Thursday
**** Still looking for the deadlock                         :SMPI:BUG:HPL:
- When HPL is ran with =smpi_usleep= but without =SMPI_SHARED_{MALLOC,FREE}=, there is no deadlock, even with the same parameters (N=40000,
  P=Q=4). Warning: testing with N=40000 require a lot of memory, about 12GB.
- When HPL is ran with =SMPI_SHARED_{MALLOC,FREE}= but without =smpi_usleep=, there is a deadlock. Note that we still use
  the option =--cfg=smpi/simulate-computation:0=. It happens in the same location, but the deadlock is different. Now, it
  is like this (and is located only in =HPL_spreadT=):
  #+begin_example
            4
            |
  mpi_recv  |
            |
            v     mpi_recv
            0 <——————————————+
            |                |
            |                |
  mpi_send  |                |
            |                |
            v                |
           12 —————————————> 8
                 mpi_send
  #+end_example
  There is the same pattern for {1, 5, 9, 13}, {2, 6, 10, 14} and {3, 7, 11, 15}.
**** Understanding HPL code                         :SMPI:TRACING:BUG:HPL:
- In file =HPL_spreadT.c=.
- In our settings, the following =if= statement is never taken:
  #+begin_src c
  if(SIDE == HplLeft)
  #+end_src
- In the =else= part, there is a big =do while= loop. Some initializations happen before this loop.
- =npm1=: initialized to =nprow - SRCDIST - 1=, not modified during the loop.
- =ip2=: initialized to the biggest power of 2 smaller or equal to =npm1=. Divided by =2= at each step. The loop stops when
  =ip2= is =0=.
- =mask=: initialized to =ip2*2-1= (=ip2= is a single bit set to =1= followed by a bunch of =0=, =mask= is the same bit set to =1=
  followed by a bunch of =1=). At the beginning of each step, the first =1= of =mask= is flipped, so =mask= is =ip2-1= after this
  statement.
- =IPMAP=: mapping of the processes.
- =IPMAPM1=: inverse mapping (=IPMAPM1[IPMAP[i]]= is equal to =i=).
- =mydist=: initialized to =IPMAP1[myrow]=, not modified after.
- =partner=: at each step, set to =mydist^ip2=, i.e. we flip exactly one bit of =mydist=.
- We do the communications only when =mydist & mask= is =0= and when =lbuf > 0=.
  + If =mydist & ip2= is not =0=, we receive.
  + If =mydist & ip2= is =0=, we send.
- Print the content of =IPMAP=. Add the following line before the =do while=:
  #+begin_src c
  printf("IPMAP: my_rank=%d, %d %d %d %d \n", my_rank,
    local_rank_to_global(IPMAP[0], comm), local_rank_to_global(IPMAP[1], comm),
    local_rank_to_global(IPMAP[2], comm), local_rank_to_global(IPMAP[3], comm));
  #+end_src
  We get this output:
  #+begin_example
  IPMAP: my_rank=0, 0 4 12 8
  IPMAP: my_rank=12, 0 4 12 8
  IPMAP: my_rank=8, 0 4 8 12
  IPMAP: my_rank=4, 0 4 12 8
  IPMAP: my_rank=0, 0 4 12 8
  IPMAP: my_rank=4, 0 4 12 8
  IPMAP: my_rank=1, 1 5 13 9
  IPMAP: my_rank=5, 1 5 13 9
  IPMAP: my_rank=13, 1 5 13 9
  IPMAP: my_rank=9, 1 5 9 13
  IPMAP: my_rank=5, 1 5 13 9
  IPMAP: my_rank=1, 1 5 13 9
  IPMAP: my_rank=2, 2 6 14 10
  IPMAP: my_rank=3, 3 7 15 11
  IPMAP: my_rank=6, 2 6 14 10
  IPMAP: my_rank=7, 3 7 15 11
  IPMAP: my_rank=10, 2 6 10 14
  IPMAP: my_rank=11, 3 7 11 15
  IPMAP: my_rank=14, 2 6 14 10
  IPMAP: my_rank=15, 3 7 15 11
  IPMAP: my_rank=6, 2 6 14 10
  IPMAP: my_rank=2, 2 6 14 10
  IPMAP: my_rank=7, 3 7 15 11
  IPMAP: my_rank=3, 3 7 15 11
  #+end_example
  Recall that our communicators are {n, n+4, n+8, n+12} for n in {0, 1, 2, 3}. We see a pattern here: when processes
  have a local rank in {0, 1, 3}, their =IPMAP= is {0, 1, 3, 2} (local ranks), but when the local rank is 2, then =IPMAP= is
  {0, 1, 2, 3}.
- Now, let’s print the other parameters. Add the following line just after the modification of =mask= at the beginning of
  the =do while=:
  #+begin_src c
  printf("### my_rank=%d (%d) id_func=%d mask=%d ip2=%d mydist=%d", my_rank,
    my_local_rank, id_func, mask, ip2, mydist);
  #+end_src
  Here, =id_func= is a static variable initialized to =-1= and incremented at the beginning of every function call.
  Later in the code, add these:
  #+begin_src c
  printf(" partner=%d", partner);
  #+end_src
  and
  #+begin_src c
  printf(" mpi_recv(%d)\n", IPMAP[SRCDIST+partner]);
  #+end_src
  or
  #+begin_src c
  printf(" mpi_send(%d)\n", IPMAP[SRCDIST+partner]);
  #+end_src
  (depending on if we do a send or a receive).
  We have this output for {0, 4, 8, 12} (this is similar for other communicators):
  #+begin_src bash
  grep "my_rank=0 " output | grep "###"
  ### my_rank=0 (0) id_func=0 mask=1 ip2=2 mydist=0 partner=2 mpi_send(3)
  ### my_rank=0 (0) id_func=0 mask=0 ip2=1 mydist=0 partner=1 mpi_send(1)
  ### my_rank=0 (0) id_func=1 mask=1 ip2=2 mydist=0 partner=2 mpi_send(3)
  grep "my_rank=4 " output | grep "###"
  ### my_rank=4 (1) id_func=0 mask=1 ip2=2 mydist=1
  ### my_rank=4 (1) id_func=0 mask=0 ip2=1 mydist=1 partner=0 mpi_recv(0)
  ### my_rank=4 (1) id_func=1 mask=1 ip2=2 mydist=1
  ### my_rank=4 (1) id_func=1 mask=0 ip2=1 mydist=1 partner=0 mpi_recv(0)
  grep "my_rank=8 " output | grep "###"
  ### my_rank=8 (2) id_func=0 mask=1 ip2=2 mydist=2 partner=0 mpi_recv(0)
  grep "my_rank=12 " output | grep "###"
  ### my_rank=12 (3) id_func=0 mask=1 ip2=2 mydist=2 partner=0 mpi_recv(0)
  ### my_rank=12 (3) id_func=0 mask=0 ip2=1 mydist=2 partner=3 mpi_send(2)
  #+end_src
  We see that the pattern of communication looks like a binary tree. At each function call, in the first step 0 sends to
  12, in the second step 0 sends to 4 and 12 sends to 8. The problem is that all the =mpi_recv= match the =mpi_send= except
  for the node 8. This node calls =mpi_recv= with node 0 for the source, but we would expect it to have 12 for source.
  The same pattern is observed for other communicators.
- We saw that the nodes with local rank 2 call =MPI_Recv= with an unexpected source. These nodes also have a different
  =IPMAP=. Hypothesis: these different =IPMAP= are a bug.
- Doing the same experiment without =SMPI_SHARED_{MALLOC,FREE}= (the case where we do not have a deadlock). Here, we
  observe that the values of =IPMAP= are the same in all processes. Also, there is a matching =MPI_Recv= for every =MPI_Send=,
  as expected.
- Thus, to fix the deadlock, we should search where =IPMAP= is defined.
**** Seminar                                                     :MEETING:
- Taking advantage of application structure for visual performance analysis
- Lucas Mello Schnorr
- Context: two models. Explicit programming (e.g. MPI) or task-based programming (e.g. Cilk).
- In task-based programming, no clear phases (contrarily to things like MPI, where we have communication phases and
  computation phases). Thus, hard to understand the performances when visualizing a trace.
- The scheduler has to assign tasks, anticipate the critical path and minimize data movements. The difficulty is that it
  does not know the whole DAG at the beginning.
- Workflow based on several tools: =pj_dump=, =R=, =tidyverse=, =ggplot2=, =plotly=. Everything can be done in org-mode. Agile
  workflow, fail fast if the idea is not working, easily share experiments with colleagues.
*** 2017-03-17 Friday
**** Let’s look at =IPMAP=                          :SMPI:C:TRACING:BUG:HPL:
- =IPMAP= is given as an argument to =HPL_spreadT=.
- The function =HPL_spreadT= is usedd in =HPL_pdlaswp01T= and =HPL_equil=.
- In our settings, all processes begin by a call to =HPL_pdlaswp01T=. Then, all processes with local ranks =0= and =1= do a
  call to =HPL_equil= (local ranks =2= and =3= are already deadlocked). Values of =IPMAP= are the same between the two different
  calls.
  We thus have to look at =HPL_pdlaswp01T=.
- =IPMAP= is defined in this function with other variables. They are all a contiguous block in =PANEL->IWORK=:
  #+begin_src c
  iflag  = PANEL->IWORK;
  // [...]
  k = (int)((unsigned int)(jb) << 1);  ipl = iflag + 1; ipID = ipl + 1;
  ipA     = ipID + ((unsigned int)(k) << 1); lindxA = ipA + 1;
  lindxAU = lindxA + k; iplen = lindxAU + k; ipmap = iplen + nprow + 1;
  ipmapm1 = ipmap + nprow; permU = ipmapm1 + nprow; iwork = permU + jb;
  #+end_src
- =PANEL->IWORK= is allocated in =HPL_pdpanel_init= with a simple =malloc=. So the bugs does not come from here.
- The content of =IPMAP= is defined in the function =HPL_plindx10=.
- Function =HPL_plindx10= firstly compute the content of array =IPLEN=, then call function =HPL_logsort= to compute =IPMAP= (the
  content of =IPMAP= depends on the content of =IPLEN=).
- Printing the content of =IPLEN= just after its initialization.
  Add this code just before the call to =HPL_logsort=:
  #+begin_src c
  int my_rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
  printf(">> my_rank=%d, icurrow=%d, IPLEN =", my_rank, icurrow);
  for(i = 0; i <= nprow; i++) {
       printf(" %d", IPLEN[i]);
  }
  printf("\n");
  #+end_src
  Here are the contents of =IPLEN= for ranks {0, 4, 8, 12}.
  + With =SMPI_SHARED_{MALLOC,FREE}=:
    | Rank | IPLEN[0] | IPLEN[1] | IPLEN[2] | IPLEN[3] | IPLEN[4] |
    |------+----------+----------+----------+----------+----------|
    |    0 |        0 |      103 |       14 |        1 |        2 |
    |    4 |        0 |      102 |       15 |        1 |        2 |
    |    8 |        0 |      102 |       14 |        2 |        2 |
    |   12 |        0 |      102 |       14 |        1 |        3 |
  + Without =SMPI_SHARED_{MALLOC,FREE}=:
    | Rank | IPLEN[0] | IPLEN[1] | IPLEN[2] | IPLEN[3] | IPLEN[4] |
    |------+----------+----------+----------+----------+----------|
    |    0 |        0 |       31 |       24 |       26 |       39 |
    |    4 |        0 |       31 |       24 |       26 |       39 |
    |    8 |        0 |       31 |       24 |       26 |       39 |
    |   12 |        0 |       31 |       24 |       26 |       39 |

  We can note two things. Firstly, without =SMPI_SHARED_{MALLOC,FREE}=, all processes have an =IPLEN= with the same
  content. This is not the case with =SMPI_SHARED_{MALLOC,FREE}=. Furthermore, values in =IPLEN= are closer in the
  =malloc/free= case.
  Thus, the issue is very likely to come from =IPLEN=.
**** Let’s look at =IPLEN= and =IPID=                   :SMPI:TRACING:BUG:HPL:
- The content of =IPLEN= depends on the content of =IPID=.
- Add a =printf= to get its content. Every element it contains is present exactly twice in the array.
- With =SHARED_{MALLOC,FREE}=,
  + =IPID= has a size of 300 for local rank =0=, 302 for the others.
  + =IPID= of local rank =1= is equal to =IPID= of local rank =0= plus twice the element =120=.
  + =IPID= of local rank =2= is equal to =IPID= of local rank =0= plus twice the element =240=.
  + =IPID= of local rank =3= is equal to =IPID= of local rank =0= plus twice the element =360=.
- Without =SHARED_{MALLOC,FREE}=,
  + =IPID= has a size of 478 for all ranks.
  + All =IPID= are equal.
- =IPID= is computed in function =HPL_pipid=.
- The content of =IPID= depends on the content of the array =PANEL->DPIV=. This array is made of =120= elements. These
  elements are of type =double=. The function cast them to =int= and do some comparisons using them, which is strange.
- Add a =printf= to get its content.
- With =SHARED_{MALLOC,FREE}=,
  + The =DPIV= of the processes having the same local rank are equal.
  + The 30 first elements of the arrays =DPIV= of the processes of a same communicator are equal. The following elements
    are different.
  + For the processes of local rank =0=, these following elements are 30, 31, 32,..., 119. In other words, for =i > 29=, we
    have =DPIV[i]= equal to =i=.
  + For the processes of local rank =1=, these elements are all equal to 120. For local rank =2=, they are equal to 240. For
    local rank =3=, they are equal to =360.
- Without =SHARED_{MALLOC,FREE}=,
  + All the =DPIV= of all processes are equal.
  + All its elements are present exactly once, except =4143= which is present twice.
– Thus, it seems that the issue come from =PANEL->DPIV=.
**** Summing up                                             :SMPI:BUG:HPL:
- The values of =IPMAP= depends on the values of =IPLEN=.
- The values of =IPLEN= depends on the values of =IPID=.
- The values of =IPID= depends on the values of =PANEL->DPIV=.
- For all these arrays, we can observe some strange things in the case =SMPI_SHARED_{MALLOC,FREE}= (comparing to the case
  =malloc/free=):
  + The content of the arrays is not the same for different ranks.
  + The content itself looks kind of strange (e.g. =DPIV= has a lot of identical values).
**** So, why do we have these =DPIV=?                         :SMPI:BUG:HPL:
- The content of =DPIV= is defined at the end of function =HPL_pdmxswp=, by the line:
  #+begin_src c
  (PANEL->DPIV)[JJ] = WORK[2];
  #+end_src
  With some =printf=, we see that =DPIV= is filled in order. The values are the same that the ones already observed in =DPIV=.
*** 2017-03-20 Monday
**** Write a small Python script to monitor memory usage          :PYTHON:
- Based on command =smemstat=.
- Run the command every second in quiet mode with json output. Parse the json file and output the information on screen,
  nicely formated.
- Future work:
  + Different sampling rate passed as a command line argument.
  + Export in CSV. This would allow to plot memory consumption over time.
**** Failed tentatives for =DPIV=                             :SMPI:BUG:HPL:
- Tried to hard-code the values of =DPIV= with something like that:
  #+begin_src c
  (PANEL->DPIV)[JJ] = 42;
  #+end_src
  Got a segmentation fault.
**** Discussion with Arnaud                         :SMPI:BUG:HPL:MEETING:
- Had a look at HPL code.
- Next steps to try to find the issue:
  + Try another block size for global =SMPI_SHARED_MALLOC=.
  + Retry local =SMPI_SHARED_MALLOC=.
  + Try other matrix sizes, other process grids.
  + In =HPL_pdmxswp=, print the values of =WORK[{0,1,2,3}]= before and after the execution.
**** Looking at =WORK[{0,1,2,3}]=                           :SMPI:C:BUG:HPL:
- Meaning of the first values of this array:
  + =WORK[0]= : local maximum absolute value scalar,
  + =WORK[1]= : corresponding local  row index,
  + =WORK[2]= : corresponding global row index,
  + =WORK[3]= : coordinate of process owning this max.
- Just before the call to =HPL_pdxswp=, these values are computed locally. Then, =HPL_pdxswp= does some computations to get
  the global values.
- Adding some =printf=.
- Without =SHARED_{MALLOC,FREE}=, the absolute value of =WORK[0]= increases at each call and quickly becomes very large. It
  reaches =3.8e+302=, then it is =NaN=. This happens regardless of if we replace BLAS operations by =smpi_usleep=.
- With =SHARED_{MALLOC,FREE}=, the absolute value of =WORK[0]= is relatively small.
- If we replace the value of =WORK[0]= by a (small) constant, the simulations terminates without deadlock.
- Recall that we run the simulation with N=40000 and P=Q=4.
- The simulation takes 197 seconds, where 170 seconds are actual computations of the application (thus, there is still
  room for optimization).
- The estimated performances are 27.5 Gflops. This is a bit higher than what we had before with the matrix of
  size 20000. We need to check if this difference is due to the higher matrix size (expected and ok) or our dirty hack
  (not ok).
*** 2017-03-21 Tuesday
**** Checking the consistency of =IPMAP=       :SMPI:PYTHON:TRACING:BUG:HPL:
- Before the modification on =WORK[0]=, the =IPMAP= were not consistent on the different processes of a same communicator
  (see the entry of =16/03/2017=).
- Let’s check if this issue is fixed.
- Add the following in =HPL_spreadT.c=:
  #+begin_src c
  int my_rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
  printf("my_rank=%d IPMAP=%d,%d,%d,%d\n", my_rank, IPMAP[0], IPMAP[1], IPMAP[2], IPMAP[3]);
  #+end_src
- Run HPL with =stdout= redirected to =/tmp/output=.
- Check that at each step, the values of =IPMAP= are the same for the processes of a same communicator. Recall that the
  communicators are ={0,4,8,12}=, ={1,5,9,13}=, ={2,6,10,14}= and ={3,7,11,15}=.
#+begin_src python :results output :exports both
import re
reg = re.compile('my_rank=([0-9]+) IPMAP=(.+)')

def process(in_file):
    results = {n: [] for n in range(16)}
    with open(in_file, 'r') as in_f:
        for line in in_f:
            match = reg.match(line)
            if match is not None:
                n = int(match.group(1))
                ipmap = match.group(2)
                results[n].append(ipmap)
    for comm in range(4):
        print('Number of entries for communicator %d: %d' % (comm, len(results[comm])))
        for rank in range(1, 4):
            assert results[comm] == results[comm+4*rank]
    print('OK')
process('/tmp/output')
#+end_src

 #+RESULTS:
 : Number of entries for communicator 0: 913
 : Number of entries for communicator 1: 904
 : Number of entries for communicator 2: 907
 : Number of entries for communicator 3: 910
 : OK
- We see here that the values of =IPMAP= are consistent.
**** Comparison with previous code                              :SMPI:HPL:
- Let’s compare with the previous version of the code (without the modification on =WORK[0]=). We use N=20000, P=Q=4.
  | Code   | Virtual time |    Gflops | Total simulation time | Time for application computations |
  |--------+--------------+-----------+-----------------------+-----------------------------------|
  | *Before* |       222.27 | 2.400e+01 |               19.2529 |                           10.0526 |
  | *After*  |       258.28 | 2.065e+01 |               48.2851 |                           41.7249 |
- We find that both the virtual time and the real time are longer, due to an higher ammount of time spent in the
  application.
- Do not forget to remove the option =--cfg=smpi/simulate-computation:0= when testing for things like that. At first, I
  did not removed it. The real time was higher but the virtual time was unchanged.
- It seems that the modification of =WORK[0]= has led to a modification of the behavior of the application, which yields
  significant differences in terms of performances.
**** Having a look at what takes time :SMPI:PYTHON:R:EXPERIMENTS:PERFORMANCE:HPL:
- Using the settings N=20000, P=Q=4. Recall that with these settings, the simulation time was nearly 52 seconds.
- Simulation time drops to 30 seconds if we disable the calls to =HPL_dgemv= (this was not the case before, according to
  experimentations of =08/03/2017=).
- There was no deadlock with N=20000, so we can compare the cases with and without the modification of =WORK[0]=.
- Modify the definition of =HPL_dgemv= in =HPL_blas.h= for both cases:
  #+begin_src c
  #define    HPL_dgemv(Order, TransA, M, N, alpha, A, lda, X, incX, beta, Y, incY) ({\
      int my_rank, buff=0;\
      MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);\
      struct timeval before = {};\
      struct timeval after = {};\
      gettimeofday(&before, NULL);\
      cblas_dgemv(Order, TransA, M, N, alpha, A, lda, X, incX, beta, Y, incY);\
      gettimeofday(&after, NULL);\
      double time_before = (double)(before.tv_sec) + (double)(before.tv_usec)*1e-6;\
      double time_after = (double)(after.tv_sec) + (double)(after.tv_usec)*1e-6;\
      double real_time = time_after-time_before;\
      printf("file=%s line=%d rank=%d m=%d n=%d lead_A=%d inc_X=%d inc_Y=%d real_time=%f\n", __FILE__, __LINE__, my_rank, M, N, lda, incX, incY, real_time);\
  })
  #+end_src
- Run HPL for both cases, with =stdout= redirected to some file (=/tmp/output_before= when =WORK[0]= is unmodified,
  =/tmp/output_after= when it is modified).
- Process the outputs:
#+begin_src python
import re
import csv
reg = re.compile('file=([a-zA-Z0-9/_.-]+) line=([0-9]+) rank=([0-9]+) m=([0-9]+) n=([0-9]+) lead_A=([0-9]+) inc_X=([0-9]+) inc_Y=([0-9]+) real_time=(-?[0-9]+.[0-9]+)')

def process(in_file, out_file):
    with open(in_file, 'r') as in_f:
        with open(out_file, 'w') as out_f:
            csv_writer = csv.writer(out_f)
            csv_writer.writerow(('file', 'line', 'rank', 'm', 'n', 'lead_A', 'inc_X', 'inc_Y', 'real_time'))
            for line in in_f:
                match = reg.match(line)
                if match is not None:
                    result = list(match.group(i) for i in range(1, 10))
                    result[0] = result[0][result[0].index('/hpl'):].lower()
                    csv_writer.writerow(result)
process('/tmp/output_before', '/tmp/parameters_before.csv')
process('/tmp/output_after', '/tmp/parameters_after.csv')
#+end_src

- Analysis with R:
#+begin_src R :results output :session *R* :exports both
parameters_before <- read.csv("/tmp/parameters_before.csv");
parameters_after <- read.csv("/tmp/parameters_after.csv");
head(parameters_before)
head(parameters_after)
#+end_src

#+RESULTS:
#+begin_example
                               file line rank    m n lead_A inc_X inc_Y
1 /hpl-2.2/src/pfact/hpl_pdpanllt.c  207    4 5040 1   5040   120     1
2 /hpl-2.2/src/pfact/hpl_pdpanllt.c  207   12 4920 1   4920   120     1
3 /hpl-2.2/src/pfact/hpl_pdpanllt.c  207    0 5039 1   5040   120     1
4 /hpl-2.2/src/pfact/hpl_pdpanllt.c  207    8 5000 1   5000   120     1
5 /hpl-2.2/src/pfact/hpl_pdpanllt.c  207   12 4920 1   4920   120     1
6 /hpl-2.2/src/pfact/hpl_pdpanllt.c  207    4 5040 1   5040   120     1
  real_time
1  0.000034
2  0.000034
3  0.000030
4  0.000156
5  0.000043
6  0.000031
                               file line rank    m n lead_A inc_X inc_Y
1 /hpl-2.2/src/pfact/hpl_pdpanllt.c  207    4 5040 1   5040   120     1
2 /hpl-2.2/src/pfact/hpl_pdpanllt.c  207   12 4920 1   4920   120     1
3 /hpl-2.2/src/pfact/hpl_pdpanllt.c  207    0 5039 1   5040   120     1
4 /hpl-2.2/src/pfact/hpl_pdpanllt.c  207    8 5000 1   5000   120     1
5 /hpl-2.2/src/pfact/hpl_pdpanllt.c  207   12 4920 1   4920   120     1
6 /hpl-2.2/src/pfact/hpl_pdpanllt.c  207    8 5000 1   5000   120     1
  real_time
1  0.000026
2  0.000030
3  0.000030
4  0.000123
5  0.000030
6  0.000028
#+end_example

#+begin_src R :results output :session *R* :exports both
sum(parameters_before$real_time)
sum(parameters_after$real_time)
#+end_src

#+RESULTS:
: [1] 0.127207
: [1] 2.61086

- There is a clear difference between the two cases. When =WORK[0]= is modified, the time spent in the function =HPL_dgemv=
  is 20 times higher. However, this makes a difference of about 2.5 seconds, whereas a difference of 20 seconds was
  observed when removing =HPL_dgemv=.
- Therefore, it seems that removing the calls to =HPL_dgemv= triggers a modification of the behavior of the application,
  resulting in a lower time, but this is not this function itself which takes time.
- Note that this was not the case for functions =HPL_dgemm= and =HPL_dtrsm=: this was the calls to these functions which
  took time, not a consequence of the calls (just tested: taking the sum of all the times gives a total of about 75
  seconds for =HPL_dtrsm= and about 2896 seconds for =HPL_dgemm=).
- In experimentation of =08/03/2017=, removing =HPL_dgemv= only resulted in a drop of 1 second for the execution time.
- Thus, it seems that modifying =WORK[0]= has increased the execution time, which is cancelled if we then remove
  =HPL_dgemv=. Therefore, we should not treat this function as =HPL_dgemm= and =HPL_dtrsm= (replacing it by a =smpi_usleep=), we
  should simply remove it.
- If we remove it, we get a virtual time of 226 seconds, i.e. 23.6 Gflops. This is much closer to what we used to
  have. Now the simulation time is 26 seconds, this is worse than what we used to have, but still better than what we
  had after the modification of =WORK[0]=.
*** 2017-03-22 Wednesday
**** Better usability of HPL                                  :SMPI:C:HPL:
- Before, HPL code had to be changed by hand to enable or disable SMPI optimizations (=SHARED_{MALLOC,FREE}= and
  =smpi_usleep=) and to enable or disable the tracing of BLAS function calls.
- Now, thanks to some preprocessor macros, these different settings can be configured on the command line when
  compiling:
  #+begin_src sh
  # Compile vanilla HPL for SMPI
  make arch=SMPI
  # Compile HPL for SMPI with the tracing of BLAS function calls
  make SMPI_OPTS=-DSMPI_MEASURE
  # Compile HPL for SMPI with the SMPI optimizations (shared malloc/free, smpi_usleep)
  make SMPI_OPTS=-DSMPI_OPTIMIZATION
  #+end_src
- Next step: automation of the computation of the linear regression coefficients, to also pass these coefficients as
  preprocessor variables.
**** Script to parse the output file and do the linear regression :SMPI:PYTHON:EXPERIMENTS:TRACING:HPL:
- Everything is done in Python (linear regression included) to simplify the procedure for the user.
- Given an output file =/tmp/output= as produced by a call to HPL (compiled with =-DSMPI_MEASURE= option), call the script:
  #+begin_src sh :results output :exports both
  python3 ../scripts/linear_regression.py /tmp/output
  #+end_src

  #+RESULTS:
  : -DSMPI_DGEMM_COEFF=1.097757e-09 -DSMPI_DTRSM_COEFF=9.134754e-08
  It outputs the list of the coefficients found by the linear regressions for the relevant BLAS functions. This list
  should then be passed to the variable =SMPI_OPTS= when compiling with =-DSMPI_OPTIMIZATION=.
**** Discussion with Arnaud         :SMPI:ORGMODE:PERFORMANCE:HPL:MEETING:
- Possible trip to Bordeaux on the week of [2017-04-10 Mon]-[2017-04-14 Fri]. The goal is to discuss with contributors
  of Simgrid.
- Found very strange the different time we get when we modify =WORK[0]=, especially because it is computation time (would
  be more understandable if it was communication time, since the communication patterns for the pivot exchange are very
  likely to be impacted). Should do a profiling.
- Some tips regarding org-mode (tags).
**** DONE Profile HPL
:LOGBOOK:
- State "DONE"       from "TODO"       [2017-03-27 Mon 17:56]
- State "TODO"       from              [2017-03-22 Wed 18:08]
:END:
- Use Valgrind with [[http://valgrind.org/docs/manual/cl-manual.html][Callgrind]] and [[http://kcachegrind.sourceforge.net/html/Shot3.html][Kcachegrind]] or [[https://sourceware.org/binutils/docs/gprof/][Gprof]].
- Do the profiling on unmodified HPL and modified HPL, to see if there is any obvious difference.
*** 2017-03-23 Thursday
**** Profiling vanilla HPL :EXPERIMENTS:PERFORMANCE:PROFILING:VALGRIND:SMPI:HPL:
- Profiling with Valgrind of vanilla HPL (no time measurements nor SMPI optimizations). Add the option =-g= in the Makefile.
- HPL commit: =4494976bc0dd67e04e54abec2520fd468792527a=.
- Settings: N=5000, P=Q=4.
- Compile with the command:
  #+begin_src sh
  make -j 4 arch=SMPI
  #+end_src
- Run with the command:
  #+begin_src sh
  smpirun -wrapper "valgrind --tool=callgrind" --cfg=smpi/bcast:mpich --cfg=smpi/running-power:6217956542.969
  --cfg=smpi/display-timing:yes --cfg=smpi/privatize-global-variables:yes -np 16 -hostfile ./hostfile_64.txt -platform
  ./cluster_fat_tree_64.xml ./xhpl
  #+end_src
- At first, the package =libatlas3-base= was used for the BLAS functions. The name of these functions were not shown in
  Kcachegrind.
  + [[file:callgrind/callgrind.out.19313][Output file]]
  + Visualization:
    [[file:callgrind/callgrind.19313.png]]
- Then, removed this package and installed =libatlas-cpp-0.6-2-dbg=.
  + [[file:callgrind/callgrind.out.26943][Output file]]
  + Visualization
    [[file:callgrind/callgrind.26943.png]]
- So now we have the names of the BLAS functions, but the layout is very different.
- Also, the executions with this library take more time, especially with Vallgrind. It also impacts the virtual time and
  the Gflops.
- What we observe with this new library seems to be consistent with what we observed previously: =dgemm= is the most time
  consumming function (by far), =dtrsm= comes after. So maybe this library is good enough to understand what happens, and
  then we could switch back to the previous library to have good performances.
**** Profiling modified HPL :EXPERIMENTS:PERFORMANCE:PROFILING:VALGRIND:SMPI:HPL:
- Profiling with Valgrind of modified HPL. Add the option =-g= in the Makefile.
- HPL commit: =4494976bc0dd67e04e54abec2520fd468792527a=. Then for each case, a small piece of the code has been modified.
- Settings: N=5000, P=Q=4.
- Compile with the command:
  #+begin_src sh
  make SMPI_OPTS=-DSMPI_OPTIMIZATION -j 4 arch=SMPI
  #+end_src
- Run with the command:
  #+begin_src sh
  smpirun -wrapper "valgrind --tool=callgrind" --cfg=smpi/bcast:mpich --cfg=smpi/running-power:6217956542.969
  --cfg=smpi/display-timing:yes --cfg=smpi/privatize-global-variables:yes -np 16 -hostfile ./hostfile_64.txt -platform
  ./cluster_fat_tree_64.xml ./xhpl
  #+end_src
- Using the library from =libatlas-cpp-0.6-2-dbg=.
- First experiment, the call to =HPL_dgemv= is a no-op and =WORK[0]= is set to a constant.
  + [[file:callgrind/callgrind.out.5388][Output file]]
  + Visualization:
    [[file:callgrind/callgrind.5388.png]]
- Second experiment, the call to =HPL_dgemv= is aliased to =cblas_dgemv= and =WORK[0]= is set to a constant.
  + [[file:callgrind/callgrind.out.10043][Output file]]
  + Visualization: similar.
- Third experiment, the call to =HPL_dgemv= is aliased to =cblas_dgemv= and =WORK[0]= is not modified.
  + [[file:callgrind/callgrind.out.12915][Output file]]
  + Visualization: similar.
- It is clear that we can shrink even further the simulation by removing the code that initialize the matrices (this is
  the code that calls the function =HPL_rand=).
- There is no explanation for the differences observed with =HPL_dgemv= and =WORK[0]=, the figures look similar. However the
  differences observed between the three cases are quite small (in terms of execution time or Gflops).
**** Comparison of the code                                     :SMPI:HPL:
- Let’s compare again the different versions of the code, this time with the new CBLAS library (package
  =libatlas-cpp-0.6-2-dbg=). We use N=20000, P=Q=4.
  | Code                           | Virtual time |    Gflops | Total simulation time | Time for application computations |
  |--------------------------------+--------------+-----------+-----------------------+-----------------------------------|
  | *WORK[0] unmodified, real dgemv* |       223.81 | 2.383e+01 |               15.5049 |                            9.5045 |
  | *WORK[0] modified, real dgemv*   |       223.74 | 2.384e+01 |               25.9935 |                           20.0480 |
  | *WORK[0] modified, no-op dgemv*  |       226.28 | 2.357e+01 |               26.3907 |                           20.3201 |
  Remark: for the first version of the code, the experiment had to be ran twice, since the first run ended in a deadlock.
- The two first rows correspond to the two rows of the table of [2017-03-21 Tue].
- There is no significant difference for the virtual time and the Gflops.
- There is a significant difference for the total simulation time and the time spent for aapplication computations, but
  it is less important than what we previously observed.
- It is strange that this difference in the computation time does not appear in the virtual time. Note that the option
  =--cfg=smpi/simulate-computation:0= was not used, so it does not come from here.
**** Seminar                                                     :MEETING:
- Decaf: Decoupled Dataflows for In Situ High-Performance Workflows
- Mathieu Dreher
- They do some physics experiment (with a particle collider). Then, they analyze the results and build a model. Thus,
  the whole process has three major steps.
- In current systems, the bottleneck is the I/O. It will be even worse for future systems (computation speed will be
  increased, not I/O speed). This is why we should have in-situ workflows (less data to move).
- In the “classical workflow”, we compute all the iterations, then we analyze them.
- In the “in situ workflow”, two things are possible. Time partitionning: we compute one iteration, then analyze it,
  then go back to the computation. Space partitioning: the analysis is done in parallel on other nodes.
**** Profiling modified HPL, bigger matrices :EXPERIMENTS:PERFORMANCE:PROFILING:VALGRIND:SMPI:HPL:
- Profiling with Valgrind of modified HPL. Add the option =-g= in the Makefile.
- HPL commit: =4494976bc0dd67e04e54abec2520fd468792527a=. Then for each case, a small piece of the code has been modified.
- Settings: N=20000, P=Q=4.
- Compile with the command:
  #+begin_src sh
  make SMPI_OPTS=-DSMPI_OPTIMIZATION -j 4 arch=SMPI
  #+end_src
- Run with the command:
  #+begin_src sh
  smpirun -wrapper "valgrind --tool=callgrind" --cfg=smpi/bcast:mpich --cfg=smpi/running-power:6217956542.969
  --cfg=smpi/display-timing:yes --cfg=smpi/privatize-global-variables:yes -np 16 -hostfile ./hostfile_64.txt -platform
  ./cluster_fat_tree_64.xml ./xhpl
  #+end_src
- Using the library from =libatlas-cpp-0.6-2-dbg=.
- First experiment, the call to =HPL_dgemv= is a no-op and =WORK[0]= is set to a constant.
  + [[file:callgrind/callgrind.out.6159][Output file]]
  + Visualization:
    [[file:callgrind/callgrind.6159.png]]
- Second experiment, the call to =HPL_dgemv= is aliased to =cblas_dgemv= and =WORK[0]= is set to a constant.
  + [[file:callgrind/callgrind.out.2590][Output file]]
  + Visualization:
    [[file:callgrind/callgrind.2590.png]]
- Third experiment, the call to =HPL_dgemv= is aliased to =cblas_dgemv= and =WORK[0]= is not modified.
  + [[file:callgrind/callgrind.out.31804][Output file]]
  + Visualization:
    [[file:callgrind/callgrind.31804.png]]
- The three figures have roughly the same pattern.
- However, some numbers of the two first figures are twice as large as the corresponding numbers of the third
  figure. For instance, the biggest =HPL_rand= has 70803540000 in the third figure and 141607080000 in the two first ones.
- The reason for that is that, in the two first cases, =HPL_pdmatgen= is called 32 times, whereas in the last case it is
  called only 16 times. In our settings, we would expect this function to be called 16 times, since we simulate 16 processes.
- This is very strange that =WORK[0]= has an impact on the behavior of matrices generation.
*** 2017-03-24 Friday
**** Why =WORK[0]= impacts the number of calls to =HPL_pdmatgen=    :SMPI:HPL:
- Everything happens in the file =HPL_pdtest.c=. This is in relation with the error code issue discussed on [2017-03-14 Tue].
- When we use SMPI optimizations (=smpi_usleep= and =SMPI_SHARED_MALLOC=) without modifying =WORK[0]=, HPL detects an error in
  the data of the matrices and returns an error code. If we also fix =WORK[0]= to some constant, HPL does not detect this
  error.
- After doing the factorization, if no error code has been returned, HPL does some additional tests on the values of the
  matrix. These tests are quite long, and imply to re-generate the matrix, by calling =HPL_pdmatgen=.
- This explains why =WORK[0]= has an impact on the simulation time and the number of time =HPL_pdmatgen= is called.
- This does not explain the difference in terms of virtual time observed on [2017-03-21 Tue], since this is only the
  time needed for the factorization and not the time for the initialization and the checks.
- This difference of virtual time was not re-observed on [2017-03-23 Thu]. Note that another BLAS library was used.
**** Comparison of the code                                     :SMPI:HPL:
- Let’s compare again the different versions of the code, with the “old” version of the CBLAS library (package
  =libatlas3-base=). We use N=20000, P=Q=4.
- This is the same experiment than [2017-03-23 Thu], except for the BLAS library.
- The two first rows correspond to the two rows of the table of [2017-03-21 Tue].
  | Code                           | Virtual time |    Gflops | Total simulation time | Time for application computations |
  |--------------------------------+--------------+-----------+-----------------------+-----------------------------------|
  | *WORK[0] unmodified, real dgemv* |       223.68 | 2.385e+01 |               15.8909 |                            9.5658 |
  | *WORK[0] modified, real dgemv*   |       257.79 | 2.069e+01 |               47.9488 |                           41.5125 |
  | *WORK[0] modified, no-op dgemv*  |       225.91 | 2.361e+01 |               26.2768 |                           20.1776 |
- The experiment of [2017-03-21 Tue] is replicated: the two first rows look similar.
- There is still the big gap in terms of both simulation time and virtual time. The former could be explained by the
  checks done at the end of HPL, but not the later (see previous entry of the journal).
- Interestingly, the first and the last rows look very similar to the first and the last row of the [2017-03-23 Thu]
  experiment, although the BLAS library has changed. The middle row however is very different.
- These gaps are not replicated when using Valgrind. For all simulations, we have a virtual time of about 262 to 263
  seconds, which is about 2.03e+01 Gflops.
**** Removing initialization and checks   :SMPI:PERFORMANCE:PROFILING:HPL:
- The previous experiments demonstrated that the initialization (done after the factorization) and the checks (done
  after) take a significant amount of time. They do not account for the estimation of the Gflops, so we can safely
  remove them.
- Quick experiment, with HPL at commit =cb54a92b8304e0cd2f1728b887cc4cc615334c2d=, N=20000 and P=Q=4, using library from
  package =libatlas3-base=.
- We get a virtual time of 227.35, which is 2.346e+01 Gflops. It confirms that the initialization and the checks are not
  accounted in this measure.
- The simulation time is now 9.63 seconds, with 3.53 seconds spent for actual computations of the application.
- We see here that the simulation is already well optimized, there is not much room for additional gains.
- Profiling with Valgrind:
  + [[file:callgrind/callgrind.out.5744][Output file]]
  + Visualization:
    [[file:callgrind/callgrind.5744.png]]
- We see here that a large part of the time is spent in functions called by Simgrid (e.g. =memcpy=).
**** Work on the experiment script                                :PYTHON:
- Add three features:
  + “Dump simulation and application times in the CSV.”
  + “Dump physical and virtual memory in the CSV.”
  + “Experiments with random sizes and number of processors.”
- Example of usage:
  #+begin_src sh
  ./run_measures.py --global_csv /tmp/bla.csv --nb_runs 10 --size 1000:2000,4000:5000,20000:21000 --nb_proc 1:4,8,16,32,64
  --fat_tree "2;8,8;1,1:4;1,1" --experiment HPL
  #+end_src
  This will run 10 times, in a random order, all combinations of the parameters:
  + Matrix size in [1000,2000]\cup[4000,5000]\cup[20000,21000]
  + Number of processes in {1,2,3,4,8,16,32,64}
  + Fat-trees =2;8,8;1,1;1,1= and =2;8,8;1,2;1,1= and =2;8,8;1,3;1,1= and =2;8,8;1,4;1,1=.
  The results are dumped in a CSV file. For each experiment, we store all the parameters (topology, size, number of
  processes) as well as the interesting metrics (virtual time, Gflops, simulation time, time spent in application, peak
  physical and virtual memory used).
*** 2017-03-25 Saturday
**** Time and memory efficiency of HPL simulation :SMPI:R:EXPERIMENTS:PERFORMANCE:HPL:
- HPL commit: =cb54a92b8304e0cd2f1728b887cc4cc615334c2d=
- Script commit: =8af35470776a0b6f2041cf8e0121739f94fdc34d=
- Command line to run the experiment:
  #+begin_src sh
  ./run_measures.py --global_csv hpl2.csv --nb_runs 3 --size 100,5000,10000,15000,20000,25000,30000,35000,40000
  --nb_proc 1,8,16,24,32,40,48,56,64 --fat_tree "2;8,8;1,8;1,1" --experiment HPL
  #+end_src
- Plots:
  #+begin_src R :results output :session *R* :exports both
  library(ggplot2)
  do_plot <- function(my_plot, title) {
      return(my_plot +
          stat_summary(geom="line", fun.y=mean)+
          stat_summary(fun.data = mean_sdl)+
          ggtitle(title)
      )
  }
  results <- read.csv('hpl_analysis/hpl.csv')
  head(results)
  #+end_src

  #+RESULTS:
  #+begin_example
         topology nb_roots nb_proc  size    time Gflops simulation_time
  1 2;8,8;1,8;1,1        8      48 40000  593.10 71.940        60.75480
  2 2;8,8;1,8;1,1        8      40 20000  144.88 36.820        24.53460
  3 2;8,8;1,8;1,1        8       8 30000 1290.99 13.940        13.39820
  4 2;8,8;1,8;1,1        8      56 10000   37.93 17.580        12.92780
  5 2;8,8;1,8;1,1        8       1 30000 9609.94  1.873         3.67895
  6 2;8,8;1,8;1,1        8      64 10000   27.20 24.510         9.96141
    application_time       uss         rss
  1         14.47840 701091840 13509701632
  2          6.44959 327905280  3533713408
  3          6.14242 217612288  7422472192
  4          2.55716 211193856  1016156160
  5          3.58312   5619712  7209476096
  6          2.10660 179879936   984698880
  #+end_example

  #+begin_src R :file hpl_analysis/1.png :results value graphics :results output :session *R* :exports both
  do_plot(ggplot(results, aes(x=size, y=simulation_time, group=nb_proc, color=nb_proc)),
     "Simulation time vs size")
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/1.png]]

  #+begin_src R :file hpl_analysis/2.png :results value graphics :results output :session *R* :exports both
  do_plot(ggplot(results, aes(x=nb_proc, y=simulation_time, group=size, color=size)),
      "Simulation time vs number of processes")
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/2.png]]

  #+begin_src R :file hpl_analysis/3.png :results value graphics :results output :session *R* :exports both
  do_plot(ggplot(results, aes(x=size, y=uss, group=nb_proc, color=nb_proc)),
      "Physical memory consumption vs size")
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/3.png]]

  #+begin_src R :file hpl_analysis/4.png :results value graphics :results output :session *R* :exports both
  do_plot(ggplot(results, aes(x=nb_proc, y=uss, group=size, color=size)),
     "Physical memory consumption vs number of processes")
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/4.png]]

- We see here that despite all the optimizations:
  + The simulation time seems to be quadratic in the matrix size.
  + The simulation time seems to be (roughly) linear in the number of processes.
  + The memory consumption seems to be linear in the matrix size.
  + The memory consumption seems to be (roughly) linear in the number of processes.
- There are some irregularities regarding the time and memory vs the number of processes. An hypothesis is that it is
  due to different virtual topologies. In this experiment, the number of processes are multiple of 8. So, some of these
  numbers are square numbers, others are not. It seems that we achieve the best performances when the number of
  processes is a square.
  To generate P and Q, the sizes of the process grid, we try to find two divisors of the number of processes that are
  reasonably close (if possible). Thus, when the number of processes is a square, we have P=Q.
*** 2017-03-27 Monday
**** DONE Remaining work on HPL (following discussion with Arnaud) [3/3] :SMPI:HPL:MEETING:
:LOGBOOK:
- State "DONE"       from "TODO"       [2017-03-27 Mon 17:01]
- State "TODO"       from "TODO"       [2017-03-27 Mon 13:27]
- State "TODO"       from "TODO"       [2017-03-27 Mon 12:33]
- State "TODO"       from "TODO"       [2017-03-27 Mon 12:33]
- State "TODO"       from              [2017-03-27 Mon 11:26]
:END:
- [X] Do not look further regarding the =WORK[0]= anomaly.
- [X] Do careful experiments to validate the optimizations.
- [X] Currently, the simulation will not scale in memory. Track the sizes of the =malloc= in =HPL_panel_init=.
**** More detailed analysis of =malloc=          :R:TRACING:PERFORMANCE:HPL:
- We saw that the memory consumption is still too high, we need to reduce it.
- Let’s take back the results from [2017-03-17 Fri]. The corresponding CSV file has been copied in repository =hpl_malloc=.
- Recall that this is a trace of all the =malloc=, with N=20000 and P=Q=4.
- We will focus on the file =HPL_pdpanel_init.c= since we suppose that these are the biggest allocations (after the
  allocation of the matrix).
  #+begin_src R :results output :session *R* :exports both
  results <- read.csv("hpl_malloc/malloc.csv");
  results <- results[results$file == "/hpl-2.2/src/panel/hpl_pdpanel_init.c",]
  results$idx <- 0:(length(results$size)-1)
  head(results)
  #+end_src

  #+RESULTS:
  :                                      file line rank    size idx
  : 99  /hpl-2.2/src/panel/hpl_pdpanel_init.c  245    0 4839432   0
  : 100 /hpl-2.2/src/panel/hpl_pdpanel_init.c  339    0    5344   1
  : 101 /hpl-2.2/src/panel/hpl_pdpanel_init.c  245    0 4839432   2
  : 102 /hpl-2.2/src/panel/hpl_pdpanel_init.c  339    0    5344   3
  : 106 /hpl-2.2/src/panel/hpl_pdpanel_init.c  245    2 9640392   4
  : 107 /hpl-2.2/src/panel/hpl_pdpanel_init.c  339    2    5344   5

  #+begin_src R :file hpl_malloc/1.png :results value graphics :results output :session *R* :exports both
  library(ggplot2)
  ggplot(results, aes(x=idx, y=size, color=factor(line))) + geom_point(alpha=.2) + ggtitle("Sizes of malloc in HPL_pdpanel_init (N=20000, P=Q=4)")
  #+end_src

  #+RESULTS:
  [[file:hpl_malloc/1.png]]

- Now that we have removed the matrix allocation, the panel allocation is clearly the one responsible of the high memory
  consumption. Here, for 16 processes and a matrix of size 20000, this allocation is responsible for 160MB of memory.
- The =malloc= of line 245 is the one that is a concern. It is made for the =WORK= attribute.
- The =malloc= of line 339 is not a concern. It is made for the =IWORK= attribute.
- It is strange that all these allocations are made. Why not allocating the panel once, and then reusing it until the
  end?
- It may be difficult to split the panel in two parts (one =SHARED_MALLOC= and one classical =malloc=). In
  =HPL_pdpanel_init.c=, we can find this comment:
  #+begin_example
  * L1:    JB x JB in all processes
  * DPIV:  JB      in all processes
  * DINFO: 1       in all processes
  * We make sure that those three arrays are contiguous in memory for the
  * later panel broadcast.  We  also  choose  to put this amount of space
  * right  after  L2 (when it exist) so that one can receive a contiguous
  * buffer.
  #+end_example
**** Validation of the optimizations              :SMPI:R:EXPERIMENTS:HPL:
- Let’s compare vanilla HPL with optimized HPL, to see if the simulation is still faithful.
- Results for optimized HPL are those of [2017-03-25 Sat].
- Results for vanilla HPL have been freshly generated:
  + Using HPL commit =6cc643a5c2a123fa549d02a764bea408b5ad6114=
  + Using script commit =7a9e467f9446c65a9dbc2a76c4dab7a3d8209148=
  + Command:
    #+begin_src sh
    ./run_measures.py --global_csv hpl_vanilla.csv --nb_runs 1 --size 100,5000,10000,15000,20000 --nb_proc
    1,8,16,24,32,40,48,56,64 --fat_tree "2;8,8;1,8;1,1" --experiment HPL
    #+end_src
- Analysis:
  #+begin_src R :results output :session *R* :exports both
  library(ggplot2)
  optimized_results <- read.csv('hpl_analysis/hpl.csv')
  vanilla_results <- read.csv('hpl_analysis/hpl_vanilla.csv')
  optimized_results$hpl = 'optimized_hpl'
  vanilla_results$hpl = 'vanilla_hpl'
  results = rbind(optimized_results, vanilla_results)
  head(results)
  #+end_src

  #+RESULTS:
  #+begin_example
         topology nb_roots nb_proc  size    time Gflops simulation_time
  1 2;8,8;1,8;1,1        8      48 40000  593.10 71.940        60.75480
  2 2;8,8;1,8;1,1        8      40 20000  144.88 36.820        24.53460
  3 2;8,8;1,8;1,1        8       8 30000 1290.99 13.940        13.39820
  4 2;8,8;1,8;1,1        8      56 10000   37.93 17.580        12.92780
  5 2;8,8;1,8;1,1        8       1 30000 9609.94  1.873         3.67895
  6 2;8,8;1,8;1,1        8      64 10000   27.20 24.510         9.96141
    application_time       uss         rss           hpl
  1         14.47840 701091840 13509701632 optimized_hpl
  2          6.44959 327905280  3533713408 optimized_hpl
  3          6.14242 217612288  7422472192 optimized_hpl
  4          2.55716 211193856  1016156160 optimized_hpl
  5          3.58312   5619712  7209476096 optimized_hpl
  6          2.10660 179879936   984698880 optimized_hpl
  #+end_example

  #+begin_src R :results output :session *R* :exports both
  plot_results <- function(nb_proc) {
      ggplot(results[results$nb_proc==nb_proc,], aes(x=size, y=Gflops, color=hpl)) +
          geom_point() + geom_line() +
          expand_limits(x=0, y=0) +
          ggtitle(paste("Gflops vs size, nb_proc = ", nb_proc))
  }
  #+end_src

  #+begin_src R :file hpl_analysis/5.png :results value graphics :results output :session *R* :exports both
  plot_results(1)
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/5.png]]

  #+begin_src R :file hpl_analysis/6.png :results value graphics :results output :session *R* :exports both
  plot_results(8)
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/6.png]]

  #+begin_src R :file hpl_analysis/7.png :results value graphics :results output :session *R* :exports both
  plot_results(16)
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/7.png]]

  #+begin_src R :file hpl_analysis/8.png :results value graphics :results output :session *R* :exports both
  plot_results(24)
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/8.png]]

  #+begin_src R :file hpl_analysis/9.png :results value graphics :results output :session *R* :exports both
  plot_results(32)
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/9.png]]

  #+begin_src R :file hpl_analysis/10.png :results value graphics :results output :session *R* :exports both
  plot_results(40)
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/10.png]]

  #+begin_src R :file hpl_analysis/11.png :results value graphics :results output :session *R* :exports both
  plot_results(48)
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/11.png]]

  #+begin_src R :file hpl_analysis/12.png :results value graphics :results output :session *R* :exports both
  plot_results(56)
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/12.png]]

  #+begin_src R :file hpl_analysis/13.png :results value graphics :results output :session *R* :exports both
  plot_results(64)
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/13.png]]

- From the above plots, it seems that optimized HPL is always too optimistic in terms of performances. However, the
  difference is not so important.

  #+begin_src R :file hpl_analysis/14.png :results value graphics :results output :session *R* :exports both
  merged_results = merge(x=vanilla_results, y=optimized_results, by=c("nb_proc", "size"))
  merged_results$error = abs((merged_results$Gflops.x - merged_results$Gflops.y)/merged_results$Gflops.y)
  ggplot(merged_results, aes(x=factor(size), y=error)) +
      geom_boxplot() + geom_jitter(aes(color=nb_proc)) +
      ggtitle("Error vs size")
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/14.png]]

  #+begin_src R :file hpl_analysis/15.png :results value graphics :results output :session *R* :exports both
  ggplot(merged_results, aes(x=factor(nb_proc), y=error)) +
      geom_boxplot() + geom_jitter(aes(color=size)) +
      ggtitle("Error vs nb_proc")
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/15.png]]

- We see here that the biggest errors made by the simulation are for a size of 100 and 1 process. For larger sizes and
  numbers of processes, the error never goes above 10%. In average, it is lower than 5%.

  #+begin_src R :file hpl_analysis/16.png :results value graphics :results output :session *R* :exports both
  ggplot(results[results$nb_proc==64,], aes(x=size, y=simulation_time, color=hpl)) +
      geom_point() + geom_line() +
      expand_limits(x=0, y=0) +
      ggtitle("Simulation time vs size, P=Q=8")
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/16.png]]

  #+begin_src R :file hpl_analysis/17.png :results value graphics :results output :session *R* :exports both
  ggplot(results[results$nb_proc==64,], aes(x=size, y=uss, color=hpl)) +
      geom_point() + geom_line() +
      expand_limits(x=0, y=0) +
      ggtitle("Real memory vs size, P=Q=8")
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/17.png]]

- There is a very important gain in terms of memory consumption and simulation time.
*** 2017-03-28 Tuesday
**** Booked the plane tickets for Bordeaux
**** Tentative of allocation hack in HPL_pdpanel_init :SMPI:C:PERFORMANCE:HPL:
- Greatly inspired from what is done for the global =SMPI_SHARED_MALLOC=.
- The idea is to reserve a large block of virtual addresses. The first part is mapped to a (short) buffer in a cyclic
  way. The second part is kept private.
- Currently some bugs (invalid writes, leading to a segmentation fault).
*** 2017-03-29 Wednesday
**** Keep trying to use some shared memory for =PANEL->WORK= :SMPI:C:PERFORMANCE:BUG:HPL:
- The invalid writes of yesterday were on accesses to =WORK= buffer. Forgot the space needed for the buffer =U= at the end
  of =WORK=. Now fixed.
- Add some =printf= to see the start and end addresses of the different buffers. Everything seems fine.
- Add a check. We fill the private zone (=DPIV= and =DINFO=) with 0. Then we fill the shared zone with garbage. Finally we
  check that the private zone is still full 0.
- Now, there is an invalid write of 4 bytes, by =HPL_plindx1=, located just after the buffer =IWORK= (the allocation of this
  buffer did not change).
- Test for N=5000, P=Q=4. Found that in file =HPL_plindx1.c=, variable =ipU= reaches 120 in the buggy case, but only 119 in
  the normal case. So it is likely that the array is not too short, but rather that this variable is incremented too much.
- In the =for= loop where this happens, =ipU= is incremented when some conditions are fulfilled. One of these conditions is
  the combination of these two =if=:
  #+begin_src c
  if( srcrow == icurrow ) {
      if( ( dstrow == icurrow ) && ( dst - ia < jb ) )
  // [...]
  #+end_src
  When =ipU= reaches 120, the illegal write is:
  #+begin_src c
  iwork[ipU] = IPLEN[il];
  #+end_src
  When this happens, the variable =dst= is 0 and thus the condition =dst-ia<jb= is met. But intuitively, this condition
  should not be met like this (=jb= is always positive).
  A bit earlier in the loop, =dst= is set with:
  #+begin_src c
  dst = IPID[i+1];
  #+end_src
  Printing this array in the buggy case and in the normal case, we see that the last element of the array is sometimes 0
  in the buggy case, but never in the normal case. Thus, it seems that there is an issue with =IPID=.
- Note that we also had issues with =IPID= when using =SHARED_MALLOC=.
**** Looking at =PANEL->DPIV= (again)                         :SMPI:BUG:HPL:
- Add a =printf= in =HPL_pipid.c= (function which compute =IPID=, using =DPIV=) to see the content of =DPIV=.
- In the buggy case, sometimes, the array =DPIV= is full of 0. It does not happen in the normal case. If we put something
  else in =DPIV= when it is allocated, then this is shown instead of the zeroes (e.g. if we put 0, 1, 2...). Thus, in
  these cases, =DPIV= is never filled after its initialization.
- Hypothesis: when the pannels are sent with MPI, the size is too short and =DPIV= is not sent.
**** Discussion with Arnaud and Augustin                         :MEETING:
- Instead of putting an empty space between the shared block and the private block (for alignment), make them really
  contiguous (and do not share the last page of the “shared” block).
**** Reimplement the shared allocation            :SMPI:C:PERFORMANCE:HPL:
- The code was a mess, let’s restart something better, using Augustin’s idea.
- The interface is as follows:
  #+begin_src c
  void *allocate_shared(int size, int start_private, int stop_private);
  #+end_src
  It allocates a contiguous block of virtual addresses of given size that all fit in a small block of physical memory,
  except for a contiguous block located between the indices start_private (included) and stop_private (excluded).
  Calling =allocate_shared(size, size, size)= is (semantically) equivalent to calling =SMPI_SHARED_MALLOC(size)=.
  Calling =allocate_shared(size, 0, size)= is (semantically) equivalent to calling =malloc(size)=.
- Similarly to =SHARED_MALLOC=, we map the shared zones by block, on a same range of addresses. The “best” block size is
  to discuss.
- Since every call to =mmap= is a syscall, we should try to not have a too low block size. Used 0x1000 at the beginning,
  the performances were terrible.
- Still for performance reasons, if the size is too low, we should simply do a malloc (and thus not have any shared zone).
- Valgrind does not report any error (it was the case with the previous implementation). There are some small memory
  leaks however.
- Performances are good. Tested with N=40000, P=Q=8. Simulation time increased from 85 seconds to 112 seconds. Memory
  consumption decreased from 675 MB to 95 MB. The virtual time and the Gflops were not impacted.
**** DONE Remaining work for shared allocation [4/4]
:LOGBOOK:
- State "DONE"       from "TODO"       [2017-04-05 Wed 17:24]
- State "TODO"       from "TODO"       [2017-03-30 Thu 09:52]
- State "TODO"       from "TODO"       [2017-03-30 Thu 09:52]
- State "TODO"       from "TODO"       [2017-03-30 Thu 09:18]
- State "TODO"       from              [2017-03-29 Wed 18:31]
:END:
- [X] Track the memory leaks (unclosed file?).
- [X] Clean the block size definition. Put it somewhere accessible by both =HPL_pdpanel_init= and =HPL_pdpanel_free=. Maybe
  use two different values for the block size and the condition to switch to a simple =malloc=.
- [X] Find the best value(s) for the block size (and maybe the =malloc= condition).
- [X] Contribute this function to Simgrid.
*** 2017-03-30 Thursday
**** Quick work on shared allocations                         :SMPI:C:HPL:
- Clean the size definitions.
  + Use a separate file that is imported in =HPL_pdpanel_init.c= and =HPL_pdpanel_free.c=.
  + Use two different sizes: the block size, and the size at which we switch for =malloc=.
- Quick look at the possibilities for the sizes
  + Some quick experiments with N=40000, P=Q=8.
  + With =BLOCK_SIZE= and =MALLOC_MAX_SIZE= equal to 0x10000:
    - Simulation time: 112 seconds
    - Physical memory: 96 MB
  + With =BLOCK_SIZE= equal to 0x10000 and =MALLOC_MAX_SIZE= equal to 0 (never do a =malloc=):
    - Simulation time: also 112 seconds
    - Physical memory: also 96 MB
  + With =BLOCK_SIZE= equal to 0x10000 and =MALLOC_MAX_SIZE= equal to 0x40000 (4 times greater):
    - Simulation time: 137 seconds
    - Physical memory: 93 MB
  + Thus, it seems that the gain of using =malloc= is not so important. Worst: it can yield a significant loss. Let’s
    remove it.
  + With =BLOCK_SIZE= equal to 0x100000 and =malloc= removed: execution cancelled, all the physical memory was used.
- Stop using =malloc=. Also move back the size definition in =HPL_pdpanel_init.c= 
- The code is simpler like this, and the =malloc= trick did not give better performances.
- Do not bother with the memory leak. It was already here before the shared allocation.
- *Warning:* calling =munmap= with a size of 0 gives a huge memory consumption. It should be called with the correct size.
**** Implement the partial =shared_malloc= in Simgrid
- Even more generic implementation than the one done in HPL. Now, we give a list of offsets of block that should be
  shared. Thus, we can have an arbitrary mix of shared zones with private zones inside an allocated block.
- Tests currently fail. To run a single test and see its output, run:
  #+begin_src sh
  ctest --verbose -R tesh-smpi-macro-shared-thread
  #+end_src
  I suspect (but did not check) that this is because we currently share only blocks aligned on the block size.
  It would be better to share blocks aligned on the page size (need to fix it). But this does not change the fact that
  some parts will not be shared. This is expected, we should modify the tests.
**** Time and memory efficiency of the partial =shared_malloc= :SMPI:R:EXPERIMENTS:PERFORMANCE:HPL:
- We switch back to the implementation of partial =shared_malloc= done in HPL, to measure its performances.
- Simgrid commit: =c8db21208f3436c35d3fdf5a875a0059719bff43= (the same commit that for the previous performance analysis)
- HPL commit: =7af9eb0ec54418bf1521c5eafa9acda1b150446f=
- Script commit: =7a9e467f9446c65a9dbc2a76c4dab7a3d8209148=
- Command line to run the experiment:
  #+begin_src sh
  ./run_measures.py --global_csv hpl_partial_shared.csv --nb_runs 1 --size 100,5000,10000,15000,20000,25000,30000,35000,40000
  --nb_proc 1,8,16,24,32,40,48,56,64 --fat_tree "2;8,8;1,8;1,1" --experiment HPL
  #+end_src
- Analysis:
  #+begin_src R :results output :session *R* :exports both
  library(ggplot2)
  partial_shared_results <- read.csv('hpl_analysis/hpl_partial_shared.csv')
  optimized_results <- read.csv('hpl_analysis/hpl.csv')
  vanilla_results <- read.csv('hpl_analysis/hpl_vanilla.csv')
  partial_shared_results$hpl = 'partial_shared_hpl'
  optimized_results$hpl = 'optimized_hpl'
  vanilla_results$hpl = 'vanilla_hpl'
  results = rbind(partial_shared_results, optimized_results, vanilla_results)
  head(results)
  #+end_src

  #+RESULTS:
  #+begin_example
         topology nb_roots nb_proc  size     time    Gflops simulation_time
  1 2;8,8;1,8;1,1        8      24 25000   319.37 32.620000      25.8119000
  2 2;8,8;1,8;1,1        8      24  5000    13.03  6.399000       2.7273300
  3 2;8,8;1,8;1,1        8      24 35000   781.76 36.570000      49.3234000
  4 2;8,8;1,8;1,1        8      40   100     0.23  0.003028       0.0779319
  5 2;8,8;1,8;1,1        8       1 35000 15257.68  1.873000       5.8686300
  6 2;8,8;1,8;1,1        8      64 40000   488.99 87.260000     111.7290000
    application_time      uss         rss                hpl
  1        8.0867100 55365632  5274730496 partial_shared_hpl
  2        0.6131710 14643200   257220608 partial_shared_hpl
  3       16.0733000 74350592 10180751360 partial_shared_hpl
  4        0.0196671        0           0 partial_shared_hpl
  5        5.7156200  4775936  9809465344 partial_shared_hpl
  6       29.3046000 95391744 13475909632 partial_shared_hpl
#+end_example

#+begin_src R :results output :session *R* :exports both
plot_results <- function(nb_proc) {
    ggplot(results[results$nb_proc==nb_proc,], aes(x=size, y=Gflops, color=hpl)) +
        geom_point() + geom_line() +
        expand_limits(x=0, y=0) +
        ggtitle(paste("Gflops vs size, nb_proc = ", nb_proc))
}
#+end_src

#+begin_src R :file hpl_analysis/18.png :results value graphics :results output :session *R* :exports both
plot_results(32)
#+end_src

#+RESULTS:
[[file:hpl_analysis/18.png]]

#+begin_src R :file hpl_analysis/19.png :results value graphics :results output :session *R* :exports both
plot_results(64)
#+end_src

#+RESULTS:
[[file:hpl_analysis/19.png]]

- It seems that this new optimization did not change the accuracy of the simulation. Let’s have a look at the time and
  memory.

  #+begin_src R :file hpl_analysis/20.png :results value graphics :results output :session *R* :exports both
  ggplot(results[results$nb_proc==64,], aes(x=size, y=simulation_time, color=hpl)) +
      geom_point() + geom_line() +
      expand_limits(x=0, y=0) +
      ggtitle("Simulation time vs size, P=Q=8")
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/20.png]]

  #+begin_src R :file hpl_analysis/21.png :results value graphics :results output :session *R* :exports both
  ggplot(results[results$nb_proc==64,], aes(x=size, y=uss, color=hpl)) +
      geom_point() + geom_line() +
      expand_limits(x=0, y=0) +
      ggtitle("Real memory vs size, P=Q=8")
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/21.png]]

- We see here that sharing some parts of the =PANEL->WORK= buffer has two effects. The simulation time is a bit larger,
  but the memory consumption is much lower.
- Let’s have a look in more details at this version of HPL.

  #+begin_src R :results output :session *R* :exports both
  do_plot <- function(my_plot, title) {
      return(my_plot +
          geom_point() + geom_line() +
          ggtitle(title)
      )
  }
  #+end_src

  #+begin_src R :file hpl_analysis/22.png :results value graphics :results output :session *R* :exports both
  do_plot(ggplot(partial_shared_results, aes(x=size, y=simulation_time, group=nb_proc, color=nb_proc)),
     "Simulation time vs size")
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/22.png]]

  #+begin_src R :file hpl_analysis/23.png :results value graphics :results output :session *R* :exports both
  do_plot(ggplot(partial_shared_results, aes(x=nb_proc, y=simulation_time, group=size, color=size)),
      "Simulation time vs number of processes")
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/23.png]]

  #+begin_src R :file hpl_analysis/24.png :results value graphics :results output :session *R* :exports both
  do_plot(ggplot(partial_shared_results, aes(x=size, y=uss, group=nb_proc, color=nb_proc)),
      "Physical memory consumption vs size")
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/24.png]]


  #+begin_src R :file hpl_analysis/25.png :results value graphics :results output :session *R* :exports both
  do_plot(ggplot(partial_shared_results, aes(x=nb_proc, y=uss, group=size, color=size)),
      "Physical memory consumption vs number of processes")
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/25.png]]

- The trend for the simulation time looks similar to what we got previously.
- The memory consumption still looks linear in the size and in the number of processes. However, it is almost flat for
  the number of processes.
**** Regression of Time and memory efficiency of the partial =shared_malloc=  (Arnaud) :SMPI:R:EXPERIMENTS:PERFORMANCE:HPL:

#+begin_src R :results output :session *R* :exports both
results$hpl=factor(results$hpl)
data = results[results$hpl=="partial_shared_hpl" & 
               results$nb_proc > 1 & results$size > 1000, # get rid of particularly small values
               c("nb_proc","size","Gflops","simulation_time","uss")]
head(data)
#+end_src

#+RESULTS:
:   nb_proc  size Gflops simulation_time      uss
: 1      24 25000 32.620        25.81190 55365632
: 2      24  5000  6.399         2.72733 14643200
: 3      24 35000 36.570        49.32340 74350592
: 6      64 40000 87.260       111.72900 95391744
: 7      24 10000 16.600         6.22743 26472448
: 8      40 40000 55.990       100.31300 91209728

#+begin_src R :results output graphics :file hpl_analysis/26.png :exports both :width 600 :height 400 :session *R*
plot(data)
#+end_src

#+RESULTS:
[[file:hpl_analysis/26.png]]

#+begin_src R :results output :session *R* :exports both
reg_rss = lm(data=data,uss ~ size+nb_proc) # Interactions do not bring much
summary(reg_rss)
#+end_src

#+RESULTS:
#+begin_example

Call:
lm(formula = uss ~ size + nb_proc, data = data)

Residuals:
     Min       1Q   Median       3Q      Max 
-6941093 -1573650  -348763  1611008  8790400 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) 7.827e+05  1.030e+06   0.760     0.45    
size        2.054e+03  3.045e+01  67.449  < 2e-16 ***
nb_proc     1.717e+05  1.903e+04   9.022 7.85e-13 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 2791000 on 61 degrees of freedom
Multiple R-squared:  0.987,	Adjusted R-squared:  0.9866 
F-statistic:  2315 on 2 and 61 DF,  p-value: < 2.2e-16
#+end_example

#+begin_src R :results output graphics :file hpl_analysis/27.png :exports both :width 600 :height 400 :session *R*
par(mfrow=c(2,3)) ; 
  plot(data=data,uss~size); 
  plot(data=data,uss~nb_proc);
  plot(reg_rss); 
par(mfrow=c(1,1))
#+end_src

#+RESULTS:
[[file:hpl_analysis/27.png]]

The Stampede HPL output indicates:
#+BEGIN_EXAMPLE
The following parameter values will be used:

N        : 3875000 
NB       :    1024 
PMAP     : Column-major process mapping
P        :      77 
Q        :      78 
PFACT    :   Right 
NBMIN    :       4 
NDIV     :       2 
RFACT    :   Crout 
BCAST    :  BlongM 
DEPTH    :       0 
SWAP     : Binary-exchange
L1       : no-transposed form
U        : no-transposed form
EQUIL    : no
ALIGN    :    8 double precision words
#+END_EXAMPLE

We aim at ~size=3875000~ and ~nb_proc=77*78~.

#+begin_src R :results output :session *R* :exports both
data[data$nb_proc==64 & data$size==40000,]
data[data$nb_proc==64 & data$size==40000,]$uss/1E6 # in MB
example=data.frame(size=c(3875000,40000), nb_proc=c(77*78,64));
predict(reg_rss, example, interval="prediction", level=0.95)/1E6
#+end_src

#+RESULTS:
:   nb_proc  size Gflops simulation_time      uss
: 6      64 40000  87.26         111.729 95391744
: [1] 95.39174
:          fit        lwr        upr
: 1 8991.32610 8664.69163 9317.96056
: 2   93.93216   88.10931   99.75501

So we should need around 8 to 9 GB. Good.

#+begin_src R :results output :session *R* :exports both
reg_time = lm(data=data,simulation_time ~ poly(size,3)*poly(nb_proc,2)) # Interactions do not bring much
summary(reg_time)
reg_time = lm(data=data,simulation_time ~ poly(size,3)+poly(nb_proc,2)+I(size*nb_proc)) # Interactions do not bring much
summary(reg_time)
reg_time = lm(data=data,simulation_time ~ poly(size,2)+poly(nb_proc,1)+I(size*nb_proc)) # Interactions do not bring much
summary(reg_time)
#+end_src

#+RESULTS:
#+begin_example

Call:
lm(formula = simulation_time ~ poly(size, 3) * poly(nb_proc, 
    2), data = data)

Residuals:
     Min       1Q   Median       3Q      Max 
-14.6972  -2.8188   0.1211   1.4618  23.6037 

Coefficients:
                                 Estimate Std. Error t value Pr(>|t|)    
(Intercept)                       34.3882     0.8715  39.458  < 2e-16 ***
poly(size, 3)1                   200.7402     6.9721  28.792  < 2e-16 ***
poly(size, 3)2                    37.6113     6.9721   5.395 1.71e-06 ***
poly(size, 3)3                     0.9386     6.9721   0.135   0.8934    
poly(nb_proc, 2)1                110.2551     6.9721  15.814  < 2e-16 ***
poly(nb_proc, 2)2                 -9.0383     6.9721  -1.296   0.2006    
poly(size, 3)1:poly(nb_proc, 2)1 619.6089    55.7771  11.109 2.43e-15 ***
poly(size, 3)2:poly(nb_proc, 2)1 101.1174    55.7771   1.813   0.0756 .  
poly(size, 3)3:poly(nb_proc, 2)1  -2.3618    55.7771  -0.042   0.9664    
poly(size, 3)1:poly(nb_proc, 2)2 -54.5865    55.7771  -0.979   0.3323    
poly(size, 3)2:poly(nb_proc, 2)2 -13.4280    55.7771  -0.241   0.8107    
poly(size, 3)3:poly(nb_proc, 2)2  -6.7984    55.7771  -0.122   0.9035    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 6.972 on 52 degrees of freedom
Multiple R-squared:  0.9597,	Adjusted R-squared:  0.9511 
F-statistic: 112.5 on 11 and 52 DF,  p-value: < 2.2e-16

Call:
lm(formula = simulation_time ~ poly(size, 3) + poly(nb_proc, 
    2) + I(size * nb_proc), data = data)

Residuals:
     Min       1Q   Median       3Q      Max 
-11.9992  -3.5157   0.0224   2.7090  25.8055 

Coefficients:
                    Estimate Std. Error t value Pr(>|t|)    
(Intercept)       -2.954e+00  3.452e+00  -0.856  0.39567    
poly(size, 3)1     4.863e+01  1.527e+01   3.184  0.00236 ** 
poly(size, 3)2     3.761e+01  6.930e+00   5.427 1.22e-06 ***
poly(size, 3)3     9.386e-01  6.930e+00   0.135  0.89275    
poly(nb_proc, 2)1 -4.186e+01  1.527e+01  -2.740  0.00818 ** 
poly(nb_proc, 2)2 -9.038e+00  6.930e+00  -1.304  0.19742    
I(size * nb_proc)  4.610e-05  4.125e-06  11.176 5.47e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 6.93 on 57 degrees of freedom
Multiple R-squared:  0.9563,	Adjusted R-squared:  0.9517 
F-statistic:   208 on 6 and 57 DF,  p-value: < 2.2e-16

Call:
lm(formula = simulation_time ~ poly(size, 2) + poly(nb_proc, 
    1) + I(size * nb_proc), data = data)

Residuals:
     Min       1Q   Median       3Q      Max 
-11.8123  -3.6614   0.2628   2.4029  25.7019 

Coefficients:
                    Estimate Std. Error t value Pr(>|t|)    
(Intercept)       -2.954e+00  3.444e+00  -0.858  0.39442    
poly(size, 2)1     4.863e+01  1.524e+01   3.191  0.00227 ** 
poly(size, 2)2     3.761e+01  6.914e+00   5.440 1.07e-06 ***
poly(nb_proc, 1)  -4.186e+01  1.524e+01  -2.747  0.00797 ** 
I(size * nb_proc)  4.610e-05  4.115e-06  11.202 3.08e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 6.914 on 59 degrees of freedom
Multiple R-squared:  0.955,	Adjusted R-squared:  0.952 
F-statistic: 313.1 on 4 and 59 DF,  p-value: < 2.2e-16
#+end_example


#+begin_src R :results output graphics :file hpl_analysis/28.png :exports both :width 600 :height 400 :session *R*
par(mfrow=c(2,3)) ; 
  plot(data=data,simulation_time~size); 
  plot(data=data,simulation_time~nb_proc);
  plot(reg_time); 
par(mfrow=c(1,1))
#+end_src

#+RESULTS:
[[file:hpl_analysis/28.png]]


#+begin_src R :results output :session *R* :exports both
data[data$nb_proc==64 & data$size==40000,]
predict(reg_time, example, interval="prediction", level=0.95)/3600 # in hours
#+end_src

#+RESULTS:
:   nb_proc  size Gflops simulation_time      uss
: 6      64 40000  87.26         111.729 95391744
:            fit          lwr          upr
: 1 467.31578577 385.82615026 548.80542127
: 2   0.03431702   0.03008967   0.03854438

Aouch. This would be a 3 weeks simulation. :( We need to speed things
up.
*** 2017-03-31 Friday
**** Found a bug in the last commits of Simgrid             :SMPI:BUG:HPL:
- Issue reported on [[https://github.com/simgrid/simgrid/issues/147][Github]].
- Bug fixed.
- There are still some problems with HPL, some unitialized values used for comparisons:
  #+begin_example
  ==3320== Memcheck, a memory error detector
  ==3320== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
  ==3320== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
  ==3320== Command: ./xhpl --cfg=surf/precision:1e-9 --cfg=network/model:SMPI --cfg=network/TCP-gamma:4194304 --cfg=smpi/bcast:mpich --cfg=smpi/running-power:6217956542.969 --cfg=smpi/display-timing:yes --cfg=smpi/privatize-global-variables:yes --cfg=smpi/shared-malloc:local --cfg=smpi/privatize-global-variables:1 ./cluster_fat_tree_64.xml smpitmp-apprXPdW8
  ==3320== 
  [0.000000] [xbt_cfg/INFO] Configuration change: Set 'surf/precision' to '1e-9'
  [0.000000] [xbt_cfg/INFO] Configuration change: Set 'network/model' to 'SMPI'
  [0.000000] [xbt_cfg/INFO] Configuration change: Set 'network/TCP-gamma' to '4194304'
  [0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/bcast' to 'mpich'
  [0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/running-power' to '6217956542.969'
  [0.000000] [xbt_cfg/INFO] Option smpi/running-power has been renamed to smpi/host-speed. Consider switching.
  [0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/display-timing' to 'yes'
  [0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/privatize-global-variables' to 'yes'
  [0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/shared-malloc' to 'local'
  [0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/privatize-global-variables' to '1'
  [0.000000] [smpi_coll/INFO] Switch to algorithm mpich for collective bcast
  ================================================================================
  HPLinpack 2.2  --  High-Performance Linpack benchmark  --   February 24, 2016
  Written by A. Petitet and R. Clint Whaley,  Innovative Computing Laboratory, UTK
  Modified by Piotr Luszczek, Innovative Computing Laboratory, UTK
  Modified by Julien Langou, University of Colorado Denver
  ================================================================================
  
  An explanation of the input/output parameters follows:
  T/V    : Wall time / encoded variant.
  N      : The order of the coefficient matrix A.
  NB     : The partitioning blocking factor.
  P      : The number of process rows.
  Q      : The number of process columns.
  Time   : Time in seconds to solve the linear system.
  Gflops : Rate of execution for solving the linear system.
  
  The following parameter values will be used:
  
  N      :      29       30       34       35 
  NB     :       1        2        3        4 
  PMAP   : Row-major process mapping
  P      :       2        1        4 
  Q      :       2        4        1 
  PFACT  :    Left    Crout    Right 
  NBMIN  :       2        4 
  NDIV   :       2 
  RFACT  :    Left    Crout    Right 
  BCAST  :   1ring 
  DEPTH  :       0 
  SWAP   : Mix (threshold = 64)
  L1     : transposed form
  U      : transposed form
  EQUIL  : yes
  ALIGN  : 8 double precision words
  
  --------------------------------------------------------------------------------
  
  - The matrix A is randomly generated for each test.
  - The following scaled residual check will be computed:
        ||Ax-b||_oo / ( eps * ( || x ||_oo * || A ||_oo + || b ||_oo ) * N )
  - The relative machine precision (eps) is taken to be               1.110223e-16
  - Computational tests pass if scaled residuals are less than                16.0
  
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x42447D: HPL_pipid (HPL_pipid.c:144)
  ==3320==    by 0x418ED8: HPL_pdlaswp00T (HPL_pdlaswp00T.c:171)
  ==3320==    by 0x40E878: HPL_pdupdateTT (HPL_pdupdateTT.c:271)
  ==3320==    by 0x41AF9F: HPL_pdgesv0 (HPL_pdgesv0.c:152)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x42476D: HPL_plindx0 (HPL_plindx0.c:246)
  ==3320==    by 0x418EF6: HPL_pdlaswp00T (HPL_pdlaswp00T.c:172)
  ==3320==    by 0x40E878: HPL_pdupdateTT (HPL_pdupdateTT.c:271)
  ==3320==    by 0x41AF9F: HPL_pdgesv0 (HPL_pdgesv0.c:152)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x4247A9: HPL_plindx0 (HPL_plindx0.c:250)
  ==3320==    by 0x418EF6: HPL_pdlaswp00T (HPL_pdlaswp00T.c:172)
  ==3320==    by 0x40E878: HPL_pdupdateTT (HPL_pdupdateTT.c:271)
  ==3320==    by 0x41AF9F: HPL_pdgesv0 (HPL_pdgesv0.c:152)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Use of uninitialised value of size 8
  ==3320==    at 0x420413: HPL_dlaswp01T (HPL_dlaswp01T.c:240)
  ==3320==    by 0x418BDD: HPL_pdlaswp00T (HPL_pdlaswp00T.c:194)
  ==3320==    by 0x40E878: HPL_pdupdateTT (HPL_pdupdateTT.c:271)
  ==3320==    by 0x41AF9F: HPL_pdgesv0 (HPL_pdgesv0.c:152)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x4E779CC: idamax_ (in /usr/lib/libblas/libblas.so.3.6.0)
  ==3320==    by 0x4E779FA: idamaxsub_ (in /usr/lib/libblas/libblas.so.3.6.0)
  ==3320==    by 0x4E4796F: cblas_idamax (in /usr/lib/libblas/libblas.so.3.6.0)
  ==3320==    by 0x4134F0: HPL_dlocmax (HPL_dlocmax.c:125)
  ==3320==    by 0x40B277: HPL_pdpanllT (HPL_pdpanllT.c:167)
  ==3320==    by 0x4243C8: HPL_pdfact (HPL_pdfact.c:129)
  ==3320==    by 0x41AF61: HPL_pdgesv0 (HPL_pdgesv0.c:146)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320== 
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x417083: HPL_pdmxswp (HPL_pdmxswp.c:238)
  ==3320==    by 0x40B4C2: HPL_pdpanllT (HPL_pdpanllT.c:221)
  ==3320==    by 0x4243C8: HPL_pdfact (HPL_pdfact.c:129)
  ==3320==    by 0x41AF61: HPL_pdgesv0 (HPL_pdgesv0.c:146)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x417098: HPL_pdmxswp (HPL_pdmxswp.c:238)
  ==3320==    by 0x40B4C2: HPL_pdpanllT (HPL_pdpanllT.c:221)
  ==3320==    by 0x4243C8: HPL_pdfact (HPL_pdfact.c:129)
  ==3320==    by 0x41AF61: HPL_pdgesv0 (HPL_pdgesv0.c:146)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x4170A2: HPL_pdmxswp (HPL_pdmxswp.c:239)
  ==3320==    by 0x40B4C2: HPL_pdpanllT (HPL_pdpanllT.c:221)
  ==3320==    by 0x4243C8: HPL_pdfact (HPL_pdfact.c:129)
  ==3320==    by 0x41AF61: HPL_pdgesv0 (HPL_pdgesv0.c:146)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x4170A4: HPL_pdmxswp (HPL_pdmxswp.c:239)
  ==3320==    by 0x40B4C2: HPL_pdpanllT (HPL_pdpanllT.c:221)
  ==3320==    by 0x4243C8: HPL_pdfact (HPL_pdfact.c:129)
  ==3320==    by 0x41AF61: HPL_pdgesv0 (HPL_pdgesv0.c:146)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x4170A6: HPL_pdmxswp (HPL_pdmxswp.c:239)
  ==3320==    by 0x40B4C2: HPL_pdpanllT (HPL_pdpanllT.c:221)
  ==3320==    by 0x4243C8: HPL_pdfact (HPL_pdfact.c:129)
  ==3320==    by 0x41AF61: HPL_pdgesv0 (HPL_pdgesv0.c:146)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x4150D5: HPL_dlocswpT (HPL_dlocswpT.c:134)
  ==3320==    by 0x40B4D2: HPL_pdpanllT (HPL_pdpanllT.c:222)
  ==3320==    by 0x4243C8: HPL_pdfact (HPL_pdfact.c:129)
  ==3320==    by 0x41AF61: HPL_pdgesv0 (HPL_pdgesv0.c:146)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x4150D7: HPL_dlocswpT (HPL_dlocswpT.c:134)
  ==3320==    by 0x40B4D2: HPL_pdpanllT (HPL_pdpanllT.c:222)
  ==3320==    by 0x4243C8: HPL_pdfact (HPL_pdfact.c:129)
  ==3320==    by 0x41AF61: HPL_pdgesv0 (HPL_pdgesv0.c:146)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x40B4DF: HPL_pdpanllT (HPL_pdpanllT.c:223)
  ==3320==    by 0x4243C8: HPL_pdfact (HPL_pdfact.c:129)
  ==3320==    by 0x41AF61: HPL_pdgesv0 (HPL_pdgesv0.c:146)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x40B4E1: HPL_pdpanllT (HPL_pdpanllT.c:223)
  ==3320==    by 0x4243C8: HPL_pdfact (HPL_pdfact.c:129)
  ==3320==    by 0x41AF61: HPL_pdgesv0 (HPL_pdgesv0.c:146)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x42483B: HPL_plindx0 (HPL_plindx0.c:255)
  ==3320==    by 0x418EF6: HPL_pdlaswp00T (HPL_pdlaswp00T.c:172)
  ==3320==    by 0x40E878: HPL_pdupdateTT (HPL_pdupdateTT.c:271)
  ==3320==    by 0x41AF9F: HPL_pdgesv0 (HPL_pdgesv0.c:152)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x424877: HPL_plindx0 (HPL_plindx0.c:269)
  ==3320==    by 0x418EF6: HPL_pdlaswp00T (HPL_pdlaswp00T.c:172)
  ==3320==    by 0x40E878: HPL_pdupdateTT (HPL_pdupdateTT.c:271)
  ==3320==    by 0x41AF9F: HPL_pdgesv0 (HPL_pdgesv0.c:152)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Use of uninitialised value of size 8
  ==3320==    at 0x420B90: HPL_dlaswp02N (HPL_dlaswp02N.c:199)
  ==3320==    by 0x418570: HPL_pdlaswp00T (HPL_pdlaswp00T.c:198)
  ==3320==    by 0x40E878: HPL_pdupdateTT (HPL_pdupdateTT.c:271)
  ==3320==    by 0x41AF9F: HPL_pdgesv0 (HPL_pdgesv0.c:152)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Use of uninitialised value of size 8
  ==3320==    at 0x422901: HPL_dlaswp04T (HPL_dlaswp04T.c:259)
  ==3320==    by 0x418CC3: HPL_pdlaswp00T (HPL_pdlaswp00T.c:329)
  ==3320==    by 0x40E878: HPL_pdupdateTT (HPL_pdupdateTT.c:271)
  ==3320==    by 0x41AF9F: HPL_pdgesv0 (HPL_pdgesv0.c:152)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x41F06D: HPL_pdpanel_free (HPL_pdpanel_free.c:79)
  ==3320==    by 0x41AF31: HPL_pdgesv0 (HPL_pdgesv0.c:141)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x4248A5: HPL_plindx0 (HPL_plindx0.c:258)
  ==3320==    by 0x418EF6: HPL_pdlaswp00T (HPL_pdlaswp00T.c:172)
  ==3320==    by 0x40E878: HPL_pdupdateTT (HPL_pdupdateTT.c:271)
  ==3320==    by 0x41AF9F: HPL_pdgesv0 (HPL_pdgesv0.c:152)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x4203FF: HPL_dlaswp01T (HPL_dlaswp01T.c:237)
  ==3320==    by 0x418BDD: HPL_pdlaswp00T (HPL_pdlaswp00T.c:194)
  ==3320==    by 0x40E878: HPL_pdupdateTT (HPL_pdupdateTT.c:271)
  ==3320==    by 0x41AF9F: HPL_pdgesv0 (HPL_pdgesv0.c:152)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Use of uninitialised value of size 8
  ==3320==    at 0x4205A0: HPL_dlaswp01T (HPL_dlaswp01T.c:245)
  ==3320==    by 0x418BDD: HPL_pdlaswp00T (HPL_pdlaswp00T.c:194)
  ==3320==    by 0x40E878: HPL_pdupdateTT (HPL_pdupdateTT.c:271)
  ==3320==    by 0x41AF9F: HPL_pdgesv0 (HPL_pdgesv0.c:152)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x4170B5: HPL_pdmxswp (HPL_pdmxswp.c:240)
  ==3320==    by 0x40B4C2: HPL_pdpanllT (HPL_pdpanllT.c:221)
  ==3320==    by 0x4243C8: HPL_pdfact (HPL_pdfact.c:129)
  ==3320==    by 0x41AF61: HPL_pdgesv0 (HPL_pdgesv0.c:146)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  ==3320== Conditional jump or move depends on uninitialised value(s)
  ==3320==    at 0x41F06D: HPL_pdpanel_free (HPL_pdpanel_free.c:79)
  ==3320==    by 0x41F040: HPL_pdpanel_disp (HPL_pdpanel_disp.c:89)
  ==3320==    by 0x41AFCD: HPL_pdgesv0 (HPL_pdgesv0.c:161)
  ==3320==    by 0x40EFC4: HPL_pdgesv (HPL_pdgesv.c:103)
  ==3320==    by 0x406F64: HPL_pdtest (HPL_pdtest.c:197)
  ==3320==    by 0x401D38: smpi_simulated_main_ (HPL_pddriver.c:223)
  ==3320==    by 0x525BCDA: smpi_main_wrapper (smpi_global.cpp:366)
  ==3320==    by 0x5129B8D: operator() (functional.hpp:48)
  ==3320==    by 0x5129B8D: std::_Function_handler<void (), simgrid::xbt::MainFunction<int (*)(int, char**)> >::_M_invoke(std::_Any_data const&) (functional:1740)
  ==3320==    by 0x5151BB1: operator() (functional:2136)
  ==3320==    by 0x5151BB1: operator() (Context.hpp:92)
  ==3320==    by 0x5151BB1: simgrid::kernel::context::RawContext::wrapper(void*) (ContextRaw.cpp:303)
  ==3320== 
  [0.884470] /home/degomme/simgrid/src/simix/smx_global.cpp:567: [simix_kernel/CRITICAL] Oops ! Deadlock or code not perfectly clean.
  [0.884470] [simix_kernel/INFO] 16 processes are still running, waiting for something.
  [0.884470] [simix_kernel/INFO] Legend of the following listing: "Process <pid> (<name>@<host>): <status>"
  [0.884470] [simix_kernel/INFO] Process 1 (0@host-0.hawaii.edu): waiting for communication synchro 0xfb4beb0 () in state 0 to finish
  [0.884470] [simix_kernel/INFO] Process 2 (1@host-1.hawaii.edu): waiting for communication synchro 0xfb4b0c0 () in state 0 to finish
  [0.884470] [simix_kernel/INFO] Process 3 (2@host-2.hawaii.edu): waiting for communication synchro 0xfb49760 () in state 0 to finish
  [0.884470] [simix_kernel/INFO] Process 4 (3@host-3.hawaii.edu): waiting for communication synchro 0xfb47590 () in state 0 to finish
  [0.884470] [simix_kernel/INFO] Process 5 (4@host-4.hawaii.edu): waiting for synchronization synchro 0xf8a1ae0 () in state 0 to finish
  [0.884470] [simix_kernel/INFO] Process 6 (5@host-5.hawaii.edu): waiting for synchronization synchro 0xf8a1f10 () in state 0 to finish
  [0.884470] [simix_kernel/INFO] Process 7 (6@host-6.hawaii.edu): waiting for synchronization synchro 0xf897500 () in state 0 to finish
  [0.884470] [simix_kernel/INFO] Process 8 (7@host-7.hawaii.edu): waiting for synchronization synchro 0xf89b190 () in state 0 to finish
  [0.884470] [simix_kernel/INFO] Process 9 (8@host-8.hawaii.edu): waiting for synchronization synchro 0xf8a3680 () in state 0 to finish
  [0.884470] [simix_kernel/INFO] Process 10 (9@host-9.hawaii.edu): waiting for synchronization synchro 0xf896280 () in state 0 to finish
  [0.884470] [simix_kernel/INFO] Process 11 (10@host-10.hawaii.edu): waiting for synchronization synchro 0xf8970d0 () in state 0 to finish
  [0.884470] [simix_kernel/INFO] Process 12 (11@host-11.hawaii.edu): waiting for synchronization synchro 0xf89b5c0 () in state 0 to finish
  [0.884470] [simix_kernel/INFO] Process 13 (12@host-12.hawaii.edu): waiting for synchronization synchro 0xf89ce30 () in state 0 to finish
  [0.884470] [simix_kernel/INFO] Process 14 (13@host-13.hawaii.edu): waiting for synchronization synchro 0xf89f530 () in state 0 to finish
  [0.884470] [simix_kernel/INFO] Process 15 (14@host-14.hawaii.edu): waiting for synchronization synchro 0xf89f100 () in state 0 to finish
  [0.884470] [simix_kernel/INFO] Process 16 (15@host-15.hawaii.edu): waiting for synchronization synchro 0xf8a0ca0 () in state 0 to finish
  ==3320== 
  ==3320== Process terminating with default action of signal 6 (SIGABRT)
  ==3320==    at 0x5619428: raise (raise.c:54)
  ==3320==    by 0x561B029: abort (abort.c:89)
  ==3320==    by 0x52347B8: xbt_abort (xbt_main.cpp:167)
  ==3320==    by 0x52F4768: SIMIX_run.part.110 (smx_global.cpp:569)
  ==3320==    by 0x52F6204: SIMIX_run (stl_algobase.h:224)
  ==3320==    by 0x5263E66: smpi_main (smpi_global.cpp:474)
  ==3320==    by 0x560482F: (below main) (libc-start.c:291)
  ==3320== 
  ==3320== HEAP SUMMARY:
  ==3320==     in use at exit: 136,159,788 bytes in 7,560 blocks
  ==3320==   total heap usage: 39,378 allocs, 31,818 frees, 140,230,437 bytes allocated
  ==3320== 
  ==3320== LEAK SUMMARY:
  ==3320==    definitely lost: 321 bytes in 4 blocks
  ==3320==    indirectly lost: 0 bytes in 0 blocks
  ==3320==      possibly lost: 134,294,280 bytes in 96 blocks
  ==3320==    still reachable: 1,865,187 bytes in 7,460 blocks
  ==3320==         suppressed: 0 bytes in 0 blocks
  ==3320== Rerun with --leak-check=full to see details of leaked memory
  ==3320== 
  ==3320== For counts of detected and suppressed errors, rerun with: -v
  ==3320== Use --track-origins=yes to see where uninitialised values come from
  ==3320== ERROR SUMMARY: 1147 errors from 24 contexts (suppressed: 0 from 0)
  valgrind --track-origins:yes ./xhpl --cfg=surf/precision:1e-9 --cfg=network/model:SMPI --cfg=network/TCP-gamma:4194304 --cfg=smpi/bcast:mpich --cfg=smpi/running-power:6217956542.969 --cfg=smpi/display-timing:yes --cfg=smpi/privatize-global-variables:yes --cfg=smpi/shared-malloc:local --cfg=smpi/privatize-global-variables:1 ./cluster_fat_tree_64.xml smpitmp-apprXPdW8
  Execution failed with code 134.
  #+end_example
- Note that this file has been obtained with a nearly-vanilla HPL (see the Github issue). No =smpi_usleep=, and shared
  =malloc= only for the matrix (no partial shared =malloc= for =PANEL->WORK=). Thus, it is quite strange to see such errors.
- The first error (=HPL_pipid.c:144=) happens because =PANEL->ia= is unitialized (checked by modifying the two operands one
  after the other to see if the error persists).
** 2017-04 April
** 2017-05 May
** 2017-06 June
*** 2017-06-01 Thursday
**** Redo validation of huge pages       :SMPI:EXPERIMENTS:HPL:REPORT:
- Simgrid commit: =9a8e2f5bce8c6758d4367d21a66466a497d136fe=
- HPL commit: =41774905395aebcb73650defaa7e2aa462e6e1a3=
- Script commit: =eb071f09d822e1031ea0776949058bf2f55cb94a=
- Compilation and execution for optimized HPL (made on =nova-10= without the huge pages, =nova-11= with the huge pages)
  #+begin_src sh
  make SMPI_OPTS="-DSMPI_OPTIMIZATION_LEVEL=4 -DSMPI_DGEMM_COEFFICIENT=1.742435e-10
  -DSMPI_DTRSM_COEFFICIENT=8.897459e-11" arch=SMPI
  #+end_src
  #+begin_src sh
  sysctl -w vm.overcommit_memory=1 && sysctl -w vm.max_map_count=40000000

  mount none /root/huge -t hugetlbfs -o rw,mode=0777 && echo 1 >> /proc/sys/vm/nr_hugepages
  #+end_src
  #+begin_src sh
  ./run_measures.py --global_csv result_size.csv --nb_runs 3 --size 50000,100000,150000,200000,250000,300000 --nb_proc
  64 --topo "2;16,32;1,16;1,1" --experiment HPL --running_power 5004882812.500 --nb_cpu 8

  ./run_measures.py --global_csv result_size.csv --nb_runs 3 --size 50000,100000,150000,200000,250000,300000 --nb_proc
  64 --topo "2;16,32;1,16;1,1" --experiment HPL --running_power 5004882812.500 --nb_cpu 8 --hugepage /root/huge
  #+end_src
- Analysis
  #+begin_src R :results output :session *R* :exports both
  library(ggplot2)
  library(gridExtra)
  old <- rbind(read.csv("validation/result_size_L4_big_nohugepage.csv"), read.csv("validation/result_size_L4_big_nohugepage_2.csv"))
  new <- read.csv("validation/result_size_L4_big_hugepage.csv")
  old$hugepage = FALSE
  new$hugepage =  TRUE
  results = rbind(old, new)
  #+end_src

  #+RESULTS:

  #+begin_src R :file validation/hugepage/1.pdf :results value graphics :results output :session *R* :exports both :width 6 :height 4
  do_plot(results, "size", "simulation_time", "hugepage", "Huge page", 64)
  #+end_src

  #+RESULTS:
  [[file:validation/hugepage/1.pdf]]

  #+begin_src R :file validation/hugepage/3.pdf :results value graphics :results output :session *R* :exports both :width 6 :height 4
  do_plot(results, "size", "memory_size", "hugepage", "Huge page", 64)
  #+end_src

  #+results:
  [[file:validation/hugepage/3.pdf]]

  #+begin_src R :file validation/hugepage/5.pdf :results value graphics :results output :session *R* :exports both :width 6 :height 4
  do_plot(results, "size", "Gflops", "hugepage", "Huge page", 64)
  #+end_src

  #+results:
  [[file:validation/hugepage/5.pdf]]

  #+begin_src R :file validation/hugepage/report_plot.pdf :results value graphics :results output :session *R* :exports both :width 12 :height 4
  grid_arrange_shared_legend(
      do_plot(results, "size", "simulation_time", "hugepage", "Huge page", 64),
      do_plot(results, "size", "memory_size", "hugepage", "Huge page", 64),
      nrow=1, ncol=2
  )
  #+end_src

  #+RESULTS:
  [[file:validation/hugepage/report_plot.pdf]]

  #+begin_src R :file validation/hugepage/2.png :results value graphics :results output :session *R* :exports both :width 800 :height 400
  plot1 = generic_do_plot(ggplot(results, aes(x=size, y=cpu_utilization, color=hugepage))) +
      ggtitle("CPU utilization for different matrix sizes\nUsing 64 MPI processes")
  plot2 = generic_do_plot(ggplot(results, aes(x=size, y=minor_page_fault, color=hugepage))) +
      ggtitle("Number of page faults for different matrix sizes\nUsing 64 MPI processes")
  grid.arrange(plot1, plot2, ncol=2)
  #+end_src

  #+RESULTS:
  [[file:validation/hugepage/2.png]]


  #+begin_src R :results output :session *R* :exports both
  library(data.table)
  aggregate_results <- function(results) {
      x = data.table(results)
      x = as.data.frame(x[, list(simulation_time=mean(simulation_time), Gflops=mean(Gflops), application_time=mean(application_time)), by=c("size", "nb_proc")])
      return(x[with(x, order(size, nb_proc)),])
  }
  aggr_old = aggregate_results(old)
  aggr_new = aggregate_results(new)
  aggr_new$Gflops_error = (aggr_new$Gflops - aggr_old$Gflops)/aggr_new$Gflops
  #+end_src

  #+begin_src R :file validation/hugepage/3.png :results value graphics :results output :session *R* :exports both :width 800 :height 400
  generic_do_plot(ggplot(aggr_new, aes(x=size, y=Gflops_error)))
  #+end_src

  #+RESULTS:
  [[file:validation/hugepage/3.png]]

  - The Gflops error is negligible.
  - The gain of using huge pages is pretty neat for both the simulation time and the memory consumption.
  - Very large variability of the CPU utilization, something weird has happened.
**** Scalability test                    :SMPI:EXPERIMENTS:HPL:REPORT:
- Simgrid commit: =9a8e2f5bce8c6758d4367d21a66466a497d136fe=
- HPL commit: =41774905395aebcb73650defaa7e2aa462e6e1a3=
- Script commit: =8cfd8d16787f39a29342b64599cf02166af6d632=
- Compilation and execution for optimized HPL (made on =nova-10= and =nova-11=)
  #+begin_src sh
  make SMPI_OPTS="-DSMPI_OPTIMIZATION_LEVEL=4 -DSMPI_DGEMM_COEFFICIENT=1.742435e-10
  -DSMPI_DTRSM_COEFFICIENT=8.897459e-11" arch=SMPI
  #+end_src
  #+begin_src sh
  sysctl -w vm.overcommit_memory=1 && sysctl -w vm.max_map_count=40000000

  mount none /root/huge -t hugetlbfs -o rw,mode=0777 && echo 1 >> /proc/sys/vm/nr_hugepages
  #+end_src
  #+begin_src sh
  ./run_measures.py --global_csv result_size_1000000_512.csv --nb_runs 1 --size 1000000 --nb_proc 512 --topo
  "2;16,32;1,16;1,1" --experiment HPL --running_power 5004882812.500 --nb_cpu 8 --hugepage /root/huge

  ./run_measures.py --global_csv result_size_1000000_1024.csv --nb_runs 1 --size 1000000 --nb_proc 1024 --topo
  "2;16,32;1,16;1,1" --experiment HPL --running_power 5004882812.500 --nb_cpu 8 --hugepage /root/huge

  ./run_measures.py --global_csv result_size_1000000_2048.csv --nb_runs 1 --size 1000000 --nb_proc 2048 --topo
  "2;16,32;1,16;1,1" --experiment HPL --running_power 5004882812.500 --nb_cpu 8 --hugepage /root/huge

  ./run_measures.py --global_csv result_size_2000000_512.csv --nb_runs 1 --size 2000000 --nb_proc 512 --topo
  "2;16,32;1,16;1,1" --experiment HPL --running_power 5004882812.500 --nb_cpu 8 --hugepage /root/huge

  ./run_measures.py --global_csv result_size_2000000_1024.csv --nb_runs 1 --size 2000000 --nb_proc 1024 --topo
  "2;16,32;1,16;1,1" --experiment HPL --running_power 5004882812.500 --nb_cpu 8 --hugepage /root/huge

  ./run_measures.py --global_csv result_size_2000000_2048.csv --nb_runs 1 --size 2000000 --nb_proc 2048 --topo
  "2;16,32;1,16;1,1" --experiment HPL --running_power 5004882812.500 --nb_cpu 8 --hugepage /root/huge
  #+end_src
- Results:
  #+begin_src R :results output :session *R* :exports both
  rbind(
      read.csv('scalability/result_1000000_512.csv'),
      read.csv('scalability/result_1000000_1024.csv'),
      read.csv('scalability/result_1000000_2048.csv'),
      read.csv('scalability/result_2000000_512.csv'),
      read.csv('scalability/result_2000000_1024.csv'),
      read.csv('scalability/result_2000000_2048.csv')
  )
  #+end_src

  #+RESULTS:
  #+begin_example
            topology nb_roots nb_proc    size full_time      time Gflops
  1 2;16,32;1,16;1,1       16     512 1000000    716521  716521.0  930.4
  2 2;16,32;1,16;1,1       16    1024 1000000    363201  363201.0 1836.0
  3 2;16,32;1,16;1,1       16    2048 1000000    186496  186495.7 3575.0
  4 2;16,32;1,16;1,1       16     512 2000000   5685080 5685077.7  938.1
  5 2;16,32;1,16;1,1       16    1024 2000000   2861010 2861012.5 1864.0
  6 2;16,32;1,16;1,1       16    2048 2000000   1448900 1448899.1 3681.0
    simulation_time application_time user_time system_time major_page_fault
  1         2635.10           500.97   2367.19      259.91                0
  2         6037.89          1036.96   5515.36      515.05                0
  3        12391.90          2092.95  11389.36      995.39                0
  4         6934.86          1169.66   6193.80      683.73                0
  5        15198.30          2551.10  13714.01     1430.93                0
  6        32263.60          5236.56  29357.92     2844.89                0
    minor_page_fault cpu_utilization        uss        rss page_table_size
  1          1916208            0.99  153665536 2317279232        10600000
  2          2002989            0.99  369676288 4837175296        21252000
  3          2154982            0.99 1010696192 7774138368        42908000
  4          3801905            0.99  150765568 2758770688        10604000
  5          3872820            0.99  365555712 5273034752        21220000
  6          4038099            0.99 1009606656 7415914496        42884000
    memory_size
  1   894443520
  2  1055309824
  3  1581170688
  4  3338420224
  5  3497111552
  6  4027408384
  #+end_example
**** Add the Stampede output file in the repository                  :HPL:
- File [[file:fullrun2.run1.notestmode.20000m.log]]
*** 2017-06-02 Friday
**** DONE New scalability tests to run [6/6]                    :SMPI:HPL:
:LOGBOOK:
- State "DONE"       from "TODO"       [2017-06-05 Mon 19:33]
- State "TODO"       from "TODO"       [2017-06-04 Sun 20:22]
- State "TODO"       from "TODO"       [2017-06-04 Sun 20:22]
- State "TODO"       from "TODO"       [2017-06-04 Sun 20:22]
- State "TODO"       from "TODO"       [2017-06-03 Sat 19:05]
- State "TODO"       from "TODO"       [2017-06-03 Sat 19:05]
- State "TODO"       from              [2017-06-02 Fri 09:48]
:END:
- [X] N=1000000, nbproc=4096, expected time \approx 206min \times 2.2 \approx 7.5h
- [X] N=2000000, nbproc=4096, expected time \approx 537min \times 2.2 \approx 19.7h
- [X] N=4000000, nbproc=512,  expected time \approx 115min \times 2.6 \approx 5h
- [X] N=4000000, nbproc=1024, expected time \approx 253min \times 2.6 \approx 11h
- [X] N=4000000, nbproc=2048, expected time \approx 537min \times 2.6 \approx 23.3h
- [X] N=4000000, nbproc=4096, expected time \approx 537min \times 2.6 \times 2.2 \approx 51h
**** Cannot connect anymore on G5K nodes in Lyon                 :BUG:G5K:
- Reserved a job and made a deployment in =lyon=. Then, *cannot* connect to the node (both as =tocornebize= and as =root=).
- Reserved a job and made a deployment in =grenoble=. Then, *can* connect to the node (both as =tocornebize= and as =root=).
- Looked at the =.ssh= directories of =grenoble= and =lyon=, they look the same.
- Can =ssh= from =lyon= to =grenoble= (or any other site) but cannot =ssh= from =grenoble= (or any other site) to =lyon=.
- Fixed by replacing the =.ssh= folder from =lyon= by the =.ssh= folder from =grenoble= (might have messed up something...).
**** First capacity planning test            :SMPI:EXPERIMENTS:HPL:REPORT:
- Simgrid commit: =9a8e2f5bce8c6758d4367d21a66466a497d136fe=
- HPL commit: =41774905395aebcb73650defaa7e2aa462e6e1a3=
- Script commit: =4ff3ccbcbb77e126e454a16dea0535493ff1ff0b=
- Compilation and execution (on =nova-6= and =nova-8=).
  #+begin_src sh
  make SMPI_OPTS="-DSMPI_OPTIMIZATION_LEVEL=4 -DSMPI_DGEMM_COEFFICIENT=1.742435e-10
  -DSMPI_DTRSM_COEFFICIENT=8.897459e-11" arch=SMPI
  #+end_src
  #+begin_src sh
  sysctl -w vm.overcommit_memory=1 && sysctl -w vm.max_map_count=40000000

  mount none /root/huge -t hugetlbfs -o rw,mode=0777 && echo 1 >> /proc/sys/vm/nr_hugepages
  #+end_src
  #+begin_src sh
  ./run_measures.py --global_csv result_capacity_50000.csv --nb_runs 1 --size 50000 --nb_proc 512 --topo "2;16,32;1,1:16;1,1"
  --experiment HPL --running_power 5004882812.500 --hugepage /root/huge

  ./run_measures.py --global_csv result_capacity_100000.csv --nb_runs 1 --size 100000 --nb_proc 512 --topo "2;16,32;1,1:16;1,1"
  --experiment HPL --running_power 5004882812.500 --hugepage /root/huge
  #+end_src
- Results:
  #+begin_src R :results output :session *R* :exports both
  library(ggplot2)
  results <- rbind(read.csv("capacity_planning/result_capacity_50000.csv"), read.csv("capacity_planning/result_capacity_100000.csv"))
  #+end_src

  #+begin_src R :file capacity_planning/1.png :results value graphics :results output :session *R* :exports both :width 800 :height 400
  ggplot(results, aes(x=nb_roots, y=Gflops, color=size, group=size)) +
      stat_summary(fun.y = mean, geom="line")+
      stat_summary(fun.y = mean, geom="point")+
      expand_limits(x=0, y=0)+
      ggtitle("Gflops estimation for different number of root switches and matrix sizes\nUsing 512 MPI processes")
  #+end_src

  #+RESULTS:
  [[file:capacity_planning/1.png]]

- In this experiment, we use a fat-tree which has a total of 512 nodes, all having only one core. We use 512 processes,
  one per node. We change the number of up-ports of the L1 switches and therefore the number of L2 switches.
- It is strange, there is apparently no impact on the performances of HPL, we get the same performances with only one L2
  switch than with 16 L2 switches.
- Maybe we could try with a bigger matrix, to maybe have some network contention? But the experiment might take some time.
- We could also try with a hostfile randomly shuffled, to maybe have a less good mapping and thus more traffic going
  through the L2 switches?
- We could also try a fat-tree more “high” and less “wide”. We could have a third layer of switches, but decrease the
  number of ports to keep the same number of nodes. For instance, =3;8,8,8;1,8,16;1,1,1= instead of =2;16,32;1,16;1,1= (all
  have 512 nodes). But it is a bit artificial, such topology would certainly never happen in “real life”.
*** 2017-06-03 Saturday
**** New scalability tests                   :SMPI:EXPERIMENTS:HPL:REPORT:
- Simgrid commit: =9a8e2f5bce8c6758d4367d21a66466a497d136fe=
- HPL commit: =41774905395aebcb73650defaa7e2aa462e6e1a3=
- Script commit: =4ff3ccbcbb77e126e454a16dea0535493ff1ff0b=
- Compilation and execution (made on =nova-5=, =nova-11=, =nova-13=, =nova-14=):
  #+begin_src sh
  make SMPI_OPTS="-DSMPI_OPTIMIZATION_LEVEL=4 -DSMPI_DGEMM_COEFFICIENT=1.742435e-10
  -DSMPI_DTRSM_COEFFICIENT=8.897459e-11" arch=SMPI
  #+end_src
  #+begin_src sh
  sysctl -w vm.overcommit_memory=1 && sysctl -w vm.max_map_count=2000000000

  mount none /root/huge -t hugetlbfs -o rw,mode=0777 && echo 1 >> /proc/sys/vm/nr_hugepages
  #+end_src
  #+begin_src sh
  ./run_measures.py --global_csv result_size_1000000_4096.csv --nb_runs 1 --size 1000000 --nb_proc 4096 --topo
  "2;16,32;1,16;1,1;8" --experiment HPL --running_power 5004882812.500 --hugepage /root/huge

  ./run_measures.py --global_csv result_size_4000000_512.csv --nb_runs 1 --size 4000000 --nb_proc 512 --topo
  "2;16,32;1,16;1,1;8" --experiment HPL --running_power 5004882812.500 --hugepage /root/huge

  ./run_measures.py --global_csv result_size_4000000_1024.csv --nb_runs 1 --size 4000000 --nb_proc 1024 --topo
  "2;16,32;1,16;1,1;8" --experiment HPL --running_power 5004882812.500 --hugepage /root/huge

  ./run_measures.py --global_csv result_size_2000000_4096.csv --nb_runs 1 --size 2000000 --nb_proc 4096 --topo
  "2;16,32;1,16;1,1;8" --experiment HPL --running_power 5004882812.500 --hugepage /root/huge

  ./run_measures.py --global_csv result_size_4000000_2048.csv --nb_runs 1 --size 4000000 --nb_proc 2048 --topo
  "2;16,32;1,16;1,1;8" --experiment HPL --running_power 5004882812.500 --hugepage /root/huge

  ./run_measures.py --global_csv result_size_4000000_4096.csv --nb_runs 1 --size 4000000 --nb_proc 4096 --topo
  "2;16,32;1,16;1,1;8" --experiment HPL --running_power 5004882812.500 --hugepage /root/huge

  #+end_src

  #+begin_src R :results output :session *R* :exports both
  rbind(
      read.csv('scalability/result_500000_512.csv'),
      read.csv('scalability/result_500000_1024.csv'),
      read.csv('scalability/result_500000_2048.csv'),
      read.csv('scalability/result_500000_4096.csv'),
      read.csv('scalability/result_1000000_4096.csv'),
      read.csv('scalability/result_2000000_4096.csv'),
      read.csv('scalability/result_4000000_512.csv'),
      read.csv('scalability/result_4000000_1024.csv'),
      read.csv('scalability/result_4000000_2048.csv'),
      read.csv('scalability/result_4000000_4096.csv')
  )
  #+end_src

  #+RESULTS:
  #+begin_example
               topology nb_roots nb_proc    size  full_time        time Gflops
  1  2;16,32;1,16;1,1;8       16     512  500000    91246.1    91246.02  913.3
  2  2;16,32;1,16;1,1;8       16    1024  500000    46990.1    46990.02 1773.0
  3  2;16,32;1,16;1,1;8       16    2048  500000    24795.5    24795.50 3361.0
  4  2;16,32;1,16;1,1;8       16    4096  500000    13561.0    13561.01 6145.0
  5  2;16,32;1,16;1,1;8       16    4096 1000000    97836.6    97836.54 6814.0
  6  2;16,32;1,16;1,1;8       16    4096 2000000   742691.0   742690.59 7181.0
  7  2;16,32;1,16;1,1;8       16     512 4000000 45305100.0 45305083.56  941.8
  8  2;16,32;1,16;1,1;8       16    1024 4000000 22723800.0 22723820.45 1878.0
  9  2;16,32;1,16;1,1;8       16    2048 4000000 11432900.0 11432938.62 3732.0
  10 2;16,32;1,16;1,1;8       16    4096 4000000  5787160.0  5787164.09 7373.0
     simulation_time application_time user_time system_time major_page_fault
  1          1191.99          204.992   1098.25       93.12                0
  2          2482.28          441.897   2296.51      184.70                0
  3          5091.97          872.425   4741.26      349.79                0
  4         11321.60         1947.320  10640.63      679.53                0
  5         26052.50         4362.660  24082.38     1966.10                0
  6         64856.30        10643.600  59444.40     5402.24                0
  7         17336.50         3030.400  15090.31     1945.23                0
  8         38380.90         6435.870  34249.71     3827.36                0
  9         83535.20        13080.500  75523.95     7684.52                0
  10       169659.00        26745.400 154314.76    15085.08                0
     minor_page_fault cpu_utilization        uss         rss page_table_size
  1            960072            0.99  155148288  2055086080        10604000
  2           1054062            0.99  369696768  4383203328        21240000
  3           1282294            0.99 1012477952  9367576576        42912000
  4           1852119            0.99 3103875072 15318568960        87740000
  5           2768705            0.99 3103895552 16934834176        87748000
  6           4704339            0.99 3102445568 19464646656        87748000
  7           7663911            0.98  151576576  2056916992        10604000
  8           7725625            0.99  369872896  4120702976        21212000
  9           7917525            0.99 1012191232  9221050368        42880000
  10          8550745            0.99 3113381888 20408209408        87808000
     memory_size
  1    282558464
  2    429948928
  3    962826240
  4   2814042112
  5   3425406976
  6   5910134784
  7  13079060480
  8  13275557888
  9  13825183744
  10 15763668992
#+end_example
- +Memory measurement failed for the experiments with 4096 nodes (=smpimain= took too much time to start, so its PID was+
  +not found by =run_measure.py= at the beginning, so it assumed it was already terminated... really need to find something+
  +more robust).+
- For the record, ran this command on the nodes (same command used in the script to estimate the memory consumption):
  #+begin_src sh
  python3 -c "import psutil; print(psutil.virtual_memory().available)"
  #+end_src
- Result:
  + For size=2000000 and nbproc=4096: 60468817920
  + For size=4000000 and nbproc=1024: 53105373184
  + For size=4000000 and nbproc=2048: 52539293696
  + For size=4000000 and nbproc=4096: 50614239232
- On a freshly deployed node, the same command returns 66365100032
*** 2017-06-04 Sunday
**** Investigate capacity planning: small test program          :SMPI:HPL:
- As mentionned in [2017-06-02 Fri], the duration of HPL does not seem to be impacted by the topology, which is strange.
- Implemented a small test program, called =network_test=. It takes as argument a size and a number of iterations. Every
  process sends the given number of messages, each having the given size, to the next process (and thus receives from
  the previous one).
- Tested with the following topology (only changing the fat-tree description):
  #+begin_example
  <?xml version='1.0' encoding='ASCII'?>
  <!DOCTYPE platform SYSTEM "http://simgrid.gforge.inria.fr/simgrid/simgrid.dtd">
  <!--2-level fat-tree with 16 nodes-->
  <platform version="4">
    <AS id="AS0" routing="Full">
      <cluster id="cluster0" prefix="host-" suffix=".hawaii.edu" radical="0-15" speed="1Gf" bw="10Gbps" lat="2.4E-5s" loopback_bw="5120MiBps" loopback_lat="1.5E-9s" core="1" topology="FAT_TREE" topo_parameters="2;4,4;1,4;1,1"/>
    </AS>
  </platform>
  #+end_example
- Results for one iteration:
  + With a size of =200000000= and the fat-tree =2;4,4;1,4;1,1=, takes a time of 1.28   seconds.
  + With a size of =200000000= and the fat-tree =2;4,4;1,1;1,1=, takes a time of 2.69   seconds.
  + With a size of =200000=    and the fat-tree =2;4,4;1,4;1,1=, takes a time of 0.0025 seconds.
  + With a size of =200000=    and the fat-tree =2;4,4;1,1;1,1=, takes a time of 0.0040 seconds.
  + With a size of =2000=      and the fat-tree =2;4,4;1,4;1,1=, takes a time of 0.0004 seconds.
  + With a size of =2000=      and the fat-tree =2;4,4;1,1;1,1=, takes a time of 0.0004 seconds.
- Thus, for large enough size, the difference is very clear, the topology *does* have a high impact. For small messages
  however, this is not the case.
- It does not seem to change for several iterations.
**** TODO Check whas are the sizes of the messages in HPL.     :SMPI:HPL:
:LOGBOOK:
- State "TODO"       from              [2017-06-04 Sun 19:44]
:END:
**** Investigate capacity planning: odd networks                :SMPI:HPL:
- Simgrid commit: =9a8e2f5bce8c6758d4367d21a66466a497d136fe=
- HPL commit: =41774905395aebcb73650defaa7e2aa462e6e1a3=
- Script commit: =4ff3ccbcbb77e126e454a16dea0535493ff1ff0b=
- Try several topologies for HPL with absurdly good or bad networks (e.g. high/null bandwidth and/or high/null latency).
- The idea is that if doing so has a little impact on performances, then it is hopeless to observe any impact from
  adding/removing switches.
- Quick and dirty experiments: do not add any option to the script, just modify the values in =topology.py= (lines
  161-164).
- Note that in the previous experiments, where nearly no impact was observed, the different values were:
  #+begin_src python
  bw = '10Gbps'
  lat = '2.4E-5s'
  loopback_bw = '5120MiBps'
  loopback_lat = '1.5E-9s'
  #+end_src
- Run this command, which outputs the Gflops:
  #+begin_src sh
  ./run_measures.py --global_csv /tmp/bla.csv --nb_runs 1 --size 10000 --nb_proc 16 --topo "2;4,4;1,4;1,1" --experiment
  HPL --running_power 6217956542.969 && tail -n 1 /tmp/bla.csv | cut -f10 -d','
  #+end_src
- Result for the same network characteristics: 21.96
- Results with other characteristics:
  + Very high bandwidth: 22.15
    #+begin_src python
    bw = '1000000Gbps'
    lat = '2.4E-5s'
    loopback_bw = '1000000GBps'
    loopback_lat = '1.5E-9s'
    #+end_src
  + Very low bandwidth: 1.505
    #+begin_src python
    bw = '10Mbps'
    lat = '2.4E-5s'
    loopback_bw = '10Mbps'
    loopback_lat = '1.5E-9s'
    #+end_src
  + Low bandwidth: 19.95
    #+begin_src python
    bw = '1Gbps'
    lat = '2.4E-5s'
    loopback_bw = '512MiBps'
    loopback_lat = '1.5E-9s'
    #+end_src
  + Very low latency: 25.95
    #+begin_src python
    bw = '10Gbps'
    lat = '0s'
    loopback_bw = '5120MiBps'
    loopback_lat = '0s'
    #+end_src
  + Very high latency: 0.1534
    #+begin_src python
    bw = '10Gbps'
    lat = '2.4E-2s'
    loopback_bw = '5120MiBps'
    loopback_lat = '1.5E-5s
    #+end_src
  + High latency: 9.477
    #+begin_src python
    bw = '10Gbps'
    lat = '2.4E-4s'
    loopback_bw = '5120MiBps'
    loopback_lat = '1.5E-8s'
    #+end_src
- Improving the network performances has a limited impact. Using a nearly infinite bandwidth increases the Gflops by
  less than 1%. Using a null latency has more impact, but still limited, it increases the Gflops by 18%.
- Degrading the network performances has more impact. Using a bandwidth 1000 times lower divides by 15 the Gflops, but
  using a bandwidth 10 times lower decreases the Gflops by only 9%. Both the very high latency and the high latency have
  a great impact.
- To sum up, the latency seems to have an higher impact on HPL performances than the bandwidth.
- It is not clear if the contention created by using less switches will only decrease the bandwidth, or also increase
  the latency. It depends if there is on the switches one queue per port, or one queue for all the ports (in the former
  case, contention will have a much lower impact on the latency than in the later case).
- Hypothesis: in the case of a one queue per port model, removing switches will not increase too much the latency and
  therefore have a very limited impact on HPL performances.
**** TODO Ask what model is used in Simgrid’s switches           :SIMGRID:
:LOGBOOK:
- State "TODO"       from              [2017-06-04 Sun 19:40]
:END:
- Is it one queue per port, or one single queue for all the ports?
**** More thoughts on capacity planning                         :SMPI:HPL:
- The plot of the Gflops as a function of the bandwidth (resp. inverse of latency) seems to look like the plot of the
  Gflops as a function of the number of processes or the size. It is a concave function converging to some finite limit.
- In the settings currently used for HPL, the bandwidth of 10Gbps seems to be already very close th the limit (since
  using a bandwidth thousands of time larger has little to no impact). This is why decreasing the bandwidth a bit has a
  very little impact. If we want to observe something when we remove switches, we should use lower bandwidths.
- Quick test, using the same command than previous section and with these values:
  #+begin_src python
  bw = '10Mbps'
  lat = '2.4E-5s'
  loopback_bw = '5120MiBps'
  loopback_lat = '1.5E-9s'
  #+end_src
  + With =2;4,4;1,4;1,1=, 1.505 Gflops.
  + With =2;4,4;1,1;1,1=, 1.025 Gflops.
  + With =2;4,4;1,4;1,1= and a random mapping, 1.268 Gflops.
  + With =2;4,4;1,1;1,1= and a random mapping, 0.6464 Gflops.
- The hypothesis seems to be confirmed. With a lower bandwidth, a difference of bandwidth has much more impact. Thus,
  removing a switch and/or using a random mapping has also much more impact.
*** 2017-06-05 Monday
**** Comparison with real Taurus experiment  :SMPI:EXPERIMENTS:HPL:REPORT:
- File [[file:hpl_analysis/taurus/real.csv]] holds real experiment data. It has been created manually, thanks to the energy
  paper [[https://gitlab.inria.fr/fheinric/paper-simgrid-energy/tree/master/experiments/mpi_hpl_dvfs/taurus_2017-01-17/original-data][repository]].

  #+begin_src R :results output :session *R* :exports both
  library(ggplot2)
  library(reshape2)
  library(gridExtra)

  get_results <- function(nb_proc) {
      result <- read.csv(paste('hpl_analysis/taurus/hpl_paper_', nb_proc, '.csv', sep=''))
      result$full_time = max(result$time)
      result$total_energy = sum(result$power_consumption)

      used_energy = 0
      result = result[with(result, order(-power_consumption)),] # sort by power consumption
      result$used_energy = sum(head(result, nb_proc/12)$power_consumption)
      result$nb_proc = nb_proc
      return(unique(result[c('nb_proc', 'full_time', 'total_energy', 'used_energy')]))
  }
  simulation_vanilla_results = data.frame()
  #  for(i in (c(1,4,8,12,48,96,144))) {
  for(i in (c(12,48,96,144))) {
    simulation_vanilla_results = rbind(simulation_vanilla_results, get_results(i))
  }
  simulation_vanilla_results$type = 'Vanilla simulation'
  simulation_vanilla_results$time = -1 # do not have it
  simulation_vanilla_results$Gflops = -1 # do not have it

  real_results = read.csv('hpl_analysis/taurus/real.csv')
  real_results$type = 'Real execution'
  real_results$used_energy = real_results$used_energy * 1e3 # kJ -> J
  sim_results <- read.csv('hpl_analysis/taurus/hpl2.csv')
  sim_results$type = 'Optimized simulation'
  results = rbind(real_results[c('nb_proc', 'full_time', 'time', 'Gflops', 'used_energy', 'type')],
                  sim_results[c('nb_proc', 'full_time', 'time', 'Gflops', 'used_energy', 'type')],
                  simulation_vanilla_results[c('nb_proc', 'full_time', 'time', 'Gflops', 'used_energy', 'type')])
  results$type <- factor(results$type, levels = c('Optimized simulation', 'Vanilla simulation', 'Real execution'))
  #+end_src

  #+RESULTS:

  #+begin_src R :file hpl_analysis/taurus/validation.pdf :results value graphics :results output :session *R* :exports both :width 12 :height 4
  p1 = generic_do_plot(ggplot(results, aes(x=nb_proc, y=full_time, color=type, shape=type)), fixed_shape=FALSE) +
           xlab("Number of processes")+
           ylab("Duration (seconds)")+
           scale_shape_manual(values = c(0, 1, 2))+
           labs(colour="Experiment type")+
           labs(shape="Experiment type")+
           ggtitle("HPL duration for different numbers of processes\nMatrix size: 20,000")
  p2 = generic_do_plot(ggplot(results, aes(x=nb_proc, y=used_energy, color=type, shape=type)), fixed_shape=FALSE) +
           xlab("Number of processes")+
           ylab("Energy consumption (joules)")+
           scale_shape_manual(values = c(0, 1, 2))+
           labs(colour="Experiment type")+
           labs(shape="Experiment type")+
           ggtitle("HPL energy consumption for different numbers of processes\nMatrix size: 20,000")
  grid_arrange_shared_legend(p1, p2, nrow=1, ncol=2)
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/taurus/validation.pdf]]

  #+begin_src R :file hpl_analysis/taurus/validation2.pdf :results value graphics :results output :session *R* :exports both :width 12 :height 4
  tmp_results = results[results$type != "Vanilla simulation",]
  grid_arrange_shared_legend(
      generic_do_plot(ggplot(tmp_results, aes(x=nb_proc, y=time, color=type))) +
        xlab("Number of processes")+
        ylab("Duration (seconds)")+
        labs(colour="Simulated")+
        ggtitle("HPL “short” duration for different numbers of processes\nMatrix size: 20,000"),
      generic_do_plot(ggplot(tmp_results, aes(x=nb_proc, y=Gflops, color=type))) +
        xlab("Number of processes")+
        ylab("Energy consumption (joules)")+
        labs(colour="Simulated")+
        ggtitle("HPL performances for different numbers of processes\nMatrix size: 20,000"),
      nrow=1, ncol=2
  )
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/taurus/validation2.pdf]]

  #+begin_src R :results output :session *R* :exports both
  library(data.table)
  aggregate_results <- function(results) {
      x = data.table(results)
      x = x[x$nb_proc %in% c(12, 48, 96, 144)]
      x = as.data.frame(x[, list(time=mean(full_time), energy=mean(used_energy)), by=c("nb_proc")])
      return(x[with(x, order(nb_proc)),])
  }
  aggr_real = aggregate_results(real_results)
  aggr_sim = aggregate_results(sim_results)
  aggr_vanilla = aggregate_results(simulation_vanilla_results)
  aggr_sim$time_error = (aggr_sim$time - aggr_real$time)/aggr_real$time * 100
  aggr_sim$energy_error = (aggr_sim$energy - aggr_real$energy)/aggr_real$energy * 100
  aggr_sim$optimized = TRUE
  aggr_vanilla$time_error = (aggr_vanilla$time - aggr_real$time)/aggr_real$time * 100
  aggr_vanilla$energy_error = (aggr_vanilla$energy - aggr_real$energy)/aggr_real$energy * 100
  aggr_vanilla$optimized = FALSE
  aggr_results = rbind(aggr_vanilla, aggr_sim)
  aggr_results$optimized <- factor(aggr_results$optimized, levels = c(TRUE, FALSE))
  #+end_src

  #+RESULTS:

- Get the three colors used for the previous plots to use the ones corresponding to vanilla and optimized.
  #+begin_src R :results output :session *R* :exports both
  x = unique(ggplot_build(p1)$data[[1]]$colour)
  x
  colors = x[c(1, 2)]
  colors
  #+end_src

  #+RESULTS:
  : [1] "#F8766D" "#00BA38" "#619CFF"
  : [1] "#F8766D" "#00BA38"

  #+begin_src R :file hpl_analysis/taurus/errors.pdf :results value graphics :results output :session *R* :exports both :width 12 :height 4
  grid_arrange_shared_legend(
      generic_do_plot(ggplot(aggr_results, aes(x=nb_proc, y=time_error, color=optimized))) +
        geom_hline(yintercept=0) +
        scale_color_manual(values=colors) +
        xlab("Number of processes")+
        ylab("Relative error (percent)")+
        labs(colour="Optimized simulation")+
        ggtitle("Error on the duration prediction")+
        expand_limits(y=15)+
        expand_limits(y=-15),
      generic_do_plot(ggplot(aggr_results, aes(x=nb_proc, y=energy_error, color=optimized))) +
        geom_hline(yintercept=0) +
        scale_color_manual(values=colors) +
        xlab("Number of processes")+
        ylab("Relative error (percent)")+
        labs(colour="Optimized simulation")+
        ggtitle("Error on the energy consumption prediction")+
        expand_limits(y=15)+
        expand_limits(y=-15),
      nrow=1, ncol=2
  )
  #+end_src

  #+RESULTS:
  [[file:hpl_analysis/taurus/errors.pdf]]

- The plots are funny. The shapes of the error plots for optimized and vanilla look similar, but shifted. They both
  reach some high errors (~ 10%), but not for the same number of processes. Also, the optimized version is always above
  0 while the vanill is below 0 for some points.
- There are some mismatches between time prediction and energy prediction. For instance, optimized has a large error for
  the time prediction of 144 processes, but nearly no error for the energy prediction. Similarly, vanilla
  over-estimates the duration for 48 processes but under-estimates the energy consumption, which seems odd.
**** Plots for scalability test              :SMPI:EXPERIMENTS:HPL:REPORT:
#+begin_src R :results output :session *R* :exports both
library(ggplot2)
library(ggrepel)
library(reshape2)
library(gridExtra)
results = rbind(
    read.csv('scalability/result_500000_512.csv'),
    read.csv('scalability/result_500000_1024.csv'),
    read.csv('scalability/result_500000_2048.csv'),
    read.csv('scalability/result_500000_4096.csv'),
    read.csv('scalability/result_1000000_512.csv'),
    read.csv('scalability/result_1000000_1024.csv'),
    read.csv('scalability/result_1000000_2048.csv'),
    read.csv('scalability/result_1000000_4096.csv'),
    read.csv('scalability/result_2000000_512.csv'),
    read.csv('scalability/result_2000000_1024.csv'),
    read.csv('scalability/result_2000000_2048.csv'),
    read.csv('scalability/result_2000000_4096.csv'),
    read.csv('scalability/result_4000000_512.csv'),
    read.csv('scalability/result_4000000_1024.csv'),
    read.csv('scalability/result_4000000_2048.csv'),
    read.csv('scalability/result_4000000_4096.csv')
)
results$simulation_time = results$simulation_time/3600
results$memory_size = results$memory_size * 1e-9
number_verb <- function(n) {
    return(format(n,big.mark=",",scientific=FALSE))
}
results$size_verb = factor(unlist(lapply(results$size, number_verb)), levels = c('500,000','1,000,000','2,000,000','4,000,000'))
results$nb_proc_verb = factor(unlist(lapply(results$nb_proc, number_verb)), levels = c('512', '1,024', '2,048', '4,096'))
results
#+end_src

#+RESULTS:
#+begin_example
             topology nb_roots nb_proc    size  full_time        time Gflops
1  2;16,32;1,16;1,1;8       16     512  500000    91246.1    91246.02  913.3
2  2;16,32;1,16;1,1;8       16    1024  500000    46990.1    46990.02 1773.0
3  2;16,32;1,16;1,1;8       16    2048  500000    24795.5    24795.50 3361.0
4  2;16,32;1,16;1,1;8       16    4096  500000    13561.0    13561.01 6145.0
5    2;16,32;1,16;1,1       16     512 1000000   716521.0   716521.00  930.4
6    2;16,32;1,16;1,1       16    1024 1000000   363201.0   363201.04 1836.0
7    2;16,32;1,16;1,1       16    2048 1000000   186496.0   186495.70 3575.0
8  2;16,32;1,16;1,1;8       16    4096 1000000    97836.6    97836.54 6814.0
9    2;16,32;1,16;1,1       16     512 2000000  5685080.0  5685077.72  938.1
10   2;16,32;1,16;1,1       16    1024 2000000  2861010.0  2861012.55 1864.0
11   2;16,32;1,16;1,1       16    2048 2000000  1448900.0  1448899.09 3681.0
12 2;16,32;1,16;1,1;8       16    4096 2000000   742691.0   742690.59 7181.0
13 2;16,32;1,16;1,1;8       16     512 4000000 45305100.0 45305083.56  941.8
14 2;16,32;1,16;1,1;8       16    1024 4000000 22723800.0 22723820.45 1878.0
15 2;16,32;1,16;1,1;8       16    2048 4000000 11432900.0 11432938.62 3732.0
16 2;16,32;1,16;1,1;8       16    4096 4000000  5787160.0  5787164.09 7373.0
   simulation_time application_time user_time system_time major_page_fault
1        0.3311083          204.992   1098.25       93.12                0
2        0.6895222          441.897   2296.51      184.70                0
3        1.4144361          872.425   4741.26      349.79                0
4        3.1448889         1947.320  10640.63      679.53                0
5        0.7319722          500.970   2367.19      259.91                0
6        1.6771917         1036.960   5515.36      515.05                0
7        3.4421944         2092.950  11389.36      995.39                0
8        7.2368056         4362.660  24082.38     1966.10                0
9        1.9263500         1169.660   6193.80      683.73                0
10       4.2217500         2551.100  13714.01     1430.93                0
11       8.9621111         5236.560  29357.92     2844.89                0
12      18.0156389        10643.600  59444.40     5402.24                0
13       4.8156944         3030.400  15090.31     1945.23                0
14      10.6613611         6435.870  34249.71     3827.36                0
15      23.2042222        13080.500  75523.95     7684.52                0
16      47.1275000        26745.400 154314.76    15085.08                0
   minor_page_fault cpu_utilization        uss         rss page_table_size
1            960072            0.99  155148288  2055086080        10604000
2           1054062            0.99  369696768  4383203328        21240000
3           1282294            0.99 1012477952  9367576576        42912000
4           1852119            0.99 3103875072 15318568960        87740000
5           1916208            0.99  153665536  2317279232        10600000
6           2002989            0.99  369676288  4837175296        21252000
7           2154982            0.99 1010696192  7774138368        42908000
8           2768705            0.99 3103895552 16934834176        87748000
9           3801905            0.99  150765568  2758770688        10604000
10          3872820            0.99  365555712  5273034752        21220000
11          4038099            0.99 1009606656  7415914496        42884000
12          4704339            0.99 3102445568 19464646656        87748000
13          7663911            0.98  151576576  2056916992        10604000
14          7725625            0.99  369872896  4120702976        21212000
15          7917525            0.99 1012191232  9221050368        42880000
16          8550745            0.99 3113381888 20408209408        87808000
   memory_size size_verb nb_proc_verb
1    0.2825585   500,000          512
2    0.4299489   500,000        1,024
3    0.9628262   500,000        2,048
4    2.8140421   500,000        4,096
5    0.8944435 1,000,000          512
6    1.0553098 1,000,000        1,024
7    1.5811707 1,000,000        2,048
8    3.4254070 1,000,000        4,096
9    3.3384202 2,000,000          512
10   3.4971116 2,000,000        1,024
11   4.0274084 2,000,000        2,048
12   5.9101348 2,000,000        4,096
13  13.0790605 4,000,000          512
14  13.2755579 4,000,000        1,024
15  13.8251837 4,000,000        2,048
16  15.7636690 4,000,000        4,096
#+end_example

#+begin_src R :file scalability/1.pdf :results value graphics :results output :session *R* :exports both :width 6 :height 4
size_time = generic_do_plot(ggplot(results, aes(x=size, y=simulation_time, color=nb_proc_verb))) +
    xlab("Matrix size") +
    ylab("Simulation time (hours)") +
    labs(colour="Number of processes")+
    ggtitle("Simulation time for different matrix sizes")+
    theme(legend.position = "none")+
    geom_text_repel(
        data = subset(results, size == max(size)),
        aes(label = nb_proc_verb),
        nudge_x = 45,
        segment.color = NA,
        show.legend = FALSE
      )
size_time
#+end_src

#+RESULTS:
[[file:scalability/1.pdf]]

#+begin_src R :file scalability/2.pdf :results value graphics :results output :session *R* :exports both :width 6 :height 4
nbproc_time = generic_do_plot(ggplot(results, aes(x=nb_proc, y=simulation_time, color=size_verb))) +
    xlab("Number of processes") +
    ylab("Simulation time (hours)") +
    labs(colour="Matrix size")+
    ggtitle("Simulation time for different number of processes")+
    theme(legend.position = "none")+
    geom_text_repel(
        data = subset(results, nb_proc == max(nb_proc)),
        aes(label = size_verb),
        nudge_x = 45,
        segment.color = NA,
        show.legend = FALSE
      )
nbproc_time
#+end_src

#+RESULTS:
[[file:scalability/2.pdf]]

#+begin_src R :file scalability/3.pdf :results value graphics :results output :session *R* :exports both :width 6 :height 4
size_mem = generic_do_plot(ggplot(results, aes(x=size, y=memory_size, color=nb_proc_verb))) +
    xlab("Matrix size") +
    ylab("Memory consumption (gigabytes)") +
    labs(colour="Number of processes")+
    ggtitle("Memory consumption for different matrix sizes")+
    theme(legend.position = "none")+
    geom_text_repel(
        data = subset(results, size == max(size)),
        aes(label = nb_proc_verb),
        nudge_x = 45,
        segment.color = NA,
        show.legend = FALSE
      )
size_mem
#+end_src

#+RESULTS:
[[file:scalability/3.pdf]]

#+begin_src R :file scalability/4.pdf :results value graphics :results output :session *R* :exports both :width 6 :height 4
nbproc_mem = generic_do_plot(ggplot(results, aes(x=nb_proc, y=memory_size, color=size_verb))) +
    xlab("Number of processes") +
    ylab("Memory consumption (gigabytes)") +
    labs(colour="Matrix size")+
    ggtitle("Memory consumption for different number of processes")+
    theme(legend.position = "none")+
    geom_text_repel(
        data = subset(results, nb_proc == max(nb_proc)),
        aes(label = size_verb),
        nudge_x = 45,
        segment.color = NA,
        show.legend = FALSE
    )
nbproc_mem
#+end_src

#+RESULTS:
[[file:scalability/4.pdf]]

#+begin_src R :file scalability/plot_size.pdf :results value graphics :results output :session *R* :exports both :width 12 :height 4
grid_arrange_shared_legend(size_time, size_mem, nrow=1, ncol=2)
#+end_src

#+RESULTS:
[[file:scalability/plot_size.pdf]]

#+begin_src R :file scalability/plot_nbproc.pdf :results value graphics :results output :session *R* :exports both :width 12 :height 4
grid_arrange_shared_legend(nbproc_time, nbproc_mem, nrow=1, ncol=2)
#+end_src

#+RESULTS:
[[file:scalability/plot_nbproc.pdf]]

*** 2017-06-06 Tuesday
**** Discussion about the report                          :MEETING:REPORT:
***** State of the art
****** Important features
- offline vs. online, en particulier pour HPL (probes pour le pipeline des comms)
- si online: language scope, besoin de modifier le code pour que ça passe
- modèles: notion de topologies et prise en compte de la contention (super important a priori), prise en compte des
  spécificités des communications avec MPI (sémantique de synchronisation, différents ranges de performance,
  probablement pas trop grave dans le cas de HPL), collectives (dans le cas de HPL, on s’en fiche)
  - modèle classique dans ce contexte = LogP* mais prend mal en compte la contention (au niveau des noeuds mais pas du
    tout au niveau de la topologie)
  - deux approches principales: packet level et flow level
- passage à l’échelle: ça motive l’utilisation de Parallel DES et de techniques d’émulation d’applications MPI un peu
  “système”
****** Projects:
- Dimemas (Barcelona Supercomputing center), offline (extrae/paraver), “performance debugging” (sensibility analysis, what if,  +performance prediction+)
- LogoPsim (Torsten Hoefler), offline (dag, GOAL), collective algorithms @ scale
- SST macro, online/offline (DUMPI), MPI only, skeletonization/templating, more robust but more specialized (C++)
- +BigSIM (?)+, offline, PDES éventuellement, projet mort. source-to-source transformation for privatization for CHARM++/AMPI
- xSim, online, PDES aux modèles sous-jacents à la validité plus que discutable, mais scalable, privatization à coup de
  copie du segment data et pas de mmap
- CODES, offline, PDES, new kid on the block
***** Validation and capacity planning
- For the comparison with a real execution (Taurus), get the data for real experiment by executing the org-file. Long (~
  5 minutes).
- On capacity planning, it is expected that removing switches has little to no impact. Computation is in O(n^3) while
  communication is in O(n^2) (and most of the communications are asynchronous, so happen during computations).
**** Webinar                                                     :MEETING:
- [[https://github.com/alegrand/RR_webinars/blob/master/9_experimental_testbeds/index.org][Testbeds in computer science]]
- Lucas Nussbaum
*** 2017-06-07 Wednesday
**** DONE Some text is displayed in the pdf but not in the printed version :REPORT:
:LOGBOOK:
- State "DONE"       from "TODO"       [2017-06-07 Wed 09:43]
- State "TODO"       from              [2017-06-07 Wed 09:16]
:END:
- It seems that no text entered between === signs (translated to =\texttt= in Latex) appear in the printed version of the
  report. It is displayed correctly in the pdf. Fix this.
- Reprint the first page of the same file, it is fixed now. The difference is that I printed it by the network and not
  by plugging a USB stick in the printer.
**** Network printer setup                                         :TOOLS:
- Make sure the right package is installed:
  #+begin_src sh
  sudo aptitude install cups-browsed
  #+end_src
- Add these lines to the file =/etc/cups/cups-browsed.conf=:
  #+begin_example
  BrowseRemoteProtocols cups
  BrowsePoll print.imag.fr:631
  #+end_example
- Enable the service:
  #+begin_src sh
  sudo systemctl enable cups-browsed
  #+end_src
- Restart the service:
  #+begin_src sh
  sudo service cups-browsed restart
  #+end_src
*** 2017-06-08 Thursday
**** Capacity planning: components           :SMPI:EXPERIMENTS:HPL:REPORT:
- Simgrid commit: =9a8e2f5bce8c6758d4367d21a66466a497d136fe=
- HPL commit: =41774905395aebcb73650defaa7e2aa462e6e1a3=
- Script commit: =c2d1d734c80f084157ad70d702e8c669772fb2e4=
- Command (used on =nova-21=, configured as above experiments):
  #+begin_src sh
  bash run_capacity_planning.sh 100000 512

  bash run_capacity_planning.sh 50000 512
  #+end_src
- Results:
  #+begin_src R :results output :session *R* :exports both
  library(ggplot2)
  library(reshape2)
  library(gridExtra)

  get_results <- function(directory, name) {
      result <- read.csv(paste('capacity_planning/', directory, '/', name, '.csv', sep=''))
      result$name = name
      return(result)
  }
  get_all_results <- function(directory) {
      results = data.frame()
      for(type in c('bandwidth', 'latency', 'speed')) {
          for(subtype in c('high', 'low')) {
              name = paste(type, subtype, sep='_')
              tmp = get_results(directory, name)
              tmp$type = type
              if(type == 'latency'){
                  if(subtype == 'high')
                      tmp$subtype = 'bad'
                  else
                      tmp$subtype = 'good'
              }
              else {
                  if(subtype == 'high')
                      tmp$subtype = 'good'
                  else
                      tmp$subtype = 'bad'
              }
              results = rbind(results, tmp)
          }
          default = get_results(directory, 'default')
          default$type = type
          default$subtype = 'default'
          results = rbind(results, default)
      }
      return(results[c('size', 'Gflops', 'type', 'subtype')])
  }
  results_1E5 = get_all_results('exp_100000_512')
  results_5E4 = get_all_results('exp_50000_512')
  results_1E5
  results_5E4
  #+end_src

  #+RESULTS:
  #+begin_example
      size  Gflops      type subtype
  1 100000  710.40 bandwidth    good
  2 100000  702.20 bandwidth     bad
  3 100000  722.70 bandwidth default
  4 100000  349.10   latency     bad
  5 100000  823.70   latency    good
  6 100000  722.70   latency default
  7 100000 3419.00     speed    good
  8 100000   83.94     speed     bad
  9 100000  722.70     speed default
     size  Gflops      type subtype
  1 50000  458.80 bandwidth    good
  2 50000  477.00 bandwidth     bad
  3 50000  475.20 bandwidth default
  4 50000  127.30   latency     bad
  5 50000  697.60   latency    good
  6 50000  475.20   latency default
  7 50000 1346.00     speed    good
  8 50000   71.95     speed     bad
  9 50000  475.20     speed default
#+end_example

#+begin_src R :results output :session *R* :exports both
do_plot <- function(results, type) {
    tmp = results[results$type == type,]
    title = paste('HPL performance estimation for different components\nMatrix size of',
        format(unique(results$size),big.mark=",",scientific=FALSE))
    plot = ggplot(results, aes(x=type, y=Gflops, color=subtype, shape=subtype)) +
        geom_point(size=4, stroke=1) +
        scale_shape_manual(values = c(0, 1, 2))+
        theme_bw()+
        expand_limits(x=0, y=0)+
        ggtitle(title)+
        xlab('Component')+
        ylab('Performance estimation (Gflops)')+
        labs(colour='Metric')+
        labs(shape='Metric')
    return(plot)
}
#+end_src

#+RESULTS:

#+begin_src R :file capacity_planning/components_perf.pdf :results value graphics :results output :session *R* :exports both :width 12 :height 4
grid_arrange_shared_legend(
    do_plot(results_5E4, 'bandwidth') + expand_limits(x=0, y=max(results_1E5$Gflops)),
    do_plot(results_1E5, 'bandwidth') + expand_limits(x=0, y=max(results_1E5$Gflops)),
    nrow=1,
    ncol=2
)
#+end_src

#+RESULTS:
[[file:capacity_planning/components_perf.pdf]]
**** Capacity planning: topology             :SMPI:EXPERIMENTS:HPL:REPORT:
- Simgrid commit: =9a8e2f5bce8c6758d4367d21a66466a497d136fe=
- HPL commit: =41774905395aebcb73650defaa7e2aa462e6e1a3=
- Script commit: =c2d1d734c80f084157ad70d702e8c669772fb2e4=
- Four series of experiments:
  + Bandwidth of 10Gbps, sequential mapping of the processes
  + Bandwidth of 10Gbps, random mapping of the processes
  + Bandwidth of 10Mbps, sequential mapping of the processes
  + Bandwidth of 10Mbps, random mapping of the processes
- For the series with a bandwidth of 10MBps, the file =topology.py= has been locally modified to use a bandwidth 1000
  times lower:
  #+begin_example
  176 % git diff                                                        -- INSERT -- 15:42:08
  diff --git a/topology.py b/topology.py
  index 2d7d76c..1a3cd67 100644
  --- a/topology.py
  +++ b/topology.py
  @@ -158,7 +158,7 @@ class FatTree:
       prefix = 'host-'
       suffix = '.hawaii.edu'
       speed = '1Gf'
  -    bw = '10Gbps'
  +    bw = '10Mbps'
       lat = '2.4E-5s'
       loopback_bw = '5120MiBps'
       loopback_lat = '1.5E-9s'
  #+end_example
- Command (used on =nova-2=, =nova-8=, =nova-15= and =nova-16= configured as above experiments):
  + For sequential mapping:
    #+begin_src sh
    ./run_measures.py --global_csv result_capacity_50000.csv --nb_runs 3 --size 50000 --nb_proc 512 --topo
    "2;16,32;1,1:16;1,1" --experiment HPL --running_power 5004882812.500 --hugepage /root/huge 
    #+end_src
  + For random mapping:
    #+begin_src sh
    ./run_measures.py --global_csv result_capacity_50000.csv --nb_runs 3 --size 50000 --nb_proc 512 --topo
    "2;16,32;1,1:16;1,1" --experiment HPL --running_power 5004882812.500 --hugepage /root/huge --shuffle_hosts
    #+end_src
- For the random mapping with 10Mbps bandwidth, more runs have been done (8 instead of 3) to get rid of any bias.
- Results:
  #+begin_src R :results output :session *R* :exports both
  library(ggplot2)
  results_highbw_sequential <- read.csv("capacity_planning/exp_topo_50000_512/result_capacity_50000.csv")
  results_highbw_random <- read.csv("capacity_planning/exp_topo_50000_512/result_capacity_50000_shuffled.csv")
  results_lowbw_sequential <- read.csv("capacity_planning/exp_topo_50000_512/result_capacity_50000_lowbw.csv")
  results_lowbw_random <- read.csv("capacity_planning/exp_topo_50000_512/result_capacity_50000_lowbw_shuffled.csv")
  results_highbw_sequential$mapping = "Sequential"
  results_highbw_random$mapping = "Random"
  results_lowbw_sequential$mapping = "Sequential"
  results_lowbw_random$mapping = "Random"
  results_highbw = rbind(results_highbw_sequential, results_highbw_random)
  results_highbw$bandwidth = '10Gbps'
  results_lowbw = rbind(results_lowbw_sequential, results_lowbw_random)
  results_lowbw$bandwidth = '10Mbps'
  #+end_src

  #+RESULTS:

  #+begin_src R :results output :session *R* :exports both
  do_plot <- function(results) {
      title = paste('HPL performance estimation for different topologies\nBandwidth of', unique(results$bandwidth))
      plot = generic_do_plot(ggplot(results, aes(x=nb_roots, y=Gflops, color=mapping, shape=mapping)), fixed_shape=FALSE) +
          ggtitle(title)+
          xlab('Number of L2 switches')+
          ylab('Performance estimation (Gflops)')+
          scale_shape_manual(values = c(1, 2))+
          labs(colour='Mapping')+
          labs(shape='Mapping')
      return(plot)
  }
  #+end_src

  #+RESULTS:

  #+begin_src R :file capacity_planning/topology.pdf :results value graphics :results output :session *R* :exports both :width 12 :height 6
  grid_arrange_shared_legend(
      do_plot(results_lowbw),
      do_plot(results_highbw),
      nrow=1, ncol=2
  )
  #+end_src

  #+RESULTS:
  [[file:capacity_planning/topology.pdf]]
  
- The results for 10MBps are somehow expected. Removing switches deteriorates the performances, using a random mapping
  of the processes makes things even worse. Also, we can observe some performance peaks for 4, 8 and 16 root
  switches. Maybe this is due to the D mod K algorithm (TODO: check that this is indeed this algorithm). For instance,
  16 divides 512 but 15 does not. So the load of all messages should be spread more uniformly with 16 root switches than
  with 15.
- For 10GBps however, this is more strange. The number of switches has no impact, but this has already been observed on
  previous experiments (see [2017-06-02 Fri]). What is more surprising however is that the random mapping yields to
  betterperformances than the sequential mapping. Is it a bug?

  #+begin_src R :file capacity_planning/topology_sim_time.pdf :results value graphics :results output :session *R* :exports both :width 6 :height 4
  tmp = rbind(results_lowbw, results_highbw)
  tmp$bandwidth <- factor(tmp$bandwidth, levels = c('10Mbps', '10Gbps'))
  generic_do_plot(ggplot(tmp, aes(x=nb_roots, y=simulation_time, color=mapping, shape=mapping, linetype=bandwidth)), fixed_shape=FALSE)+
          ggtitle('Simulation time for different networks')+
          xlab('Number of L2 switches')+
          ylab('Simulation time (seconds)')+
          scale_shape_manual(values = c(1, 2))+
          labs(colour='Mapping')+
          labs(shape='Mapping')+
          labs(linetype='Bandwidth')
  #+end_src

  #+RESULTS:
  [[file:capacity_planning/topology_sim_time.pdf]]

  #+begin_src R :file capacity_planning/2.png :results value graphics :results output :session *R* :exports both
  results_lowbw$simgrid_time = results_lowbw$simulation_time - results_lowbw$application_time
  generic_do_plot(ggplot(results_lowbw, aes(x=nb_roots, y=simgrid_time, color=mapping)))
  #+end_src

  #+RESULTS:
  [[file:capacity_planning/2.png]]

  #+begin_src R :results output :session *R* :exports both
  library(data.table)
  aggregate_results <- function(results) {
      x = data.table(results)
      x = as.data.frame(x[, list(Gflops=mean(Gflops)), by=c("nb_roots")])
      return(x[with(x, order(nb_roots)),])
  }
  aggr_seq = aggregate_results(results_lowbw_sequential)
  aggr_rand = aggregate_results(results_lowbw_random)
  aggr_rand$gflops_ratio = aggr_seq$Gflops / aggr_rand$Gflops
  #+end_src

  #+begin_src R :file capacity_planning/3.png :results value graphics :results output :session *R* :exports both
  generic_do_plot(ggplot(aggr_rand, aes(x=nb_roots, y=gflops_ratio)))
  #+end_src

  #+RESULTS:
  [[file:capacity_planning/3.png]]

- There are *huge* differences (factor 10) in the simulation time depending on the mapping and the number of root
  switches. This time is spent in Simgrid. It certainly comes from more complex communication behaviors (congestion)
  that gives much more work to the network part of Simgrid.

*** 2017-06-09 Friday
**** TODO Work on =run_measure.py= script [0/5]                     :PYTHON:
:LOGBOOK:
- State "TODO"       from              [2017-06-09 Fri 15:47]
:END:
- [ ] Clean the code. In particular, remove the stuff related to the small matrix product test.
- [ ] Write some unit tests.
- [ ] Add options, e.g. to set the bandwidth or the latency without modifying the code.
- [ ] Add flexibility in the way the series of experiments are described. Maybe describe them with Python code in a
  separate file? Or a JSON file?
- [ ] Parallelism: allow to launch experiments on remote machines by ssh.
*** 2017-06-12 Monday
**** Add [[file:hplpaper.pdf][The LINPACK Benchmark: Past, Present and Future]]           :PAPER:
Bibtex: Dongarra03thelinpack
*** 2017-06-14 Wednesday
**** Add [[file:simgrid.pdf][Versatile, Scalable and Accurate Simulation of Distributed Applications and Platforms]] :PAPER:
Bibtex: casanova:hal-01017319
**** Add [[file:CKP93.pdf][LogP: Towards a Realistic Model of Parallel Computation]]   :PAPER:
Bibtex: Culler_1993
**** Finally found a grammar checker \o/                           :TOOLS:
- Check https://www.languagetool.org/
- It can be used in several ways, one of which is a command line tool
*** 2017-06-19 Monday
**** Discussion about the slides                                 :MEETING:
- Grosse partie sur le contexte (~10min).
  + Supercalculateurs du top500 (dont Stampede), avec leur topologie. Aussi Piz-daint (dragonfly, en Suisse). Monter la
    variabilité des topologies. Photos des supercalculateurs et schéma de la topo.
  + Routage, workload, placement des processus.
  + HPL, HPL sur Stampede, lutte contre le n^3, et aussi contre le p (p*n^2 dans la complexité).
- Pas d’état de l’art (ou seulement on-line vs off-line).
- Dessins: inkscape ou xfig
- Contribution: pas trop long (~7min).
- Validation: but final, comparer avec Stampede.
- Ouverture: capacity planning, étude de topologie.
*** 2017-06-20 Tuesday
**** Pré-soutenance Tom                                          :MEETING:
:LOGBOOK:
CLOCK: [2017-06-20 mar. 10:30]--[2017-06-20 mar. 10:51] =>  0:21
:END:

- Numéroter les slides 1/18
- Essayer d'inclure le schéma général des modifications
- distiller des informations sur le type de gain.

Slides:
1. Rank ?
   - +On peut plus accélérer les processeurs?+
   - Informations sur l'échelle, sur la topologie, la diversité
     #+BEGIN_QUOTE
     As an answer to the power and heat challenges, processor
     constructors have increased the amount of computing units (or
     cores) per processor. Modern High Performance Computing (HPC)
     systems comprise thousands of nodes, each of them holding several
     multi-core processors. For example, one of the world fastest
     computers, the IBM Sequoia system 1 Laurence Livermoor National
     Laboratory (USA), contains 96 racks of 98,304 nodes
     interconnected through a 5-dimensional torus and comprising
     16-core each, for a total of 1,572,864 cores. The Cray Titan
     system 2 at Oak Ridge National Laboratory is made of 18,688 AMD
     Opteron (16-core CPUs) and 18,688 Nvidia Tesla K20X GPUs
     interconnected through a Gemini three-dimensional torus. Another
     recent Cray machine, Piz Daint 3 at the Swiss National
     Supercomputing Centre, comprises 5,272 nodes (with 8 cores and a
     Nvidia Tesla K20X GPU each) interconnected through a custom Aries
     dragonfly topology. More recently, the Tianhe-2 4 was built with
     32,000 Intel Xeon (12 cores) and 48,000 Xeon Phi 31S1P
     interconnected through a TH-Express fat tree. Finally the Sunway
     TaihuLight 5 (Jiangsu, China), which is currently the fastest
     supercomputer in the world, is made of 40,950 nodes
     interconnected through a custom five level hierarchy of cabinets
     and comprising each 260 custom RISC cores for a total 10,649,600
     cores
     #+END_QUOTE
2. HPL:
   - où N est le rang de la matrice
   - ça marche comme un pivot de Gauss
     - recherche du maximum, petite factorization, diffusion, update
       et on recommence
     - dans le code, c'est un peu mélangé afin de bien recouvrir les
       calculs et les communications
3. Lien avec le slide 1...
   - Transitions un peu maladroites. Expliquer que c'est un domaine
     très actif.
4. +SimGrid+ Simulation of HPC applications
   - Trace.
     - Deux problèmes (taille pour obtention de la trace, application
       dynamique -faire le lien avec HPL-)
     - Émulation à la simgrid: *exclusion mutuelle*. Avantage =
       émulation sans modification mais ne passe pas à l'échelle. Il
       faut des approches hybrides.
   - Plein de projets. Majoritairement offline. SimGrid permet les deux.
5. 10:36. Pourquoi Stampede ? On s'en fiche. On est là pour donner un
   ordre de grandeur
   - 500 jours de calcul sans même compter la simulation de
     l'application elle même.
6.
   - Laboratory notebook and scripts
   - Modified HPL
   - Modifications to SimGrid
7. Intégrer le "To sum up" dans cette série de slides.
   - T_dgemm = .4354*M*N*L
   - T_dtrsm = .234234*M*N^2
   Gain = ??? ordre de grandeur. Illustration sur une config donnée
   (une petite et une grande ?).
8.
   - Négligeable mais gain important ?
   - Quantité de modifications sur HPL ?
   - À ce stade, On ne fait quasiment plus de calcul, mais la
     consommation mémoire reste importante.
9.
   - L'application accède de temps en temps à ces zones donc on ne
     peut pas simplement supprimer ces allocations...
10. Panel = information échangée entre les processus au cours de
    l'exécution.
11. 10:44
    - Cette allocation pose un "problème".
    - Modification HPL ?
12.
    - Consequences = observation à grande échelle.
13.
    -
14.
15.
16. 10:50
    Cas difficile, erreur principalement sur 1 noeud et qui diminue
    apprès, *sous-estimation systématique*
    - Expérience à petite échelle
    - Sous-estimation systématique
    - Facteur 2 sur les outliers ?

    Optimistic après dgemm.
17. Conclusion
    - Modifications légères de HPL
    - Nouvelles fonctionnalités dans SG
    - Démontré qu'on pouvait simuler à cette échelle tout en prenant
      en compte les caractéristiques fines de la topologie, du
      placement, ...
18. Rajouter capacity planning ?
**** Last remarks from Arnaud                                    :MEETING:
- Various functions → swap, max, ...
- Simulation of HPC application → parler de Simgrid
- Slide 7: écrire que c’est très optimiste (remplacer \approx par \geq )
- Slide 18: ajouter un mot sur les aspects failure et énergétiques
- Slide 16 → systematically
- Slide 1: ajouter nom, rang, nombre de noeuds et de coeurs, topologie
*** 2017-06-21 Wednesday
**** Pré-soutenance Tom V2                                       :MEETING:
:LOGBOOK:
CLOCK: [2017-06-21 mer. 10:43]--[2017-06-21 mer. 11:03] =>  0:20
:END:

Intro: simulation MPI à large échelle... et capacity planning ?
1.
2. Expliquer l'algo avant l'animation
   - mentionner l'overlapping
3. Il y a des +questions+... Il y a plusieurs leviers sur lesquels on
   peut agir pour que ça aille plus vite.
   - Il y a des "recettes". Les gens disent "je veux ça" mais c'est
     leur expérience/avis et l'argumentation est limitée
4.
   - applications adaptatives. C'est d'ailleurs le cas de HPL
   - avantage/inconvénient de l'approche émulée?
5.
6.
   - several optimizations (certaines assez logiques et d'autres qui
     étaient moins évidentes)
7.
8.
9.
   - pourquoi ne pas simplement enlever les mallocs ?
10.
11. Bon, ben maintenant, on a enlevé tous les calculs, toutes les
    allocations, il ne reste quasiment plus que le contrôle. Et
    pourtant, à grande échelle, ça ne passe toujours pas.
12.
13.
    - Vous voyez, l'effet quadratique en N et en P est toujours là et
      c'est ça qui était dur.
14.
    - Expliquer la courbe! C'est très petit.
    - Pas d'outlier. Dire plutôt pas de variabilité, et du coup pas
      d'outlier. Problème: ça à des conséquences à cause des synchros.
    - Modèle optimiste (pas de variabilité injectée, partage de bande
      passante parfait)
15.
*** 2017-06-23 Friday
**** Trying to understand the low CPU utilization for large allocations :C:EXPERIMENTS:
- According to Olivier, the low CPU utilization when doing large allocations (without huge pages) is not expected. Let’s
  investigate.
- Script commit: =80c6cd6f0853821a08da3994ce89572c9996b5ea=
- Command (the size correspond to an allocation of a matrix of size at most 600,000):
  #+begin_src sh
  ./cpu_utilization.py 8 2880000000000 /tmp/cpu_exp.csv
  #+end_src
- Analysis:
  #+begin_src R :results output :session *R* :exports both
  library(ggplot2)
  results <- read.csv('cpu_utilization/cpu_exp.csv')
  #+end_src

  #+begin_src R :file cpu_utilization/1.png :results value graphics :results output :session *R*
  ggplot(results, aes(x=size, y=cpu_utilization)) +
      geom_point() + geom_line()
  #+end_src

  #+RESULTS:
  [[file:cpu_utilization/1.png]]

- So we reproduce this behavior outside of HPL and Simgrid.
**** DONE Draw a flame graph with this small program and a large allocation.
:LOGBOOK:
- State "DONE"       from "TODO"       [2017-06-23 Fri 19:41]
- State "TODO"       from              [2017-06-23 Fri 16:13]
:END:
**** Flame graph for the CPU utilization                   :C:EXPERIMENTS:
- Script commit: =80c6cd6f0853821a08da3994ce89572c9996b5ea=
- Command (the size correspond to an allocation of a matrix of size 600,000):
  #+begin_src sh
  sudo perf record -F1000 --call-graph dwarf ./page_faults 1 2880000000000 1

  sudo perf script | ~/Documents/FlameGraph/stackcollapse-perf.pl --kernel | ~/Documents/FlameGraph/flamegraph.pl > /tmp/flame_2880000000000.svg
  #+end_src
- Kernel version:
  #+begin_src sh
  uname -r
  #+end_src

  #+RESULTS:
  : 4.4.0-81-generic

- Result:
  [[file:cpu_utilization/flame_2880000000000.svg]]
- This flame graph is very interesting, although incomplete. First, note that the function =main= accounts for less than
  40% of the samples, which is approximately equal to the CPU utilization. It means that this approach also captures
  what is done when the process is *not* executed.
- Most of the time spent in the function =main= is spent in a function =do_page_fault=.
- The remaining 60% of the whole execution time is spent on two functions, one unknown, and one called =native_irq_return_iret=.
- It is also strange to see this very large function =page_faults= located below the function =main= (and =_start=) and not on
  the side, although these functions are (a priori) not called by the function =page_faults=. Maybe a bug of =perf=?
**** TODO Next steps in the investigation of low CPU utilization [4/7]
:LOGBOOK:
- State "TODO"       from "TODO"       [2017-06-26 Mon 10:27]
- State "TODO"       from "TODO"       [2017-06-26 Mon 10:27]
- State "TODO"       from "TODO"       [2017-06-26 Mon 10:27]
- State "TODO"       from              [2017-06-23 Fri 19:52]
:END:
- [X] Plot the CPU utilization for different number of calls to =memset= (including 0).
- [ ] Draw the flame graph with more calls to =memset=.
- [ ] Draw the flame graph with no call to =memset=.
- [X] Try other flags for the =mmap=, try adding the flag =MAP_POPULATE=.
- [X] Try with another kernel version.
- [X] Try with huge pages, to see the difference.
- [ ] Speak with someone (Olivier? Samuel? Vincent? Stack Overflow?).
*** 2017-06-24 Saturday
**** Small test: several calls to =memset=                   :C:EXPERIMENTS:
- Script commit: =b8a110e9a57c821b37a3843738b97bc0affb52f6=
- No call to =memset=:
  #+begin_src sh
  /usr/bin/time ./page_faults 1 2880000000000 0
  #+end_src
  #+begin_example
  2.00202
  0.04user 1.95system 0:02.00elapsed 99%CPU (0avgtext+0avgdata 5108maxresident)k
  0inputs+4096outputs (0major+521minor)pagefaults 0swaps
  #+end_example
- One call to =memset=:
  #+begin_src sh
  /usr/bin/time ./page_faults 1 2880000000000 1
  #+end_src
  #+begin_example
  2013.29
  158.71user 604.73system 33:33.29elapsed 37%CPU (0avgtext+0avgdata 2812501956maxresident)k
  0inputs+102400outputs (0major+703125270minor)pagefaults 0swaps
  #+end_example
- Ten call to =memset=:
  #+begin_src sh
  /usr/bin/time ./page_faults 1 2880000000000 10
  #+end_src
  #+begin_example
  23344.3
  1622.97user 5224.14system 6:29:04elapsed 29%CPU (0avgtext+0avgdata 2812502520maxresident)k
  0inputs+958464outputs (0major+7031250411minor)pagefaults 0swaps
  #+end_example
- No call to =memset=, but using the flag =MAP_POPULATE=:
  #+begin_src sh
  /usr/bin/time ./page_faults 1 2880000000000 0
  #+end_src
  #+begin_example
  136.016
  0.04user 103.22system 2:16.01elapsed 75%CPU (0avgtext+0avgdata 2812501680maxresident)k
  0inputs+4096outputs (0major+43946592minor)pagefaults 0swaps
  #+end_example
- When no accesses are made and the flag =MAP_POPULATE= is not used, then the execution is very fast, there is nearly no
  page fault and the CPU utilization is high.
- With one access, we get the very low CPU utilization and the very large time.
- With ten accesses, the CPU utilization is even lower, the number of page faults is ten times higher and both user time
  and system time are also about ten times higher. This is very strange.
- With no access but with the flag =MAP_POPULATE=, there is a much larger time and number of page faults, but still about
  ten times lower than with one access and no =MAP_POPULATE=.
*** 2017-06-25 Sunday
**** More experments about the low CPU utilization for large allocations :C:R:EXPERIMENTS:
- Script commit: =b8a110e9a57c821b37a3843738b97bc0affb52f6=, modified to have between 0 and 3 calls to =memset=.
- With these results, the flag =MAP_POPULATE= is *not* used when calling =mmap=.
- Command (the size correspond to an allocation of a matrix of size at most 300,000):
  #+begin_src sh
  ./cpu_utilization.py 100 720000000000 cpu_exp2.csv
  #+end_src
- Run on =nova-17= with kernel =4.9.0-2-amd64=.
- Analysis:
  #+begin_src R :results output :session *R* :exports both
  library(gridExtra)
  library(ggplot2)
  results <- read.csv('cpu_utilization/cpu_exp2.csv')
  #+end_src

  #+begin_src R :file cpu_utilization/2.png :results value graphics :results output :session *R* :width 800
  p1 = ggplot(results, aes(x=size, y=cpu_utilization, color=factor(mem_access))) +
      geom_point() + geom_line()
  p2 = ggplot(results, aes(x=size, y=total_time, color=factor(mem_access))) +
      geom_point() + geom_line()
  grid.arrange(p1, p2, ncol=2)
  #+end_src

  #+RESULTS:
  [[file:cpu_utilization/2.png]]

  #+begin_src R :file cpu_utilization/3.png :results value graphics :results output :session *R* :width 800
  p1 = ggplot(results, aes(x=size, y=user_time, color=factor(mem_access))) +
      geom_point() + geom_line()
  p2 = ggplot(results, aes(x=size, y=system_time, color=factor(mem_access))) +
      geom_point() + geom_line()
  grid.arrange(p1, p2, ncol=2)
  #+end_src

  #+RESULTS:
  [[file:cpu_utilization/3.png]]

  #+begin_src R :file cpu_utilization/4.png :results value graphics :results output :session *R* :width 800
  p1 = ggplot(results, aes(x=size, y=memory_size, color=factor(mem_access))) +
      geom_point() + geom_line()
  p2 = ggplot(results, aes(x=size, y=nb_page_faults, color=factor(mem_access))) +
      geom_point() + geom_line()
  grid.arrange(p1, p2, ncol=2)
  #+end_src

  #+RESULTS:
  [[file:cpu_utilization/4.png]]

- Finally, the number of accesses to the buffer does not seem to impact the CPU utilization. The difference we observed
  on [2017-06-24 Sat] was probably only noise.
- The user time seems to be proportionnal to both the allocation size and the number of calls to =memset=. This is expected.
- The system time also seems to be proportionnal to them. We could expect the impact of the allocation size, but the
  impact of the number of accesses is not trivial. It seems to come from the number of page faults (which is also
  proportionnal to both the allocation size and the number of accesses). But the plot of the number of page faults is
  hard to understand. Why would more accesses cause more page faults, when the page table is already initialized?
- Another strange thing is that the memory consumption is lower with only one access than with two or three. They should
  all have the same page table size and thus the same memory consumption.
*** 2017-06-26 Monday
**** Small test: several calls to =memset= with huge pages   :C:EXPERIMENTS:
- Script commit: =005461dad4c06a2e2463d54eec228e65c07b1015=
- Compilation:
  #+begin_src sh
  gcc -DHUGEPAGE -std=gnu11 -ggdb3 -O3 -o page_faults page_faults.c -Wall
  #+end_src
- So, same experiment than [2017-06-24 Sat], except that huge pages and the =MAP_POPULATE= flag are used.
- No call to =memset=, but using the flag =MAP_POPULATE=:
  #+begin_src sh
  3.34278
  0.04user 3.29system 0:03.34elapsed 99%CPU (0avgtext+0avgdata 1476maxresident)k
  0inputs+0outputs (0major+65minor)pagefaults 0swaps
  #+end_src
  Much lower number of page faults and system time. Higher CPU utilization.
- One call to =memset=:
  #+begin_src sh
  /usr/bin/time ./page_faults 1 2880000000000 1
  #+end_src
  #+begin_example
  102.2
  98.77user 3.26system 1:42.20elapsed 99%CPU (0avgtext+0avgdata 1492maxresident)k
  0inputs+0outputs (0major+67minor)pagefaults 0swaps
  #+end_example
  In comparison with the case where no huge pages are used, the number of page faults and the time are much lower.
  Also, the system time and the number of page faults are the same than the previous test, where no =memset= were done,
  only the user time increased.
  It is strange that the number of page faults is so low. With such an allocation size, we have about 1.3M huge pages.
- Ten call to =memset=:
  #+begin_src sh
  /usr/bin/time ./page_faults 1 2880000000000 10
  #+end_src
  #+begin_example
  988.682
  984.74user 3.45system 16:28.68elapsed 99%CPU (0avgtext+0avgdata 1488maxresident)k
  0inputs+0outputs (0major+66minor)pagefaults 0swaps
  #+end_example
  Same system time and number of page faults than with only one call to =memset=, only the user time increases. This is
  the expected behavior.
- Let’s try without =MAP_POPULATE= flag.
- One call to =memset=:
  #+begin_src sh
  /usr/bin/time ./page_faults 1 2880000000000 1
  #+end_src
  #+begin_example
  102.302
  99.10user 3.18system 1:42.30elapsed 99%CPU (0avgtext+0avgdata 1520maxresident)k
  0inputs+0outputs (0major+1373356minor)pagefaults 0swaps
  #+end_example
  The number of page faults is now as expected, but this did not change the system time.
- Ten calls to =memset=:
  #+begin_src sh
  /usr/bin/time ./page_faults 1 2880000000000 10
  #+end_src
  #+begin_example
  1001.42
  997.40user 3.30system 16:41.41elapsed 99%CPU (0avgtext+0avgdata 1572maxresident)k
  0inputs+0outputs (0major+1373359minor)pagefaults 0swaps
  #+end_example
  We observe the same behavior than with the flag =MAP_POPULATE=: going from 1 call to =memset= to 10 does not impact the
  number of page faults or the system time, it only changes the user time.
***** Conclusion
Using classical pages or huge pages does not only change the page size (and thus the page table size). It actually
changes the *behavior* of the OS. With classical pages, the system time and the number of page faults are proportionnal to
both the allocation size and the number of accesses, whereas with huge pages they are only proportionnal to the
allocation size.
**** Flame graph for the CPU utilization                   :C:EXPERIMENTS:
- Script commit: =005461dad4c06a2e2463d54eec228e65c07b1015= (the file has been modified to remove the flag =MAP_POPULATE=).
- Command (the size correspond to an allocation of a matrix of size 600,000):
  #+begin_src sh
  sudo perf record -F1000 --call-graph dwarf ./page_faults 1 2880000000000 1

  sudo perf script | ~/Documents/FlameGraph/stackcollapse-perf.pl --kernel | ~/Documents/FlameGraph/flamegraph.pl > /tmp/flame_2880000000000_hugepage.svg
  #+end_src
- Kernel version:
  #+begin_src sh
  uname -r
  #+end_src

  #+RESULTS:
  : 4.4.0-81-generic

- Result:
  [[file:cpu_utilization/flame_2880000000000_hugepage.svg]]
- This flame graph is hard to relate with the previous results.
- We saw that there was a high CPU utilization (99%) and that most of the time was spent in user mode. But the graph
  shows that a very large part of the time is spent in some other function, outside of the program scope. My guess would
  be that such function should not be accounted in the program execution time and that we should therefore have a very
  low CPU utilization.
**** Segmented regression                                              :R:
- [[https://en.wikipedia.org/wiki/Segmented_regression][Wikipedia page]]
- [[https://stats.stackexchange.com/questions/20890/how-to-use-segmented-package-to-fit-a-piecewise-linear-regression-with-one-break][Example on StackExchange]]
- Let’s try with dummy data.
  #+begin_src R :results output :session *R* :exports both
  NB = 100
  A1 = 2 # coeff for first part
  A2 = 1 # coeff for second part
  B1 = 0 # intercept for first part
  B2 = 100 # intercept for second part
  df = data.frame(n=1:NB)
  df$n = sample(500, size=NB, replace=TRUE)
  df$noise = sample(20, size=NB, replace=TRUE)-10
  my_func <- function(n, noise) {
      if(n < 100) {
          return(A1*n+B1 + noise)
      }
      else {
          return(A2*n+B2 + noise)
      }
  }
  df$fn = mapply(my_func, df$n, df$noise)
  #+end_src

  #+RESULTS:

  #+begin_src R :file segmented_regression/1.png :results value graphics :results output :session *R*
  library(ggplot2)
  ggplot(df, aes(x=n, y=fn)) + geom_point()
  #+end_src

  #+RESULTS:
  [[file:segmented_regression/1.png]]

- The two modes are clearly visible, let’s try some regressions.

  #+begin_src R :results output :session *R* :exports both
  library(segmented)
  lm = segmented(lm(fn~n, data=df), seg.Z = ~ n)
  summary(lm)
  #+end_src

  #+RESULTS:
  #+begin_example

          ***Regression Model with Segmented Relationship(s)***

  Call: 
  segmented.lm(obj = lm(fn ~ n, data = df), seg.Z = ~n)

  Estimated Break-Point(s):
     Est. St.Err 
  99.197  3.361 

  Meaningful coefficients of the linear terms:
              Estimate Std. Error t value Pr(>|t|)    
  (Intercept)  1.22041    4.02077   0.304    0.762    
  n            1.99373    0.06389  31.208   <2e-16 ***
  U1.n        -0.98928    0.06420 -15.409       NA    
  ---
  Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

  Residual standard error: 6.183 on 96 degrees of freedom
  Multiple R-Squared: 0.9985,  Adjusted R-squared: 0.9985 

  Convergence attained in 6 iterations with relative change 2.230614e-15
  #+end_example

  #+begin_src R :file segmented_regression/2.png :results value graphics :results output :session *R*
  plot(lm)
  #+end_src

  #+RESULTS:
  [[file:segmented_regression/2.png]]

- Need to check, but it seems that:
  + It expects the underlying “function” to be “continuous”, which is not the case of what we have with =dgemm= on
    Stampede. If there is a discontinuity at the break point, the estimation fails.
  + The intercept value is =B1=.
  + The =n= coefficient is =A1=.
  + The =U1.n= coefficient is =A2-A1=.
*** 2017-06-27 Tuesday
**** Keep trying the segmented regression                              :R:
- Using code from [[https://stackoverflow.com/questions/8758646/piecewise-regression-with-r-plotting-the-segments][stackoverflow]]
- Asked a question on [[https://stackoverflow.com/questions/44778954/segmented-linear-regression-with-discontinuous-data][stackoverflow]].
- Let’s try with dummy data.
  #+begin_src R :results output :session *R* :exports both
  NB = 100
  A1 = 2 # coeff for first part
  A2 = 1 # coeff for second part
  B1 = 0 # intercept for first part
  B2 = 300 # intercept for second part
  df = data.frame(n=1:NB)
  df$n = sample(500, size=NB, replace=TRUE)
  df$noise = sample(20, size=NB, replace=TRUE)-10
  my_func <- function(n, noise) {
      if(n < 100) {
          return(A1*n+B1 + noise)
      }
      else {
          return(A2*n+B2 + noise)
      }
  }
  df$fn = mapply(my_func, df$n, df$noise)
  #+end_src

  #+RESULTS:

  #+begin_src R :file segmented_regression/3.png :results value graphics :results output :session *R*
  library(ggplot2)
  ggplot(df, aes(x=n, y=fn)) + geom_point()
  #+end_src

  #+RESULTS:
  [[file:segmented_regression/3.png]]

- First, using =segmented= package.

  #+begin_src R :results output :session *R* :exports both
  library(segmented)
  model_segmented = segmented(lm(fn~n, data=df), seg.Z = ~ n)
  summary(model_segmented)
  #+end_src

  #+RESULTS:
  #+begin_example

          ***Regression Model with Segmented Relationship(s)***

  Call: 
  segmented.lm(obj = lm(fn ~ n, data = df), seg.Z = ~n)

  Estimated Break-Point(s):
      Est.  St.Err 
  136.566   5.677 

  Meaningful coefficients of the linear terms:
              Estimate Std. Error t value Pr(>|t|)    
  (Intercept) -61.0463    11.7827  -5.181 1.22e-06 ***
  n             3.6374     0.1534  23.706  < 2e-16 ***
  U1.n         -2.6332     0.1593 -16.525       NA    
  ---
  Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

  Residual standard error: 33.92 on 96 degrees of freedom
  Multiple R-Squared: 0.9804,  Adjusted R-squared: 0.9798 

  Convergence attained in 4 iterations with relative change -7.90412e-16
  #+end_example

  #+begin_src R :file segmented_regression/4.png :results value graphics :results output :session *R*
  predict_segmented = data.frame(n = df$n, fn = broken.line(model_segmented)$fit)
  ggplot(df, aes(x = n, y = fn)) +
  geom_point() + geom_line(data = predict_segmented, color = 'blue')
  #+end_src

  #+RESULTS:
  [[file:segmented_regression/4.png]]

- Then, doing the segmentation by hand.

  #+begin_src R :file segmented_regression/5.png :results value graphics :results output :session *R*
  Break<-sort(unique(df$n))
  Break<-Break[2:(length(Break)-1)]
  d<-numeric(length(Break))
  for (i in 1:length(Break)) {
      model_manual<-lm(fn~(n<Break[i])*n + (n>=Break[i])*n, data=df)
      d[i]<-summary(model_manual)[[6]]
  }
  plot(d)
  #+end_src

  #+RESULTS:
  [[file:segmented_regression/5.png]]

  #+begin_src R :results output :session *R* :exports both
  # Smallest breakpoint
  breakpoint = Break[which.min(d)]
  breakpoint
  df$group = df$n >= breakpoint
  model_manual<-lm(fn~n*group, data=df)
  summary(model_manual)
  #+end_src

  #+RESULTS:
  #+begin_example
  [1] 100

  Call:
  lm(formula = fn ~ n * group, data = df)

  Residuals:
      Min      1Q  Median      3Q     Max 
  -9.6223 -5.0330 -0.5436  4.7791 10.4031 

  Coefficients:
               Estimate Std. Error t value Pr(>|t|)    
  (Intercept)   1.02021    2.39788   0.425    0.671    
  n             1.98517    0.04128  48.090   <2e-16 ***
  groupTRUE   300.21629    3.07455  97.646   <2e-16 ***
  n:groupTRUE  -0.98826    0.04174 -23.678   <2e-16 ***
  ---
  Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

  Residual standard error: 5.984 on 96 degrees of freedom
  Multiple R-squared:  0.9994,	Adjusted R-squared:  0.9994 
  F-statistic: 5.248e+04 on 3 and 96 DF,  p-value: < 2.2e-16
  #+end_example

  #+begin_src R :file segmented_regression/6.png :results value graphics :results output :session *R*
  dat_pred = data.frame(n = df$n, fn = predict(model_manual, df))
  ggplot(df, aes(x = n, y = fn)) +
      geom_point() +
      geom_line(data=dat_pred[dat_pred$n < breakpoint,], color = 'blue')+
      geom_line(data=dat_pred[dat_pred$n >= breakpoint,], color = 'blue')
  #+end_src

  #+RESULTS:
  [[file:segmented_regression/6.png]]

- The =segmented= package fails when the data is discontinuous.
- The dirty method works great.
