Here I report the note on the attivity (and in this directory tree) all the result I will optain on characterizing the performace of the Einstein Toolkit on varoiu machine I do have access. This log book will cover all my activity starting from December the 7th 2012. This section contains various parts that will help to understand how well Catus behave on different platform. Main purpose of this testing is to find out how to run simulations on "Fermi".

The main directory where I store the result of the Cactus Speed Test is "/work/staff/roberto.depietri/OrstedSpeedTest"

I decided to consider the November 2012 verdion announced as follow: We are pleased to announce the sixth release (code name "Ørsted") of the Einstein Toolkit, an open, community developed software infrastructure for relativistic astrophysics.

The main problems on previous test I did where strang scaling properties of Carpet going to 256 or more processor and a lack of a proper log of the activity I did. Thanks to Frank Loeffler I realized that the main scaling problem I observed were due to CARPET IOASCII for 1d output. I pointed to me that all the processor write in an order sequence to the 1d files and indeed the writing time scales linearly with the number of MPI processes involved. Lesson lerned: do no output in testing speed and scalig. Do separate IO testing and do not mix up the to type of speed testing.

The good lesson I learned in previous test is the need to have standarzide configuration to compare and use as reference. Alway do strong a week scaling check.

First check UNIGRID: border at 60,60,60 doing 32 integration steps

  PUGH:   PUGHit32.rpar   generate par files like PUGHdx1.000it32.par      
  CARPET: CARPETit32.rpar generate par files like CARPETdx1.000it32.par
  #################################################################################
  ### dx=[1.5 ....... 0.15]; nx=(60./dx *2 +1 +4);vol=nx.^3;[dx ;nx;vol/vol(1.5PUGH)]
  ##################################################################################
  ##  2.00  1.50  1.00  0.75  0.625  0.60  0.50  0.40  0.30  0.25  0.20  0.15  0.125
  ##    65    85   125   165    197   205   245   305   405   485   605   805    965
  ##  0.44  1.00  3.18  7.31  12.45  14.1  23.9  46.2   108   185   361   849   1463
  ##
  ##  dx=1.0 Carpet requires 4312.518 MB
  ##  dx=1.5 Carpet requires 1356.006 MB
  ##  dx=2.0 Carpet requires  606.390 MB
  #################################################################################

Then check 3 refinement levels. Borders at 120 and subgrid at 60 and 30. Also in this case we will do 32 integration steps on the finest grid. Resolution dx will refer to the finer grid

  CARPET: CARPET_RL3_it32.rpar generate par files like CARPET_RL3_dx1.000it32.par
  #################################################################################
  ### dx=[1.5 ....... 0.15]; nx=(120./(4*dx) *2 +1 +4);vol=3*nx.^3;[dx ;nx;vol/vol(1.5PUGH)]
  ##################################################################################
  ##  2.00  1.50  1.00  0.75  0.625  0.60  0.50  0.40  0.30  0.25  0.20  0.15  0.125
  ##   35     45    65    85    101   105   125   205   245   305   405   485    605
  ##  0.21  0.45  1.34     3   5.03  5.66  9.54  42.1  71.8   139   324   557   1081
  ##
  ##  dx=0.75 Carpet requires 6468.078 MB
  ##  dx=1.0 Carpet requires 3318.366 MB
  ##  dx=1.5 Carpet requires 1471.679 MB
  ##  dx=2.0 Carpet requires 1068.414 MB
  ##     Total time for simulation  (np1..t1)= 701 sec (11 minuti)
  ## Su Blue Gene Q se perfect scaling will require (1024 cores)
  ##  2.00  1.50  1.00  0.75  0.625  0.60  0.50  0.40  0.30  0.25  0.20  0.15  0.125
  ##  0.01  0.03  0.09  0.23  0.37         0.73  1.42  3.38  5.84  11.4  27.1  46.7
  #################################################################################

First test had shown that the use of

ActiveThorns = "TimerReport"
TimerReport::output_all_timers_readable ="yes"
TimerReport::out_every=32
TimerReport::out_filename = "TimerReport"
TimerReport::output_schedule_timers = "no"
TimerReport::output_all_timers = "no"

deeply effect tests results. For example "CARPET_RL3_dx0.400it32.par" have the following result for timer "Total time for simulation" .

     Blue Gene Size    Np  Nt  Total time for simulation
With TimerReport  64   64  16  249 s
                  64  128   8  251 s
                  64  256   4  272 s
                  64  512   2  316 s
                  64 1024   1  414 s
                 128 2048   1  564 s
Without           64 1024   1  273 s
                 128 2048   1  278 s
                 256 4096   1  362 s

All the speed tests will be performed without the activation of "TimerReport".

The second stage of testing involved just the output of various reduction of "rho". No other outputs.

(*) This run was also performed doing as much as four time the number of time integration of it=128 and the corresponding CCTK_EVOL changed from 510 to 2100 and simulation from 696 to 2524.