HYBRED MONTECARLO EVOLUTION (CHROMA)

The step here refer to the compilation and execution of the Hybrid-Montecarlo simulations program. It targets the Pure Gauge SU(3) Theory using the publicly available USQCD collaboration and it is based on the "chroma" library (http://usqcd.jlab.org/usqcd-docs/chroma/).

The CHROMA application can be downloaded from the following web pages:

wget http://usqcd.jlab.org/usqcd-software/chroma/chroma-3.38.0.tar.gz

It has three main dependence. Two are from the same collaboration and provide the encapsulation of the MPI combination and the definition of the basic C++ class used:

wget http://usqcd.jlab.org/usqcd-software/qmp/qmp-2.1.6.tar.gz
wget http://usqcd.jlab.org/usqcd-software/qdp++/qdp++-1.36.1.tar.gz

The last dependence is on the libxml2 library (http://xmlsoft.org/) that is used to parse the simulation parameter file. Usually this library is already installed on a machine but in our case (on turing) it is not. That will give as an example of the thinks that one needs to do when a required library is not installed.

The CHROMA application need has prerequisite to have installed the

• The qmp wrapper library for MPI comunications.
• The qdp++ C++ Library

And the compilation can be achieved with the following commands

#!/bin/bash
export COPT=`mpicc -showme:compile`
export LINKOPT=`mpicc -showme:link`
export MYDIR=`pwd`
export PATH=$PATH:$MYDIR/local/bin/
if [ "a${Nd}" = "a" ] ; then export Nd=4 ;fi
if [ "a${Nc}" = "a" ] ; then export Nc=3 ;fi
if [ "a${NT}" = "a" ] ; then export NT=1 ;fi
##--------------------------------------

First we have to check if there is a working installation of the xml2 library. In case is not installed we have to compile a local copy

##--------------------------------------
if [ "a$(which xml2-config)" = "a" ] ; then
  echo "** XML2 not found!"
  echo "** COMPILING libxml2-2.7.2"
  tar xzvf $MYDIR/libxml2-2.7.2.tar.gz
  cd $MYDIR/libxml2-2.7.2
  ./configure --prefix=${MYDIR}/local --disable-shared
  make -j ${NT}
  make -j ${NT} install
  cd $MYDIR/
else
  echo "**Found xml2-config in "
  echo "**[$(which xml2-config)]"
fi
##--------------------------------------

The next step is to install the qmp wrapper library for MPI communications

##--------------------------------------
if [ "a$(which qmp-config)" = "a" ] ; then
  echo "** NOT Found qmp-config"
  echo "** Compiling qmp-2.1.6"
  tar xzvf $MYDIR/qmp-2.1.6.tar.gz
  cd $MYDIR/qmp-2.1.6
  ./configure --with-qmp-comms-type=MPI  \
     --with-qmp-comms-ldflags="$LINKOPT" \
     --with-qmp-comms-cflags="$COPT" \
     --prefix=${MYDIR}/local
  make -j ${NT}
  make -j ${NT} install
  cd ${MYDIR}
else
  echo "** Found qmp-config in "
  echo "** [$(which qmp-config)]"
fi
##--------------------------------------

Here the main part of the compilation procedure

##--------------------------------------
export DirCHROMA=${MYDIR}
echo "======================================================="
echo "About to compile chroma for "
echo "Nd = ${Nd}"
echo "Nc = ${Nc}"
echo "DIR =${DIR}"
echo "MYDIR=  " $MYDIR
echo "COPT=   " $COPT
echo "LINKOPT=" $LINKOPT
echo "DirCHROMA = ${DirCHROMA}" 
echo "======================================================="
tar xzf ${MYDIR}/qdp++-1.36.1.tar.gz
tar xzf ${MYDIR}/chroma-3.38.0.tar.gz

Then we have the first real component of the QCD package

echo "======================================================="
echo " Compiling QDP for --enable-Nd=${Nd}  --enable-Nc=${Nc} "
echo "======================================================="
cd ${MYDIR}/qdp++-1.36.1
./configure --prefix=${MYDIR}/local    \
            --with-qmp=${MYDIR}/local/   \
            --enable-parallel-arch=parscalar     \
            --with-qmp-comms-type=MPI            \
            --enable-sse2  --enable-precision=double \
            --enable-Nd=${Nd}  --enable-Nc=${Nc} \
                      &> ${MYDIR}/qdp_configure.out
make -j ${NT}         &> ${MYDIR}/qdp_make.out
make -j ${NT} install &> ${MYDIR}/qdp_make_install.out

An finally we produce the executable to run the real simulations

echo "======================================================="
echo " Compiling CHROMA --with-qdp=${MYDIR}"
echo "======================================================="
cd ${MYDIR}/chroma
./configure  --with-qdp=${MYDIR}/local    \
             --prefix=${MYDIR}/local \
                      &> ${MYDIR}/chroma__configure.out   
make -j ${NT}         &> ${MYDIR}/chroma_make.out	     
make -j ${NT} install &> ${MYDIR}/chroma_make_install.out
cd ${MYDIR}

The execution of a CHROMA simulation is done using the sintax

purgaug -i IN.xml -o OUT.xml

where IN.xml contains the parameter of the simulation and OUT.xml the corresponding results. It also allow to have checkpoint/restart but we are not interested to this possibilities for this notes.

An example of an xml file that control the execution of our simulation is 8_8_8_8_B6.0.xml:

<?xml version="1.0"?>
<purgaug>
  <Cfg>
    <cfg_type>WEAK_FIELD</cfg_type>
    <cfg_file>dummy</cfg_file>
  </Cfg>
  <MCControl>
    <RNG>
      <Seed>testautocorr.sh
        <elem>17</elem>
        <elem>13 </elem>
        <elem>240 </elem>
        <elem>30 </elem>
      </Seed>
    </RNG>
    <StartUpdateNum>0</StartUpdateNum>
    <NWarmUpUpdates>200</NWarmUpUpdates>
    <NProductionUpdates>2000</NProductionUpdates>
    <NUpdatesThisRun>2000</NUpdatesThisRun>
    <SaveInterval>200</SaveInterval>
    <SavePrefix>8_8_8_8_B6.00.data</SavePrefix>
    <SaveVolfmt>SINGLEFILE</SaveVolfmt>
  </MCControl>
  <InlineMeasurements>
  </InlineMeasurements>
  <HBItr>
    <GaugeAction>
       <Name>WILSON_GAUGEACT</Name>
       <beta>6.00</beta>
       <AnisoParam>
            <anisoP>false</anisoP>
            <t_dir>3</t_dir>
            <xi_0>1</xi_0>
            <nu>1</nu>
       </AnisoParam>
       <GaugeState>
         <Name>SIMPLE_GAUGE_STATE</Name>
         <GaugeBC>
           <Name>PERIODIC_GAUGEBC</Name>
         </GaugeBC>
       </GaugeState>
    </GaugeAction>
    <HBParams>
      <nOver>3</nOver>
      <NmaxHB>1</NmaxHB>
    </HBParams>
    <nrow>8 8 8 8</nrow>
  </HBItr>
</purgaug>

We have create a small bash script command to simplify our copy operation to the grid and we called it "CopyToGrid.sh"

#!/bin/bash                                                                                                     
lcg-cr -v --vo theophys -d srm://gridsrm.pi.infn.it/theophys/IS_OG51/Parma/CorsoGrid/$1  \
       -l lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/$1 file://$(pwd)/$1

And then we evaluate the following commands:

../CopyToGrid.sh qdp++-1.36.1.tar.gz
../CopyToGrid.sh libxml2-2.7.2.tar.gz
../CopyToGrid.sh qmp-2.1.6.tar.gz
../CopyToGrid.sh chroma-3.38.0.tar.gz

Now that we have all the data we can create a JOB script (CompileChroma.jdl) and a JOB descriptor (CompileChroma.jdl) that will do the compilation for us. The actual form of this two files is:

# CompileChroma.jdl
JobType = "Normal";
CPUNumber = 8;
Executable    = "CompileChroma.sh";
Arguments     = "";
StdOutput     = "std.out";
StdError      = "std.err";
PerusalFileEnable = true;
PerusalTimeInterval = 10;
InputSandbox  = {"CompileChroma.sh"}; 
OutputSandbox = {"std.out", "std.err"};
MyProxyServer = "myproxy.cnaf.infn.it";
Requirements =(other.GlueCEInfoHostName == "gridce3.pi.infn.it");
CeRequirements = "hostsmpsize==8 &&  wholenodes==\"true\"  && hostnumber==1";

#!/bin/bash
## ----------------------------------------
## First we get the tra file we previously
## saved on the GRID storage 
## ----------------------------------------
export LFN=lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid
lcg-cp -v --vo theophys ${LFN}/qdp++-1.36.1.tar.gz  file://$(pwd)/qdp++-1.36.1.tar.gz 
lcg-cp -v --vo theophys ${LFN}/libxml2-2.7.2.tar.gz file://$(pwd)/libxml2-2.7.2.tar.gz
lcg-cp -v --vo theophys ${LFN}/qmp-2.1.6.tar.gz     file://$(pwd)/qmp-2.1.6.tar.gz    
lcg-cp -v --vo theophys ${LFN}/chroma-3.38.0.tar.gz file://$(pwd)/chroma-3.38.0.tar.gz
## ----------------------------------------
## We can now start the procedure we tested
## on our local machine
## ----------------------------------------
export COPT=`mpicc -showme:compile`
export LINKOPT=`mpicc -showme:link`
export MYDIR=`pwd`
export PATH=$PATH:$MYDIR/local/bin/
export Nd=$1
export Nc=$2 
export NT=$3
if [ "a${Nd}" = "a" ] ; then export Nd=4 ;fi
if [ "a${Nc}" = "a" ] ; then export Nc=3 ;fi
if [ "a${NT}" = "a" ] ; then export NT=1 ;fi
##--------------------------------------
##--------------------------------------
if [ "a$(which xml2-config)" = "a" ] ; then
  echo "** XML2 not found!"
  echo "** COMPILING libxml2-2.7.2"
  tar xzvf $MYDIR/libxml2-2.7.2.tar.gz
  cd $MYDIR/libxml2-2.7.2
  ./configure --prefix=${MYDIR}/local --disable-shared
  make -j ${NT}
  make -j ${NT} install
  cd $MYDIR/
else
  echo "**Found xml2-config in "
  echo "**[$(which xml2-config)]"
fi 
##--------------------------------------
##--------------------------------------
if [ "a$(which qmp-config)" = "a" ] ; then
  echo "** NOT Found qmp-config"
  echo "** Compiling qmp-2.1.6"
  tar xzvf $MYDIR/qmp-2.1.6.tar.gz
  cd $MYDIR/qmp-2.1.6
  ./configure --with-qmp-comms-type=MPI  \
     --with-qmp-comms-ldflags="$LINKOPT" \
     --with-qmp-comms-cflags="$COPT" \
     --prefix=${MYDIR}/local
  make -j ${NT}
  make -j ${NT} install
  cd ${MYDIR}
else
  echo "** Found qmp-config in "
  echo "** [$(which qmp-config)]"
fi
##--------------------------------------
##--------------------------------------
export DirCHROMA=${MYDIR}
echo "======================================================="
echo "About to compile chroma for " 
echo "Nd = ${Nd}"
echo "Nc = ${Nc}"
echo "DIR =${DIR}"
echo "MYDIR=  " $MYDIR
echo "COPT=   " $COPT
echo "LINKOPT=" $LINKOPT
echo "DirCHROMA = ${DirCHROMA}" 
echo "======================================================="
tar xzf ${MYDIR}/qdp++-1.36.1.tar.gz
tar xzf ${MYDIR}/chroma-3.38.0.tar.gz
echo "======================================================="
echo " Compiling QDP for --enable-Nd=${Nd}  --enable-Nc=${Nc} "
echo "======================================================="
cd ${MYDIR}/qdp++-1.36.1
./configure --prefix=${MYDIR}/local    \
          --with-qmp=${MYDIR}/local/   \
          --enable-parallel-arch=parscalar     \
          --with-qmp-comms-type=MPI            \
          --enable-sse2  --enable-precision=double \
          --enable-Nd=${Nd}  --enable-Nc=${Nc} \
                    &> ${MYDIR}/qdp_configure.out
make -j ${NT}         &> ${MYDIR}/qdp_make.out
make -j ${NT} install &> ${MYDIR}/qdp_make_install.out
echo "======================================================="
echo " Compiling CHROMA --with-qdp=${MYDIR}"
echo "======================================================="
cd ${MYDIR}/chroma
./configure  --with-qdp=${MYDIR}/local    \
           --prefix=${MYDIR}/local \
                    &> ${MYDIR}/chroma__configure.out   
make -j ${NT}         &> ${MYDIR}/chroma_make.out	     
make -j ${NT} install &> ${MYDIR}/chroma_make_install.out
cd ${MYDIR}
## ----------------------------------------
## We now just have to copy back the executable
## and save it on the GRID storage 
## ----------------------------------------
lcg-cr -v --vo theophys -d srm://gridsrm.pi.infn.it/theophys/IS_OG51/Parma/CorsoGrid/purgaug  \
   -l lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/purgaug file://$(pwd)/local/bin/purgaug
## -----------------------------------------------
## We now show some basic fact on where the
## compilation wes run
## -----------------------------------------------
echo "**************************************"
echo "*** command:  df"
df
echo "*** command:  pwd"
pwd
echo "*** command:  ls"
ls

We can now execute the compilation submitting the job

 glite-wms-job-submit -a -o 1.job CompileChroma.jdl

The "RUN_RANKED_chroma.sh" that allow the execution of the CHROMA executable may look like as follows:

## ------------------------------------
##  CHECK local options
## ------------------------------------
export XMLparFILE=$1
export RUNid=$2
export EXE=purgaug
export MYDIR=`pwd`
echo " ========================== ================================="
echo "XMLparFILE=" $XMLparFILE
echo "MYDIR=" $MYDIR
echo "PATH= " $PATH
echo " ========================== ================================="
## Here we produce various file needed by mpirun
## In future all will be achived using the
## "mpi-start" hawk
NODEFILE1=/tmp/hf1.$(date +"%m%d%y%H%M%S")
NODEFILE2=/tmp/hf2.$(date +"%m%d%y%H%M%S")
RANKFILE=/tmp/hf3.$(date +"%m%d%y%H%M%S")
AWKcommand=/tmp/hf4.$(date +"%m%d%y%H%M%S")
touch $NODEFILE1
touch $NODEFILE2
touch $AWKcommand
echo "{print \"rank \" i++ \"=\" \$1 \" slot=0\"}"   >> $AWKcommand
echo "{print \"rank \" i++ \"=\" \$1 \" slot=1\"}"   >> $AWKcommand
echo "{print \"rank \" i++ \"=\" \$1 \" slot=2\"}"   >> $AWKcommand
echo "{print \"rank \" i++ \"=\" \$1 \" slot=3\"}"   >> $AWKcommand
echo "{print \"rank \" i++ \"=\" \$1 \" slot=4\"}"   >> $AWKcommand
echo "{print \"rank \" i++ \"=\" \$1 \" slot=5\"}"   >> $AWKcommand
echo "{print \"rank \" i++ \"=\" \$1 \" slot=6\"}"   >> $AWKcommand
echo "{print \"rank \" i++ \"=\" \$1 \" slot=7\"}"   >> $AWKcommand
for host in $LSB_HOSTS; do echo $host >> $NODEFILE1; done;
sort -u  ${NODEFILE1} > ${NODEFILE2}
awk -f $AWKcommand  ${NODEFILE2} > ${RANKFILE}
NP=$[`cat ${RANKFILE} | wc --lines`]
echo "."
cmdmpirun="mpirun -np $NP -hostfile $NODEFILE2 --rankfile ${RANKFILE}"
echo "========= MPI PARAMETER ================"
echo "NP = ${NP}"
echo "mpirun = $(which mpirun)"
echo "      => ${cmdmpirun}"
echo "========================================" 
## ------------------------------------
# copy the input file from the SE to the WN
## ------------------------------------
cmd1="lcg-cp -v --vo theophys lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/${XMLparFILE}.xml file://$(pwd)/${XMLparFILE}.xml"
cmd2="lcg-cp -v --vo theophys lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/${EXE}            file://$(pwd)/${EXE}"
echo "=====================  1(a) ===================="
echo $cmd1
echo $cmd2
echo "=====================  1(b) ===================="
eval $cmd1
eval $cmd2
ls
echo "=====================  1(c) ===================="
## ------------------------------------
#  Now we the the exectuable and the 
#  Parameter File
## ------------------------------------
echo "==============================="
echo "RANKFILE:"
cat ${RANKFILE}
echo "==============================="
echo "==============================="
echo "NODEFILE2:"
cat ${NODEFILE2}
echo "==============================="
echo "----------------------------- Command ----------------"
cmd="(${cmdmpirun} ${MYDIR}/$EXE -i ${XMLparFILE}.xml -o ${XMLparFILE}_OUT.xml 2> stderror )"
cmd2="time (${cmd}) 2> ${XMLparFILE}.timing" 
echo $cmd
echo $cmd2
echo "------------------------------------------------------"
ls -l ${MYDIR}/$EXE
chmod +x  ${MYDIR}/$EXE
ls -l ${MYDIR}/$EXE
echo "**************************************************"
echo "**Started at $(date)"
time (${cmdmpirun} ${MYDIR}/$EXE -i ${XMLparFILE}.xml -o ${XMLparFILE}_OUT.xml 2> stderror)  2> ${XMLparFILE}.timing
echo "**Ended at   $(date)"
rm -f $NODEFILE1
rm -f $NODEFILE2
rm -f $RANKFILE
echo "************************************"
ls -l 
tar czf localRESULTS.tgz *
echo "************************************"
## ------------------------------------
# Save the result file from the WN  to the SE
## ------------------------------------
FnameToSave="${XMLparFILE}_${RUNid}.tgz"
lcg-cr -v --vo theophys -d srm://gridsrm.pi.infn.it/theophys/IS_OG51/Parma/CorsoGrid/${FnameToSave}  \
       -l lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/${FnameToSave} \
        file://$(pwd)/localRESULTS.tgz
echo "====================="

The corresponding jdl file is:

# RUN_RANKED_chroma.jdl
JobType = "Normal";
CPUNumber = 64;
Executable    = "RUN_RANKED_chroma.sh";
Arguments     = "16_16_16_16_B6.0 id8x4";
StdOutput     = "std.out";
StdError      = "std.err";
PerusalFileEnable = true;
PerusalTimeInterval = 10;
InputSandbox  = {"RUN_RANKED_chroma.sh"}; 
OutputSandbox = {"std.out", "std.err"};
MyProxyServer = "myproxy.cnaf.infn.it";
Requirements =(other.GlueCEInfoHostName == "gridce3.pi.infn.it");
CeRequirements = "hostsmpsize==8 &&  wholenodes==\"true\"  && hostnumber==4";

One should note a whole section that create auxiliary file for the mpirun (openmp flavour) command to create the actual list of execution nodes and, very important, to manage MEMORY-CORE affinity.

In future all of this will be dealt with using the mpi-start command. (…commet in details…)

One should note that doubling the number of processor the execution time is better than linear scaling … 2000 sweep on a 16^4 hypercube.

real	164m 2.279s   16 cores
real	 67m40.271s   32 cores