Strumenti Utente

Strumenti Sito


grid:mpi

OLD STUFF

Aggiornato al 25/03/08 (r.a.)

Nella distribuzione SLC4/SL4 (WN) c'e':

mpich-1.2.7p1-2.slc4   (/opt/mpich-1.2.7p1/bin/mpicc  /opt/mpich-1.2.7p1/bin/mpirun )
mpiexec-0.82-1.slc4    (/opt/mpiexec-0.82/bin/mpiexec )
i2g-mpi-start-0.0.52-1 (/opt/i2g/bin/mpi-start)
glite-yaim-mpi-0.1.6-3 (/opt/glite/yaim/examples/siteinfo/services/glite-mpi/..)

openmpi (1.2.3):
openmpi-1.2.3-1         ( /usr/bin/orterun   mpirun -> orterun )
openmpi-libs-1.2.3-1
libibverbs-1.1.1-7
openib-1.2-7 
  libmthca is a device-specific driver for Mellanox InfiniBand host channel adapters (HCAs) 
  for the libibverbs library. This allows userspace processes to access Mellanox HCA hardware
  directly with low latency and low overhead.
dapl-1.2.1.1-7 
lam-libs-7.1.2-8 
librdmacm-1.0.1-7 

Alternativa:
rpm -ivh  http://grid-serv.pr.infn.it/install/post/openmpi-1.2.5-1.i386-opt.rpm
/opt/openmpi/1.2.5/bin/mpicc
/opt/openmpi/1.2.5/bin/orterun ( mpirun -> orterun )
/opt/openmpi/1.2.5/bin/mpirun
/opt/openmpi/1.2.5/bin/mpiexec

Nella distribuzione SL3 (CE) non c'e' nulla.

MPI

OpenMPI

glite_MPI_utils

Aggiornato al 26/03/08 (r.a.)

Incluso in update 14 SL3.1: http://glite.web.cern.ch/glite/packages/R3.1/updates.asp

Implementa la config specificata in: http://www.grid.ie/mpi/wiki/YaimConfig

Configurazione in my-site-info.def: VEDI /opt/glite/yaim/examples/siteinfo/services

#MPI_MPICH_ENABLE="yes"
#MPI_MPICH_PATH="/opt/mpich-1.2.7p1/"
#MPI_MPICH_VERSION="1.2.7p1"

#MPI_MPICH_MPIEXEC="/usr/bin/mpiexec"

MPI_OPENMPI_ENABLE="yes"
MPI_OPENMPI_PATH="/opt/openmpi/1.2.5/"
MPI_OPENMPI_VERSION="1.2.5"
MPI_SHARED_HOME="yes"
MPI_SSH_HOST_BASED_AUTH="yes"

# If WNs and CE share /home, set this (orig is JOB_MANAGER=lcgpbs )
JOB_MANAGER=pbs

Config WN:

/opt/glite/yaim/bin/ig_yaim -c -s /root/my-site-info4.def -n ig_WN_torque_noafs -n MPI_WN 

Config CE (manca yaim-mpi)

/opt/glite/yaim/bin/ig_yaim -c -s /root/my-site-info.def -n ig_CE_torque -n BDII_site -n MPI_CE

MPI su GRID

Note di utilizzo su GRID

Repository dei programmi

Ricerca sullo stato delle installazioni MPI sui siti di theophys:

lcg-info --list-ce  -vo theophys  --query "Tag=*MPI*"  --attrs "Tag" | less

Ricerca dei siti che supportano MPI-START e OPENMPI

lcg-info --list-ce  --query "Tag=MPI-START*" --query "Tag=OPENMPI"  --attrs "CE" | less

Esecuzione con Mpi-Start

Mpi-Start dovrebbe semplificare e di molto la vita agli utenti Mpi; teoricamente dovrebbe essere possibile utilizzare Mpi-Start anche con siti che non lo supportano (es tutta la vo theophys).

si prepara un Jdl nel quale si inseriscono come arguments il nome dell'eseguibile che verrà compilato e il flavour di MPI scelto (implementazione); ovviamente il flavour viene inserito nei requirements…

[fabrizio.cornia@grid-ui2 mpitest]$ cat mpi-start.jdl
JobType = "MPICH";
NodeNumber = 4;
Executable = "mpi-start-wrapper.sh";
Arguments = "mpi-test OPENMPI";
StdOutput = "mpi-test.out";
StdError = "mpi-test.err";
InputSandbox = {"mpi-start-wrapper.sh","mpi-hooks.sh","mpi-test.c","mpi-start-0.0.46.tar.gz"};
OutputSandbox = {"mpi-test.err","mpi-test.out"};
Requirements =
  Member("MPICH", other.GlueHostApplicationSoftwareRunTimeEnvironment) &&
  Member("OPENMPI", other.GlueHostApplicationSoftwareRunTimeEnvironment) &&
  TRUE;

e uno script wrapper che include lo spacchettamento di mpi-start

[fabrizio.cornia@grid-ui2 mpitest]$ cat mpi-start-wrapper.sh
#!/bin/bash

# untar mpi-start and set up variables
tar xzf mpi-start-0.0.46.tar.gz
export I2G_MPI_START=bin/mpi-start
MPIRUN=`which mpirun`
export MPI_MPICH_PATH=`dirname $MPIRUN`

# Pull in the arguments.
MY_EXECUTABLE=`pwd`/$1
MPI_FLAVOR=$2

# Convert flavor to lowercase for passing to mpi-start.
MPI_FLAVOR_LOWER=`echo $MPI_FLAVOR | tr '[:upper:]' '[:lower:]'`

# Pull out the correct paths for the requested flavor.
eval MPI_PATH=`printenv MPI_${MPI_FLAVOR}_PATH`

# Ensure the prefix is correctly set.  Don't rely on the defaults.
eval I2G_${MPI_FLAVOR}_PREFIX=$MPI_PATH
export I2G_${MPI_FLAVOR}_PREFIX

# Touch the executable.  It exist must for the shared file system
check.
# If it does not, then mpi-start may try to distribute the executable
# when it shouldn't.
touch $MY_EXECUTABLE

# Setup for mpi-start.
export I2G_MPI_APPLICATION=$MY_EXECUTABLE
export I2G_MPI_APPLICATION_ARGS=
export I2G_MPI_TYPE=$MPI_FLAVOR_LOWER
export I2G_MPI_PRE_RUN_HOOK=mpi-hooks.sh
export I2G_MPI_POST_RUN_HOOK=mpi-hooks.sh

# If these are set then you will get more debugging information.
export I2G_MPI_START_VERBOSE=1
#export I2G_MPI_START_DEBUG=1

# Invoke mpi-start.
$I2G_MPI_START

nel file mpi-hooks.sh invece si inseriscono le istruzioni per la compilazione; in questo modo l'eseguibile viene compilato direttamente dai nodi che lo eseguono, evitando problemi di portabilità a causa dell'utilizzo di versioni differenti dell'implementazione.

[fabrizio.cornia@grid-ui2 mpitest]$ cat mpi-hooks.sh
#!/bin/sh

#
# This function will be called before the MPI executable is started.
# You can, for example, compile the executable itself.
#
pre_run_hook () {

  # Compile the program.
  echo "Compiling ${I2G_MPI_APPLICATION}"

  # Actually compile the program.
  cmd="mpicc ${MPI_MPICC_OPTS} -o ${I2G_MPI_APPLICATION} ${I2G_MPI_APPLICATION}.c"
  echo $cmd
  $cmd
  if [ ! $? -eq 0 ]; then
    echo "Error compiling program.  Exiting..."
    exit 1
  fi

  # Everything's OK.
  echo "Successfully compiled ${I2G_MPI_APPLICATION}"

  return 0
}

#
# This function will be called before the MPI executable is finished.
# A typical case for this is to upload the results to a storage
element.
#
post_run_hook () {

  echo "Executing post hook."
  echo "Finished the post hook."

  return 0
}

Cambiamenti recenti

Il TCG workinggroup on MPI ha proposto di apportare delle modifiche al codice sorgente del middleware per consentire di eseguire job distribuiti su nodi multipli utilizzando come JobType “Normal” anziché “MPICH”, in questo modo il wrapper verrebbe eseguito su un singolo nodo e la chiamata a mpirun contenuta al suo interno non creerà problemi. In alternativa propongono di sostituire mpirun con uno script dummy che invece che distribuire il job su N worker node lo esegue semplicemente come se non fosse un job mpi. Oltre a questo il TCG workinggroup propone anche l'uso di mpi-start come sostituti di mpirun per rendere semplice la coesistenza di varie implementazioni e versioni di mpi. Infine in soluzioni più a lungo termine introducono l'uso di nuovi JobType, come per esempio “Parallel” o “MultiParallel” per migliorare la gestione dei nodi e delle cpu.

Intercompatibilità tra versioni

OpenMPI FAQ: System administrator-level technical information about Open MPI

11) I have an MPI application compiled for another MPI; will it work with Open MPI?

      It is strongly unlikely.
      Open MPI does not attempt to interface to other MPI implementations, nor executables that were compiled for them.
      Sorry!
      MPI applications need to be compiled and linked with Open MPI in order to run under Open MPI.

In sintesi è meglio evitare

Open RTE

Summary

Il documeto delinea gli obiettivi della versione 2 dell'Open RunTime Environment, tra i quali:

  • creazione e gestione di un ambiente da parte di uno user, come era per LAM;
  • un ambiente può essere reso persistente, legato al concetto di fault-tollerance;
  • hotplug-in/out di un job da un ambiente;
  • migrazione di un job in un altro nodo;
  • gestione più accurata degli errori;
  • monitoraggio degli ambienti;
  • General-Purpose Registry;
  • in futuro:
    • gestione di nodi con home not shared;
    • shared memory tra i job dell'universo.

LINPACK: HPL

BLAS

Con yum o apt-get installare “The BLAS (Basic Linear Algebra Subprograms) library”.

ATLAS

(Compilare e) copiare le librerie ATLAS in /usr/lib:

libatlas.a
libcblas.a
libf77blas.a
liblapack.a

e gli header in /usr/header:

cblas.h
clapack.h

HPL

Copiare nella directory hpl il seguente file “Make.Linux_PII_CBLAS_gm”. Compilare il tutto:

make arch=Linux_PII_CBLAS_gm


Se tutto è andato a buon fine, nella cartella hpl/bin/Linux_PII_CBLAS_gm è stato creati il file:

xhpl


[user@grid-ui hpl]$ cat Make.Linux_PII_CBLAS_gm
#
#  -- High Performance Computing Linpack Benchmark (HPL)
#     HPL - 1.0a - January 20, 2004
#     Antoine P. Petitet
#     University of Tennessee, Knoxville
#     Innovative Computing Laboratories
#     (C) Copyright 2000-2004 All Rights Reserved
#
#  -- Copyright notice and Licensing terms:
#
#  Redistribution  and  use in  source and binary forms, with or without
#  modification, are  permitted provided  that the following  conditions
#  are met:
#
#  1. Redistributions  of  source  code  must retain the above copyright
#  notice, this list of conditions and the following disclaimer.
#
#  2. Redistributions in binary form must reproduce  the above copyright
#  notice, this list of conditions,  and the following disclaimer in the
#  documentation and/or other materials provided with the distribution.
#
#  3. All  advertising  materials  mentioning  features  or  use of this
#  software must display the following acknowledgement:
#  This  product  includes  software  developed  at  the  University  of
#  Tennessee, Knoxville, Innovative Computing Laboratories.
#
#  4. The name of the  University,  the name of the  Laboratory,  or the
#  names  of  its  contributors  may  not  be used to endorse or promote
#  products  derived   from   this  software  without  specific  written
#  permission.
#
#  -- Disclaimer:
#
#  THIS  SOFTWARE  IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
#  ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,  INCLUDING,  BUT NOT
#  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
#  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE UNIVERSITY
#  OR  CONTRIBUTORS  BE  LIABLE FOR ANY  DIRECT,  INDIRECT,  INCIDENTAL,
#  SPECIAL,  EXEMPLARY,  OR  CONSEQUENTIAL DAMAGES  (INCLUDING,  BUT NOT
#  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
#  DATA OR PROFITS; OR BUSINESS INTERRUPTION)  HOWEVER CAUSED AND ON ANY
#  THEORY OF LIABILITY, WHETHER IN CONTRACT,  STRICT LIABILITY,  OR TORT
#  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
#  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
# ######################################################################
#
# ----------------------------------------------------------------------
# - shell --------------------------------------------------------------
# ----------------------------------------------------------------------
#
SHELL        = /bin/sh
#
CD           = cd
CP           = cp
LN_S         = ln -s
MKDIR        = mkdir
RM           = /bin/rm -f
TOUCH        = touch
#
# ----------------------------------------------------------------------
# - Platform identifier ------------------------------------------------
# ----------------------------------------------------------------------
#
ARCH         = Linux_PII_CBLAS_gm
#
# ----------------------------------------------------------------------
# - HPL Directory Structure / HPL library ------------------------------
# ----------------------------------------------------------------------
#
TOPdir       = $(HOME)/hpl
INCdir       = $(TOPdir)/include
BINdir       = $(TOPdir)/bin/$(ARCH)
LIBdir       = $(TOPdir)/lib/$(ARCH)
#
HPLlib       = $(LIBdir)/libhpl.a
#
# ----------------------------------------------------------------------
# - Message Passing library (MPI) --------------------------------------
# ----------------------------------------------------------------------
# MPinc tells the  C  compiler where to find the Message Passing library
# header files,  MPlib  is defined  to be the name of  the library to be
# used. The variable MPdir is only used for defining MPinc and MPlib.
#
MPdir        =
MPinc        =
MPlib        =
#
# ----------------------------------------------------------------------
# - Linear Algebra library (BLAS or VSIPL) -----------------------------
# ----------------------------------------------------------------------
# LAinc tells the  C  compiler where to find the Linear Algebra  library
# header files,  LAlib  is defined  to be the name of  the library to be
# used. The variable LAdir is only used for defining LAinc and LAlib.
#
#LAdir        = $(HOME)/netlib/ARCHIVES/Linux_PII
LAinc        =
LAlib        = /usr/lib/libcblas.a /usr/lib/libatlas.a
#
# ----------------------------------------------------------------------
# - F77 / C interface --------------------------------------------------
# ----------------------------------------------------------------------
# You can skip this section  if and only if  you are not planning to use
# a  BLAS  library featuring a Fortran 77 interface.  Otherwise,  it  is
# necessary  to  fill out the  F2CDEFS  variable  with  the  appropriate
# options.  **One and only one**  option should be chosen in **each** of
# the 3 following categories:
#
# 1) name space (How C calls a Fortran 77 routine)
#
# -DAdd_              : all lower case and a suffixed underscore  (Suns,
#                       Intel, ...),                           [default]
# -DNoChange          : all lower case (IBM RS6000),
# -DUpCase            : all upper case (Cray),
# -DAdd__             : the FORTRAN compiler in use is f2c.
#
# 2) C and Fortran 77 integer mapping
#
# -DF77_INTEGER=int   : Fortran 77 INTEGER is a C int,         [default]
# -DF77_INTEGER=long  : Fortran 77 INTEGER is a C long,
# -DF77_INTEGER=short : Fortran 77 INTEGER is a C short.
#
# 3) Fortran 77 string handling
#
# -DStringSunStyle    : The string address is passed at the string loca-
#                       tion on the stack, and the string length is then
#                       passed as  an  F77_INTEGER  after  all  explicit
#                       stack arguments,                       [default]
# -DStringStructPtr   : The address  of  a  structure  is  passed  by  a
#                       Fortran 77  string,  and the structure is of the
#                       form: struct {char *cp; F77_INTEGER len;},
# -DStringStructVal   : A structure is passed by value for each  Fortran
#                       77 string,  and  the  structure is  of the form:
#                       struct {char *cp; F77_INTEGER len;},
# -DStringCrayStyle   : Special option for  Cray  machines,  which  uses
#                       Cray  fcd  (fortran  character  descriptor)  for
#                       interoperation.
#
F2CDEFS      =
#
# ----------------------------------------------------------------------
# - HPL includes / libraries / specifics -------------------------------
# ----------------------------------------------------------------------
#
HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) $(LAinc) $(MPinc)
HPL_LIBS     = $(HPLlib) $(LAlib) $(MPlib)
#
# - Compile time options -----------------------------------------------
#
# -DHPL_COPY_L           force the copy of the panel L before bcast;
# -DHPL_CALL_CBLAS       call the cblas interface;
# -DHPL_CALL_VSIPL       call the vsip  library;
# -DHPL_DETAILED_TIMING  enable detailed timers;
#
# By default HPL will:
#    *) not copy L before broadcast,
#    *) call the BLAS Fortran 77 interface,
#    *) not display detailed timing information.
#
HPL_OPTS     = -DHPL_CALL_CBLAS
#
# ----------------------------------------------------------------------
#
HPL_DEFS     = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES)
#
# ----------------------------------------------------------------------
# - Compilers / linkers - Optimization flags ---------------------------
# ----------------------------------------------------------------------
#
CC           = mpicc
CCNOOPT      = $(HPL_DEFS)
CCFLAGS      = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops -W -Wall
#
# On some platforms,  it is necessary  to use the Fortran linker to find
# the Fortran internals used in the BLAS library.
#
LINKER       = mpif77
LINKFLAGS    = $(CCFLAGS)
#
ARCHIVER     = ar
ARFLAGS      = r
RANLIB       = echo
#
# ----------------------------------------------------------------------

Avviare i test

Predisporre un file HPL.dat con le specifiche dei test da eseguire: vedi Progetto QUARTO - Test LINPACK

/var/www/html/dokuwiki/data/pages/grid/mpi.txt · Ultima modifica: Y/m/d H:i da