Strumenti Utente

Strumenti Sito


grid:infiniband

openmpi/infiniband

packages installation

From SL distribution:

yum groupinstall -y openfabrics-enterprise-distribution  #in case of infiniband
yum install      -y libibverbs-devel

From Mellanox distribution (MLNX_OFED_LINUX-1.5.1-rhel5.4.iso)

 mount -o loop MLNX_OFED_LINUX-1.5.1-rhel5.4.iso /mnt/cdrom
 /mnt/cdrom/mlnxofedinstall

Components

openib

The base package that includes the openibd init script that loads all the necessary kernel modules on boot up

libmthca, libmlx (HCA Drivers)

Modules that implement the hardware specific driver for Mellanox InfiniBand host channel adapters (HCAs) for the libibverbs library.

ibverbs, Mads, SA, CM, uverbs umads (Mid-layer core)

The core user space library that implements the hardware abstracted verbs protocol on both infiniband

opensm

An infiniband subnet manager.

Configuration

On all nodes:

/etc/init.d/openibd start
chkconfig openibd on

On the node acting as “head node”:

yum install -y opensm.x86_64 #in case
chkconfig opensmd on
service opensmd start

Device installation

SL5.4 distribution:

libmthca-1.0.5-4.el5.x86_64
    /usr/lib64/libmthca.so
    /etc/libibverbs.d/mthca.driver
libibverbs-devel-1.1.2-4.el5.x86_64
    /usr/include/infiniband/driver.h
libibverbs-1.1.2-4.el5.x86_64
    /etc/libibverbs.d
    /usr/lib64/libibverbs.so.1
libibverbs-utils-1.1.2-4.el5.x86_64
    /usr/bin/ibv_devinfo
    /usr/bin/ibv_devices
openib-1.4.1-3
    /etc/ofed/openib.conf
    /etc/rc.d/init.d/openibd
    /etc/sysconfig/network-scripts/ifup-ib
Diagniostics
 ibstat
 ibv_devinfo
     port: 1  state: active (4)
     port: 2  state: down (1)
 ibv_devices
    device                 node GUID
    ------              ----------------
    mthca0              0002c90200002514
 albert202> ib_write_bw
 albert203> ib_write_bw albert202
------------------------------------------------------------------
#bytes #iterations    BW peak[MB/sec]    BW average[MB/sec]  
 65536        5000             410.85                410.85
------------------------------------------------------------------

albert202> ib_write_lat

albert203> ib_write_lat albert202
------------------------------------------------------------------
#bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]
     2        1000           4.80          44.52             4.92
------------------------------------------------------------------
perfquery 
ibhosts
ibswitch
ibnetdiscover

Verify

Verify supported communications:

  ompi_info | grep btl

Most common btl modules are:

  • self (loopback)
  • sm (Shared Memory)
  • tcp
  • gm (GM MYRINET)
  • mx (MX MYRINET)
  • openib (INFINIBAND OPENIB)
  • mvapi (INFINIBAND MELLANOX)

The fastest module is automatically selected.

ompi_info -a | grep exclusivity
> MCA btl: parameter "btl_ofud_exclusivity" (current value: "1024", data source: default value)
> MCA btl: parameter "btl_self_exclusivity" (current value: "65536", data source: default value)
> MCA btl: parameter "btl_sm_exclusivity" (current value: "65535", data source: default value)
> MCA btl: parameter "btl_tcp_exclusivity" (current value: "100", data source: default value)

RLIMIT_MEMLOCK problem

Per il problema del memlock: in /etc/init.d/sshd:

start()
{
      # Create keys if necessary
      do_rsa1_keygen 
      do_rsa_keygen
      do_dsa_keygen                     
      ulimit -l unlimited #!!
..
}
service sshd restart

IPoIB

lsmod | grep ib_ipoib
modprobe ib_ipoib
ifconfig ib0
ifconfig ib0 10.0.0.50/24
route

recompile openMPI

yum install      -y libibverbs-devel   
rpmbuild -bb --define 'configure_options --with-openib             --disable-ipv6' ..
rpmbuild -bb --define 'configure_options --with-openib=/usr --with-openib-libdir=/usr/lib64   --disable-ipv6' ..'

BTL selection

It is possible to force a specific module:

mpirun -mca btl openib,self -np 2  -host albert202,albert203 mpi_name
cexec albert2: "echo btl = self,sm,ofud,tcp >   /opt/openmpi/1.4.1/etc/openmpi-mca-params.conf"
cexec wn2:  "echo btl = self,sm,ofud,tcp >>  /usr/lib64/openmpi/1.4-gcc/etc/openmpi-mca-params.conf"

Annex

Kernel modules

kernel 2.6 introduces a new file naming convention: kernel modules now have a .ko extension (in place of the old .o extension) which easily distinguishes them from conventional object files. http://tldp.org/LDP/lkmpg/2.6/html/x181.html

modinfo /lib/modules/2.6.18-194.3.1.el5/updates/kernel/net/rds/rds.ko
/var/www/html/dokuwiki/data/pages/grid/infiniband.txt · Ultima modifica: Y/m/d H:i da