Thursday, March 31, 2011

How to install Soft-iWARP on Ubuntu 10.10, AKA how to have RDMA enabled system without the expensive hardware.

What is Soft-iWARP and why installing it : 

Soft-iWARP [1] is a software-based iWARP stack that runs at reasonable performance levels and seamlessly fits into the OFA RDMA environment provides several benefits:
  • As a generic (RNIC-independent) iWARP device driver, it immediately enables RDMA services on all systems with conventional Ethernet adapters, which do not provide RDMA hardware support.
  • Soft-iWARP can be an intermediate step when migrating applications and systems to RDMA APIs and OpenFabrics.
  • Soft-iWARP can be a reasonable solution for client systems, allowing RNIC-equipped peers/servers to enjoy the full benefits of RDMA communication.
  • Soft-iWARP seamlessly supports direct as well as asynchronous transmission with multiple outstanding work requests and RDMA operations.
  • A software-based iWARP stack may flexibly employ any available hardware assists for performance-critical operations such as MPA CRC checksum calculation and direct data placement. The resulting performance levels may approach those of a fully offloaded iWARP stack.

How to install Soft-iWARP  :

When I tried to install Soft-iWARP   I ran into several issue and I am writing this installation manual so i don’t forget how to do it and other people can enjoy  a semi seamless installation.

My setup: 

  •          Core 2 duo
  •          e1000 nic
  •          4 Gb of RAM
  •          Ubuntu 10.10 server
  •          Linux kernel : 2.6.35-28-generic

Getting the Linux kernel dev environment:
  • sudo apt-get install fakeroot build-essential crash kexec-tools makedumpfile kernel-wedge
  • sudo apt-get build-dep linux
  • sudo apt-get install git-core libncurses5 libncurses5-dev libelf-dev binutils-dev

Setting up  the Infiniband / libibverbs  environment
  • sudo apt-get install libibverbs1 libibcm1 libibcm-dev  libibverbs-dev  libibcommon1 ibverbs-utils
  • check if  “/etc/udev/rules.d/40-ib.rules” exist
1.       If the file doesn’t exist , create it and populate it with (you need to reboot after):

 ####  /etc/udev/rules.d/40-ib.rules  ####
 KERNEL=="umad*", NAME="infiniband/%k"
 KERNEL=="issm*", NAME="infiniband/%k"
 KERNEL=="ucm*", NAME="infiniband/%k", MODE="0666"
 KERNEL=="uverbs*", NAME="infiniband/%k", MODE="0666"
 KERNEL=="uat", NAME="infiniband/%k", MODE="0666"
 KERNEL=="ucma", NAME="infiniband/%k", MODE="0666"
 KERNEL=="rdma_cm", NAME="infiniband/%k", MODE="0666"

Verifying the Infiniband  / libibverbs  environment

·         Run these command in the following order:
1.  sudo modprobe  rdma_cm
2.  sudo modprobe ib_uverbs
3.  sudo modprobe rdma_ucm
4.  sudo lsmod /dev/infiniband/

Getting the compile / dev environment for Soft-iWARP   : 
  • sudo apt-get install libtool autoconf         

Installing Soft-iWARP   :

Getting Soft-iWARP   
Getting the source from the git repository :

Compiling and installing the Soft-iWARP    kernel module : 

Inside the soft-iwarp kernel module folder:
  1.          make
  2.          sudo make install
Compiling and installing the Soft-iWARP   userlib: 

Inside the soft-iwarp userlib folder:
  1.          ./
  2.          Run AGAIN  ./
  3.          ./configure
  4.          make
  5.          sudo make install
·         at that stage soft-iwrap install the infiniband stuff in “/usr/local/etc/libibverbs.d” we need to make asymbolic link t /etc/infiniband.d
o   sudo ln -s /usr/local/etc/libibverbs.d /etc//libibverbs.d

Inserting the  Soft-iWARP    kernel module:
1.  sudo modprobe  rdma_cm
2.  sudo modprobe ib_uverbs
3.  sudo modprobe rdma_ucm
4.  sudo    insmod  /lib/modules/"your_kernel"/extra/siw.ko
5.  sudo lsmod
§  check if all the modules are correctly installed
6.  ls /dev/infiniband/

Soft-iWARP    kernel  module parameters:
        loopback_enabled: if set, attaches siw also to the looback device.
                To be set only during module insertion.

        mpa_crc_enabled:  if set, the MPA CRC gets generated and checked
                both in tx and rx path (kills all throughput).
                To be set only during module insertion.

        zcopy_tx:         if set, payload of non signalled work requests
                (such as non signalled WRITE or SEND as well as all READ
                 responses) are transferred using the TCP sockets
                sendpage interface. This parameter can be switched on and
                off dynamically (echo 1 >> /sys/module/siw/parameters/zcopy_tx
                for enablement, 0 for disabling). System load benefits from
                using 0copy data transmission: a servers load of
                one 2.5 GHz CPU may drop from 55 percent down to 23 percent
                if serving 100KByte READ responses at 10GbE's line speed.

Verifying  the Soft-iWARP    is correctly installed :
running :
  •          ibv_devinfo
  •          ibv_devices
·         You should get :

  • if you get:
libibverbs: Warning: couldn't load driver 'siw': cannot open shared object file: No such file or directory
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
o   you need to run “sudo  ldconfig” to update the library path

Running the simple test: 

Install rdma-utils :
  • sudo apt-get install  rdmacm-utils

Then you can test the soft-iwarp with :
  • rping : RDMA ping
  • udaddy : Establishes a set of unreliable RDMA datagram communication paths between two nodes using the librdmacm, optionally transfers datagrams between the nodes, then tears down the communication.
  • mckey : Establishes a set of RDMA multicast communication paths between nodes using the librdmacm, optionally transfers datagrams to receiving nodes, then tears down the communication.
  • ucmatose  : Establishes a set of reliable RDMA connections between two nodes using the librdmacm, optionally transfers data between the nodes, then disconnects.

Note : the ibv_* utils don't work with iWARP device . you will get errors if you try to use them such as:

[root@iwarp1 ~]# ibv_rc_pingpong
Couldn't get local LID
[root@iwarp1 ~]# ibv_srq_pingpong
Couldn't create SRQ
[root@iwarp1 ~]# ibv_srq_pingpong
Couldn't create SRQ
[root@iwarp1 ~]# ibv_uc_pingpong
Couldn't create QP
[root@iwarp1 ~]# ibv_ud_pingpong
Couldn't create QP
Note 2 : If you only have one machine and you want to use the  loopback (optional you can still use your normal nic), to test if you have it enabled: