What is Soft-iWARP and why installing it :
Soft-iWARP [1] is a software-based iWARP stack that runs at reasonable performance levels and seamlessly fits into the OFA RDMA environment provides several benefits:
- As a generic (RNIC-independent) iWARP device driver, it immediately enables RDMA services on all systems with conventional Ethernet adapters, which do not provide RDMA hardware support.
- Soft-iWARP can be an intermediate step when migrating applications and systems to RDMA APIs and OpenFabrics.
- Soft-iWARP can be a reasonable solution for client systems, allowing RNIC-equipped peers/servers to enjoy the full benefits of RDMA communication.
- Soft-iWARP seamlessly supports direct as well as asynchronous transmission with multiple outstanding work requests and RDMA operations.
- A software-based iWARP stack may flexibly employ any available hardware assists for performance-critical operations such as MPA CRC checksum calculation and direct data placement. The resulting performance levels may approach those of a fully offloaded iWARP stack.
How to install Soft-iWARP :
When I tried to install Soft-iWARP I ran into several issue and I am writing this installation manual so i don’t forget how to do it and other people can enjoy a semi seamless installation.
My setup:
- Core 2 duo
- e1000 nic
- 4 Gb of RAM
- Ubuntu 10.10 server
- Linux kernel : 2.6.35-28-generic
Pre-Install
Getting the Linux kernel dev environment:
sudo apt-get install fakeroot build-essential crash kexec-tools makedumpfile kernel-wedge
sudo apt-get build-dep linux
sudo apt-get install git-core libncurses5 libncurses5-dev libelf-dev binutils-dev
Setting up the Infiniband / libibverbs environment
sudo apt-get install libibverbs1 libibcm1 libibcm-dev libibverbs-dev libibcommon1 ibverbs-utils
check if “/etc/udev/rules.d/40-ib.rules” exist
1. If the file doesn’t exist , create it and populate it with (you need to reboot after):
#### /etc/udev/rules.d/40-ib.rules ####KERNEL=="umad*", NAME="infiniband/%k"KERNEL=="issm*", NAME="infiniband/%k"KERNEL=="ucm*", NAME="infiniband/%k", MODE="0666"KERNEL=="uverbs*", NAME="infiniband/%k", MODE="0666"KERNEL=="uat", NAME="infiniband/%k", MODE="0666"KERNEL=="ucma", NAME="infiniband/%k", MODE="0666"KERNEL=="rdma_cm", NAME="infiniband/%k", MODE="0666"########
Verifying the Infiniband / libibverbs environment
· Run these command in the following order:
1. sudo modprobe rdma_cm
2. sudo modprobe ib_uverbs
3. sudo modprobe rdma_ucm
4. sudo lsmod
5.ls /dev/infiniband/
Getting the compile / dev environment for Soft-iWARP :
- sudo apt-get install libtool autoconf
Installing Soft-iWARP :
Getting Soft-iWARP
Getting the source from the git repository :
- You can get the source from : https://github.com/zrlio/softiwarp
Compiling and installing the Soft-iWARP kernel module :
Inside the soft-iwarp kernel module folder:
- make
- sudo make install
Compiling and installing the Soft-iWARP userlib:
Inside the soft-iwarp userlib folder:
- ./autogen.sh
- Run AGAIN ./autogen.sh
- ./configure
- make
- sudo make install
· at that stage soft-iwrap install the infiniband stuff in “/usr/local/etc/libibverbs.d” we need to make asymbolic link t /etc/infiniband.d
o sudo ln -s /usr/local/etc/libibverbs.d /etc//libibverbs.d
Inserting the Soft-iWARP kernel module:
1. sudo modprobe rdma_cm
2. sudo modprobe ib_uverbs
3. sudo modprobe rdma_ucm
4. sudo insmod /lib/modules/"your_kernel"/extra/siw.ko
5. sudo lsmod
§ check if all the modules are correctly installed
6. ls /dev/infiniband/
Soft-iWARP kernel module parameters:
loopback_enabled: if set, attaches siw also to the looback device.
To be set only during module insertion.
mpa_crc_enabled: if set, the MPA CRC gets generated and checked
both in tx and rx path (kills all throughput).
To be set only during module insertion.
zcopy_tx: if set, payload of non signalled work requests
(such as non signalled WRITE or SEND as well as all READ
responses) are transferred using the TCP sockets
sendpage interface. This parameter can be switched on and
off dynamically (echo 1 >> /sys/module/siw/parameters/zcopy_tx
for enablement, 0 for disabling). System load benefits from
using 0copy data transmission: a servers load of
one 2.5 GHz CPU may drop from 55 percent down to 23 percent
if serving 100KByte READ responses at 10GbE's line speed.
Verifying the Soft-iWARP is correctly installed :
running :
- ibv_devinfo
- ibv_devices
· You should get :
·
- if you get:
libibverbs: Warning: couldn't load driver 'siw': libsiw-rdmav2.so: cannot open shared object file: No such file or directory
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
o you need to run “sudo ldconfig” to update the library path
Running the simple test:
- sudo apt-get install rdmacm-utils
Then you can test the soft-iwarp with :
- rping : RDMA ping
- udaddy : Establishes a set of unreliable RDMA datagram communication paths between two nodes using the librdmacm, optionally transfers datagrams between the nodes, then tears down the communication.
- mckey : Establishes a set of RDMA multicast communication paths between nodes using the librdmacm, optionally transfers datagrams to receiving nodes, then tears down the communication.
- ucmatose : Establishes a set of reliable RDMA connections between two nodes using the librdmacm, optionally transfers data between the nodes, then disconnects.
Note : the ibv_* utils don't work with iWARP device . you will get errors if you try to use them such as:
[root@iwarp1 ~]# ibv_rc_pingpong Couldn't get local LID [root@iwarp1 ~]# ibv_srq_pingpong Couldn't create SRQ [root@iwarp1 ~]# ibv_srq_pingpong Couldn't create SRQ [root@iwarp1 ~]# ibv_uc_pingpong Couldn't create QP [root@iwarp1 ~]# ibv_ud_pingpong Couldn't create QP
Note 2 : If you only have one machine and you want to use the loopback (optional you can still use your normal nic), to test if you have it enabled: