On allowing shorter timeout on Mellanox cards and other tips and tricks

Modifying the minimum Timeout detection on Mellanox cards: 

If you want to leverage  the connection timeout detection of Mellanox card to setup/design a fault tolerant system you very quickly realize that the tools for detecting a crash at your disposition are using a resolution time an order of magnitude higher then the actual latency you are aiming for. This has some significant effect on the overall cluster management , fault detection and fault recovery system you can design. But luckily there is some workaround the problem. 

First the issue : 

Mellanox Connext2 NICs enforce a lower limit on timeouts (specifically, the IBV_QP_TIMEOUT option). For these cards the minimum timeout value on conenctX2 is 500ms combined with the default setting of 7 retries, this means that after a timeout (e.g., a crashed server) the transmit buffer is held by the NIC for about 4 seconds before it is returned with an error.

The consequence: 

You can have the nodes that maintained a connection with the faulty server  running out of transmit buffers , which either leads to errors, or leave the the whole cluster hanging for a couple of seconds... Not really nice. 

The solution : 

To fix the problem, you need to modify  the firmware in the NICs as follow: 

• Get from Mellanox the appropriate version of the firmware to start with. 
• This file needs to be combined with an appropriate .ini file.  First, fetch the existing .ini file from the NIC:
      flint -d /dev/mst/mtXXXXXX_pci_cr0 dc > MT_XXXXXX.ini
  Check /dev/mst to verify the file name there.  In this case the .ini file is named after the board_id printed by ibv_devinfo 
• Edit the .ini file to add a new qp_minimal_timeout_val parameter with a value of zero. It goes in the HCA section, like this:

[HCA]
hca_header_device_id = 0x673c
hca_header_subsystem_id = 0x0018
dpdp_en = true
eth_xfi_en = true
mdio_en_port1 = 0
qp_minimal_timeout_val = 0

Generate a new image from the .mlx file and the .ini file:     mlxburn -fw fw-ConnectX2-rel.mlx -conf MT_XXXXXX.ini -wrimage MT_XXXXX.bin
Upload the image into the NIC:
    flint -d /dev/mst/mtXXXXX_pci_cr0 -i MT_0DD0120009.bin -y  b
Beware: if different NICs have different board ids, they will need different .ini files, and they may need different .mlx files (ask Mellanox for help).




Now you can setup timeout as low as you want!! But beware of the consequence of unnecessary retries operations. You really need to profile your setup in order to find the best timeout resolution.

Bonus: Testing your IB setup speeds: 


Checking Link Speeds

Run iblinkinfo as root. This will show link speeds of all ports in the network (both on switches and HCAs).








 iblinkinfo |grep \"rc

          50   15[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>      51    1[  ] "rc41 HCA-1" ( )

 ...

          41   35[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>      45    1[  ] "rcnfs HCA-1" ( )

          41   36[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>       1    1[  ] "rcmaster HCA-1" ( )

          48   15[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>      26    1[  ] "rc02 HCA-1" ( )

          48   16[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>       6    1[  ] "rc22 HCA-1" ( )

          48   17[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>      23    1[  ] "rc30 HCA-1" ( )

          48   18[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>      30    1[  ] "rc10 HCA-1" ( )

          48   19[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>      16    1[  ] "rc28 HCA-1" ( )

          48   20[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>      40    1[  ] "rc06 HCA-1" ( )

          48   21[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>      37    1[  ] "rc18 HCA-1" ( )

          48   22[  ] ==( 4X  2.5 Gbps Active/  LinkUp)==>      36    1[  ] "rc14 HCA-1" ( Could be 10.0 Gbps)

          48   23[  ] ==( 4X 10.0 Gbps Active/  LinkUp)==>      29    1[  ] "rc16 HCA-1" ( )

...







Measuring Bandwidth

ib_send_lat will measure bandwidth between two hosts using the send/recv verbs. An example follows below.

Src host:






 ib_send_bw rc16ib

------------------------------------------------------------------

                    Send BW Test

 Number of qps   : 1

 Connection type : RC

 TX depth        : 300

 CQ Moderation   : 50

 Link type       : IB

 Mtu             : 2048

 Inline data is used up to 0 bytes message

 local address: LID 0x24 QPN 0x80049 PSN 0xe895fd

 remote address: LID 0x1d QPN 0x200049 PSN 0xb960d2

------------------------------------------------------------------

 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]

 65536     1000           939.34             939.34

------------------------------------------------------------------







Dst host:






 ib_send_bw

------------------------------------------------------------------

                    Send BW Test

 Number of qps   : 1

 Connection type : RC

 RX depth        : 600

 CQ Moderation   : 50

 Link type       : IB

 Mtu             : 2048

 Inline data is used up to 0 bytes message

 local address: LID 0x1d QPN 0x200049 PSN 0xb960d2

 remote address: LID 0x24 QPN 0x80049 PSN 0xe895fd

------------------------------------------------------------------

 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]

 65536     1000           -nan               940.85

------------------------------------------------------------------







Measuring Latency

Use ib_send_lat or ibv_ud_pingpong as above. Note that the two apps may have different defaults for packet sizes, inlining, etc.

On avoiding vendor lock-in by leveraging Openstack

One of the main drivers for user to adopt Openstack is to avoid vendor lock-in (see stats here ).

Architecture is rapidly becoming a commodity

Arguably, if you develop you own cloud solution you are locking yourself into yourself. Openstack in its current state require so much effort, customization, and maintenance that you end up building your own cage. Managing your maintenance and devs cost becomes critical in order to have a good ROI. Unless you plan to resell these services or expose them directly to your customers you won't benefit from the scaling strategy.

Often you might be better off  with a vendor lock-in as you "should" more easily control your costs and ROI. Or better, contract out your Openstack implementation from a third party and outsource the maintenance and development cost while retaining a certain degree of flexibility.

Different size , different strategy different risks

SMB customers can be very aggressive about getting into the cloud, and they do not have a legacy to deal with, whereas the enterprises tend to be very risk-averse. They have to protect what they have, and they cannot be as aggressive.

As a result we are seeing a number of mature enterprises looking toward a multicloud strategy. Whether that is through multiple platforms or whether it's deploying on an open cloud platform, the outcome that they are trying to achieve is the same. Enterprises are increasingly transitioning from general-purpose tools to point solutions as their IT environments become bigger and more complex.

Openstack consumption model : DiY vs Enterprise

Here are some personal comments on Openstack adoption in the industry and their statistics.

Openstack statistics: 

While the Openstack consortium is putting great effort in diffusing its statistics  to market / broaden its adoption ( you can see the latest batch here:  http://www.openstack.org/blog/2013/11/openstack-user-survey-statistics-november-2013/ ) you have to take them with a grain of salt.

One of the thing that is not immediately visible, but you can guess it, is the consumption model of Openstack. Openstack cloud deployment can be broadly classified in two different category: Do it yourself  ( DiY) , and (semi) contracted out  ( aka enterprise cloud).

 DiY vs Enterprise cloud consumption model

My own brew definition (partially adapted-stolen from some of Simon Wardley  blog posts , check out his great blog ):

• DiY : Complete redesign of an  infrastructure architecture, most of the time as close as possible to a true public cloud architecture ( or exact copy in the case of public cloud provider) and/or often to fit very specific needs. These solution   aim to have the lowest cost possible by reusing as much open source tools out there combine with custom solution.
• Enterprise: "It's like cloud but the virtual infrastructure has higher levels of resilience at a high cost when compared to using cheap components in a distributed fashion.  The core reason behind Enterprise Cloud is often to avoid any transitional costs of redesigning architecture i.e. it's principally about reducing disruption risks (including previous investment and political capital) by making the switch to a utility provider relatively painless." I.e. they want to be able to run "legacy" application alongside new one without having to throw away a lot of their existing investment ( HW - software - skillset - HR , etc.. ). These Enterprise cloud solution are sometimes delivered/consumed  as (heavily) custom packaged Openstack solution from specific vendor :  HP , mirantis, etc... These solutions are slowly making their way within the heavily regulated company segment ( compliance issues require on premise deployment).

DiY Openstack use internal resource and knowledge, in order to minimize TCO and maximize ROI they heavily rely on open source solution and adapt it to their need. This is why most of the time DiY are almost 100% home-grown solution.

On the other hand the Enterprise ones tend to be hybrid where you have a mix of opensource solution and "enterprise" solution. Sometimes this is also sprinkled with a dose of consulting. End consumer of these cloud are sometime  not aware that the solution they are buying is based on open stack ( ex:  in PaaS – Managed services scenario). By example Swisscom cloud managed services ( offer SAP -ERP - BW - etc..)  relies on Piston cloud who relies on Openstack. 

Often some  key aspect are being bought separately for their enterprise features ( or performance, support)  by ex: -storage : coraid (pNFS) / inktank (CEPH) , and networking : arista- .

Openstack user base is skewed:

Now, why the distinction is relevant to Openstack statistics: Cost savings  open technology , avoid vendor lock-in are the main drivers,  this should be hinting which one of the two is predominantly represented in the users surveyed => DiY.

As a result, the stats are quite skewed toward home grown solution where most of the cloud stack is build using open source solution almost exclusively. This is why CEPH or Ubuntu and other open source solution  heavily dominate the stats but they do NOT dominate  cloud spending ( this is reinforced with the fact that most of the surveyed user have a 1-50 nodes cloud ... ~60% + of the overall and only 20% of the clouds are used for production purpose  ). I would love to see a venn diagram of the different dimensions. If anybody has access to the raw survey data I would be happy to do the diagram for them

By example, for the cloud storage solution of some  Openstack  enterprise deployment in companies I came across  you would see all the usual suspect (EMC - Netapp - HP, etc..). However, they would be sometime relegated as secondary player role while a certain number of newcomers took the front stage. This newcomers obviously leverage the emerging market stage / first mover to grab market share : Inktank ( commercial CEPH version)  , Coraid ( pNFS ,  popular as it allow a natural transition from classical NFS setup ) , and finally the wild card :  Intel who is trying to push its Lustre solution ( from their whamcloud acquisition) into cloud storage . Note: I am not too aware on how glusterFS is fairing..

Openstack , a complex and rapidly evolving environment: 

As you can see Openstack ecosystem is heavily dominated by company using the DiY model.  This is further fuelled by the current state of Openstack which is more a toolkit that you need to (heavily) customize to build your own cloud compared to the self-contained solution like cloudstack, eucalyptus, flexiant, etc.. It makes me feel like openstack is the Debian equivalent for cloud.

However, its adoption is growing heavily in the corporate world (and as a result fragmentation risk too): Oracle has its own home-brew version (but does not contribute back anything and is completely closed) , SAP is starting some effort, Intel is a big proponent (see IDF 13), Big managed services player are using it as their basic building block  ( T-system, Swisscom , Huawei , IBM etc..) .

A tough ecosystem for startups and established company alike:

Complex Ecosystem, fragmentation, heavy reliance on custom solution makes it tricky for company to position themselves  within this environment. Too low and they might be completely ignored due to the heavy DiY proportion , leading to a long struggle ( or death) hoping that the crossing of the desert will come soon.Too high level and they end up fighting with a multitude of Openstack clone as well as with other cloud solution. 
There is no right decision there, maybe using a coherent layered approach across the openstack ecosystem would enable the creation of consistent revenue stream while limiting the race to the bottom competition ( competing on price). I probably will expand more on this concept in a follow up blog post.

 

PS: as I write this postI came across the following Gartner blog post that echo in some way my thought on Openstack ecosystem http://blogs.gartner.com/alessandro-perilli/what-i-saw-at-the-openstack-summit/ . Again this is to be taken with a grain of salt as Gartner has a long history of being a big VmWare supporter.