Architecture Overview of an Open Source Low TCO cloud storage system

I present here a possible solution for a low TCO open source cloud storage system, For those out there creating their own cloud (or hosting service).

I am not claiming that it will suits everyone  needs but at least i hope it will give you some valuable pointers and alternatives.
Also you might want to adapt it for your specific needs because you might not require every single feature of the system.


Summary:

This setup allows you to build your own redundant storage network with common PC hardware, easier but far more expensive way to achieve this would be to get a SAN and some fiber channel attached hosts. This setup provide similar feature as the one provided by Amazon ESB as well as a HR cluster file system for your cloud storage.

Features: 

• High availability (DRBD , cluster file system)
• High reliability ( DRBD)
• Flexible Dynamic storage resource managment 
• File system export or Block device " amazon ESB style"
• Dynamic fail over  configuration ( Pacemaker Corosync )
• Active / Passive ; N+1 ; N to N ; Split site

Overview:

• A set of paired storage back end composed of  hosts that use DRBD to keep the data redundant between each paired hosts.
• On top of DRBD we have LVM (or CLVM ) using LVM we can do on-the-fly logical partition resizing, snapshots ,including hosting snapshot+diffs,you can even resize a logical partition across multiple underlying DRBD  partition .
•  note: LVM can be used as a front end and backend of DRBD
• LVM block device  will be exported to the cluster nodes using GNBD. Another node makes a GNBD import and the block device appears to be a local block device there, ready to mount into the file hierarchy.
• OCFS2 as a cluster file system allow all cluster nodes to access this file system concurrently.
•  Another possibility is to export a GNBD device for each virtual machine (but you still need a distributed/ network file system for config etc..).
• Use of Pacemaker and Corosync to manage resources for  HA / HR 
• For the managment / control and monitoring part, a custom made solution might be needed. 
• GRAM could be used to expose the resource management
• Any monitoring framework should be able to do the trick

"Simple" Schema: 

Pro:

• Most of the independent parts are Proven solution used in large scale production environment
• Open source / readily available tools
• OCFS2 provide back-end storage for Image while GNBD can provide on demand block storage for the cloud instance
• Easy Accounting : as any other file system ( might need custom build tools thought depending of the needs/ requirement) 
• COTS components

Con :

• DRBD provide HA/HR through replication ( think RAID 1 ) which means you have  HA/ HR and speed at the expense of  half of your storage( slightly more if you are using raid for your actual disk storage)
• Complex / and risk of cascading failure due to dominoes effect ( similar to what happen in Amazon cloud recently with their ESB) 
• Performance will be extremely dependent of the set of physical resource available as well as the topology usage :
• It will require a lot of tweaking / customization to extract the best performance ( ex dual head for DRBD, load balancing etc.. ) and every setup will be different
• Creation of dedicated monitoring tools will be require in order to manage and automate the performance tweaking
• Require to create custom tool for management, scheduling / job management etc..( the hard bit)

Tools , Links / Pointers :

• DRBD: http://www.drbd.org/
  
• LVM or CLVM : http://goo.gl/WQHO /  http://sourceware.org/cluster/clvm/
• GNBD:  http://sourceware.org/cluster/gnbd/
• OCFS2 (you can use GFS if you prefer) :  http://www.oracle.com/us/technologies/linux/025995.htm
• For clustering automatic resource management and fail over ( take the hassle out of  the HA/HR management):
• Pacemaker: http://www.clusterlabs.org/
• Corosync : http://corosync.org
• GRAM http://www.globus.org/toolkit/docs/2.4/gram/ 
• Some link providing  a step in the right  direction.   However,  none of them provide the full range of features i presented in  the overview / schema , but it shouldn't be to hard for you to figure out how to get there ( if i have time i might post the actual how-to). 
• Ubuntu corosync/pacemaker/drbd8/ocfs2 : http://goo.gl/WWhTB 
• Xen with DRBD, GNBD and OCFS2  : http://xenamo.sourceforge.net/
• Using LVM + DRBD + NFS + Heartbeat + VTun To Gain Data Persistence, Redundancy, Automatic Fail-Over, and Read/Write Disk Access Across Multiple EC2 Nodes http://goo.gl/wVgQu
• DRBD for xen ( but lack of flexibility) : http://goo.gl/ap4d8
• KVM Cluster with DRBD/GFS http://goo.gl/MFDZ3
  

Voila!

How to install Soft-iWARP on Ubuntu 10.10, AKA how to have RDMA enabled system without the expensive hardware.

What is Soft-iWARP and why installing it : 

Soft-iWARP [1] is a software-based iWARP stack that runs at reasonable performance levels and seamlessly fits into the OFA RDMA environment provides several benefits:

• As a generic (RNIC-independent) iWARP device driver, it immediately enables RDMA services on all systems with conventional Ethernet adapters, which do not provide RDMA hardware support.
• Soft-iWARP can be an intermediate step when migrating applications and systems to RDMA APIs and OpenFabrics.
• Soft-iWARP can be a reasonable solution for client systems, allowing RNIC-equipped peers/servers to enjoy the full benefits of RDMA communication.
• Soft-iWARP seamlessly supports direct as well as asynchronous transmission with multiple outstanding work requests and RDMA operations.
• A software-based iWARP stack may flexibly employ any available hardware assists for performance-critical operations such as MPA CRC checksum calculation and direct data placement. The resulting performance levels may approach those of a fully offloaded iWARP stack.

How to install Soft-iWARP  :

When I tried to install Soft-iWARP   I ran into several issue and I am writing this installation manual so i don’t forget how to do it and other people can enjoy  a semi seamless installation.

My setup: 

•          Core 2 duo
•          e1000 nic
•          4 Gb of RAM
•          Ubuntu 10.10 server
•          Linux kernel : 2.6.35-28-generic

Pre-Install

Getting the Linux kernel dev environment:

• sudo apt-get install fakeroot build-essential crash kexec-tools makedumpfile kernel-wedge

• sudo apt-get build-dep linux

• sudo apt-get install git-core libncurses5 libncurses5-dev libelf-dev binutils-dev

Setting up  the Infiniband / libibverbs  environment

• sudo apt-get install libibverbs1 libibcm1 libibcm-dev  libibverbs-dev  libibcommon1 ibverbs-utils

• check if  “/etc/udev/rules.d/40-ib.rules” exist

1.       If the file doesn’t exist , create it and populate it with (you need to reboot after):

 ####  /etc/udev/rules.d/40-ib.rules  ####

 KERNEL=="umad*", NAME="infiniband/%k"

 KERNEL=="issm*", NAME="infiniband/%k"

 KERNEL=="ucm*", NAME="infiniband/%k", MODE="0666"

 KERNEL=="uverbs*", NAME="infiniband/%k", MODE="0666"

 KERNEL=="uat", NAME="infiniband/%k", MODE="0666"

 KERNEL=="ucma", NAME="infiniband/%k", MODE="0666"

 KERNEL=="rdma_cm", NAME="infiniband/%k", MODE="0666"

 ########

Verifying the Infiniband  / libibverbs  environment

·         Run these command in the following order:

1.  sudo modprobe  rdma_cm

2.  sudo modprobe ib_uverbs

3.  sudo modprobe rdma_ucm

4.  sudo lsmod  

5.ls /dev/infiniband/

    

Getting the compile / dev environment for Soft-iWARP   : 

• sudo apt-get install libtool autoconf         
  

Installing Soft-iWARP   :

Getting Soft-iWARP   

Getting the source from the git repository :

• You can get the source from :  https://github.com/zrlio/softiwarp

Compiling and installing the Soft-iWARP    kernel module : 

Inside the soft-iwarp kernel module folder:

1.          make
2.          sudo make install

Compiling and installing the Soft-iWARP   userlib: 

Inside the soft-iwarp userlib folder:

1.          ./autogen.sh
2.          Run AGAIN  ./autogen.sh
3.          ./configure
4.          make
5.          sudo make install

·         at that stage soft-iwrap install the infiniband stuff in “/usr/local/etc/libibverbs.d” we need to make asymbolic link t /etc/infiniband.d

o   sudo ln -s /usr/local/etc/libibverbs.d /etc//libibverbs.d

Inserting the  Soft-iWARP    kernel module:

1.  sudo modprobe  rdma_cm

2.  sudo modprobe ib_uverbs

3.  sudo modprobe rdma_ucm

4.  sudo    insmod  /lib/modules/"your_kernel"/extra/siw.ko

5.  sudo lsmod

§  check if all the modules are correctly installed

6.  ls /dev/infiniband/

Soft-iWARP    kernel  module parameters:

        loopback_enabled: if set, attaches siw also to the looback device.

                To be set only during module insertion.

        mpa_crc_enabled:  if set, the MPA CRC gets generated and checked

                both in tx and rx path (kills all throughput).

                To be set only during module insertion.

        zcopy_tx:         if set, payload of non signalled work requests

                (such as non signalled WRITE or SEND as well as all READ

                 responses) are transferred using the TCP sockets

                sendpage interface. This parameter can be switched on and

                off dynamically (echo 1 >> /sys/module/siw/parameters/zcopy_tx

                for enablement, 0 for disabling). System load benefits from

                using 0copy data transmission: a servers load of

                one 2.5 GHz CPU may drop from 55 percent down to 23 percent

                if serving 100KByte READ responses at 10GbE's line speed.

Verifying  the Soft-iWARP    is correctly installed :

running :

•          ibv_devinfo
•          ibv_devices

·         You should get :

·          

• if you get:

libibverbs: Warning: couldn't load driver 'siw': libsiw-rdmav2.so: cannot open shared object file: No such file or directory

libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0

o   you need to run “sudo  ldconfig” to update the library path

Running the simple test: 

Install rdma-utils :

• sudo apt-get install  rdmacm-utils

Then you can test the soft-iwarp with :

• rping : RDMA ping
• udaddy : Establishes a set of unreliable RDMA datagram communication paths between two nodes using the librdmacm, optionally transfers datagrams between the nodes, then tears down the communication.
• mckey : Establishes a set of RDMA multicast communication paths between nodes using the librdmacm, optionally transfers datagrams to receiving nodes, then tears down the communication.
• ucmatose  : Establishes a set of reliable RDMA connections between two nodes using the librdmacm, optionally transfers data between the nodes, then disconnects.

Note : the ibv_* utils don't work with iWARP device . you will get errors if you try to use them such as:

[root@iwarp1 ~]# ibv_rc_pingpong
Couldn't get local LID
[root@iwarp1 ~]# ibv_srq_pingpong
Couldn't create SRQ
[root@iwarp1 ~]# ibv_srq_pingpong
Couldn't create SRQ
[root@iwarp1 ~]# ibv_uc_pingpong
Couldn't create QP
[root@iwarp1 ~]# ibv_ud_pingpong
Couldn't create QP

Note 2 : If you only have one machine and you want to use the  loopback (optional you can still use your normal nic), to test if you have it enabled: 

The dark side of Green I.T.

To begin some definitions sustainability:

• “Meeting the needs of the present without compromising the ability of future generations to meet their own needs.” (EPA)
• "Sustainability in its simplest form describes a characteristic of a process or state that can be maintained at a certain level indefinitely."(Wikipedia)

 Quite often Green I.T. strategy (if you are not familiar with the concept check Green I.T for dummies by HP) is a key aspect of sustainable efforts of companies . However these efforts , while positive when we look at them at the micro level of the economy and the environment. They tend to have a negative impact at the macro level if they are not taken far enough. In order to make the problem more easily palpable i will use the all time favourite: virtualization.

Virtualization allow  the combination of  several physical systems into virtual ones onto a single, powerful system, thereby unplugging the original hardware hence reducing power and cooling consumption.

The benefit of virtualization is rather obvious  when you concider that a server produce  4 tons of carbon dioxide (CO2) annually. Hence, using virtualization reduce the carbon footprint and TCO per service provided. Does virtualization  constitute a sustainable strategy?

Sadly, most of the time, companies  forget that Green I.T. efforts impact go beyond data center, organizational and business efficiency.  If they look at the figures they will quickly realise that by reducing their ecological footprint per service or product they contribute to a reinforce the negative trends on our environment at a global scale.

Why ?  Simple , Economics.  The Key word here is "lower": lower TCO ,  Lower cost  to provide a service , lower the production cost, etc...  Green I.T. is  touted as  good for consumers, businesses, the economy overall, and the environment because it can make service more efficient and, by extension, less expensive. When a something becomes less expensive it automaticaly widen your potential customer base form the one you had before. Most of the time this  drive up the demand for your products, which in return requires you expand your operation in order to cope (good for business). The problem arise from the size of the additional demand.  As you can see in the following figure, the income repartition is "similar" to an inverse exponential function.   

 

 So, every time you lower the price of a certain product, the potential demand increase exponentially! Which means that if a company wants to stay  carbon neutral while coping with the increased demand, this company needs to make exponential efforts to "green" its operation. At that moment the law of diminishing returns kicks in ( see figure below),  the ever increasing  cost of  the greening  operation start to hurt the company profits margin, which leads to its discontinuation.

To be sustainable a company has to understand that it needs become first efficient and then  transition to  become sufficient. However , companies quite often stop at being efficient, claiming to be sustainable and dismiss any further efforts as being unnecessary and counter-productive. By doing so, they are "sustainable washing" themselves.

Worse, these limited Green I.T. Environmental improvements at a micro level (understand company) increase  the rate of degradation at the macro level (planet). Because the rewards for an efficiency gain is substantial surpluses, enough boost economic prosperity of the company and, most significantly, enough surplus to allow further investment leading to yet more surplus. 

Not to mention that all this is also exacerbated by I.T. becoming a utility ( understand cloud).

The Efficiency strategy through green I.T. is often destructive in practice. To become truly sustainable , companies and the society will need to accept that  economic  growth is no longer possible.

 

The network performance within the cloud, an hidden enemy

A lot of people talked about the latency issue when hosting services in the cloud . Recently amazon latency hiccup revealed a deeper problem, but seems to be rarely discussed. While most focus on the network access and consume services from the cloud. I realise that their is a big unknown concerning network performance inside the cloud.

Could provider  don't disclose their real infrastructure underlying their cloud offers. By doing so, cloud customers are completly left in the dark regarding the network linking their different instances. Leaving them with the false warm feeling that their are on top their own flat network.

What does it mean:

• You have no idea of  your  network or I/O performance for your instance. Your virtual interface is sharing a  physical (sometimes trunked) one(s) with  other tenants collocated on the same physical server and theycompete with you for a share of the network pipe.
  
• You have no idea of your network performance  between multiple instances within the same cloud:
• First your instances can be located in different branch of the infrastructure. Which means more network gears between them.
• Then, Virtualizated  network gears can also be thrown into the mix. Which add virtual switches and routers with sub optimal performance (remember they are software) but add greater flexibility.
• Finally, the network traffic generated by all the tenants makes it very difficult (and expensive) to guaranty QoS throughout the infrastructure. Not to mention that capacity planning , measurement and management becomes extremely difficult because it is impossible to predict  the(often asymmetric) bandwidth  network consumption of the instance.  A reason why cloud providers dream for hugely dense, multi-terabit, wire speed L2 switching fabrics.

As a consequence, there is not generally a published service level associated with throughput and latency  within cloud.  When oversubscription hit you, you often don't see it coming.  Maybe cloud will become similar to the home broadband  with  advertised "unilimited" offers but with content ratio.

All this, makes it extremely difficult to deploy  and guaranty the performance of  services that rely on low latency and/or high bandwidth architectures such as high performance computing, web and database clusters, storage access, seismic analysis, large scale data analytics, financial services and algorithmic trading platform.

I can think of  some solutions to these problems but this will be for another post.

The 5 Reasons You're Failing at innovating

1. You really don't want to engage directly with customers, employees, etc. You just want them to hear how innovative you are. Ex: You end up following trends rather than creating them.  However, you are quick at claiming  ownership.
2. You perceive innovation as a threat to your historical revenue streams. However, you still pay lip service out of fear.
3. You claim to have a strong in house innovation culture. However, you make somebody else pay for it (public funding) or absorbe (and dissolve ) it through acquisition .
4. You get locked into thinking of ideas at just the higher end of the innovation continuum (next big thing) as a result your innovation program is unbalanced and miss out the small quick wins that are faster and easier to obtain. ( and  vice versa)
   
5. Your risk management strategy for Innovation confuse risk aversion and educated risk taking.  Your fear of failure motivate your strategy to minimize the level of uncertainty, however it sclerose you to the point of inaction. 

After follow the moon, Avoid the law

After the follow the moon green trend for cloud computing. Companies are finally catching up to the potential of cloud computing to avoid constraining law and tax systems: like this company from india .

Companies will soon realise that they can leverage cloud computing the same way they use off shore accounts and tax heavens. Not only they will be able to dodge  "optimize"  taxes but also they will be able to avoid limitating law systems by moving the data or the processing of data in a less restrictive location.
Currently, governments force companies to maintain their data on the  country soil in order to control its use through legal means. However, it is really easy to move compute loads with the cloud to where the legal and regulatory environment is more favourable, while  leaving  the  data where it is. 

How fast the law will catch up? Not fast enough. And the problem gets worse when you think that the law system will need to be harmonised at a planetary scale to of any use. I already foresee country creating data and processing taxes heaven law in order to attract cloud providers and their customers. The same way Ireland slashed its corporate taxe to attract company on its soil.

By allowing  to access, store and process data  in a  fast,  seamless and transparent way, the cloud is creating a legal void that companies will exploit in order to maximise their profits and minimize their exposure. How long will we have to wait for cloud providers to advertise their new features :  legal cloud  location zone? Already been done :

According to Microsoft, the geolocation feature is also necessary for legal reasons since many Azure users apparently have “requirements on where they can place their code and data and where they cannot.” 

I think their is a burgeoning  market for IT consultants and lawyers in legal IT optimization  strategy.

IT Trading systems and Cloud take one

Investment banks, insurance companies, and hedge fund firms are running HPC applications to keep their financial services running smoothly. More specifically Algorithmic trading require a huge amount of processing power as well as fast network capabilities ( speed is money ).

I recently came across this company :  Marketcetera. I think we will soon see the emergence of PaaS , SaaS and IaaS  company   specifically dedicated for algo trading solution or financial computing.

The cost of creating testing and deploying such trading platform as well as testing new trading algorithm is rather prohibitive nowadays.  With the advent of cloud computing,  it becomes possible in the near future that companies will start offering at a reasonable price ( and  even at dynamic price) resource specifically dedicated and collocated for financial market operations.

However financial trading applications  have requirements that are very different from the classical web one that are  run on the cloud. I will just  expose some of the one affecting IaaS and PaaS:

• IaaS , Performance is key: 
  
• Network: 
• Speed and location: Financial trading services have high bandwidth and low latency requirements. To satisfy such stringent requirements IaaS provider will have to be as close as possible to the stock exchange. They will need to have cloud's location s to be physically close or better collocated with  the exchange systems (a la ec2 zone). Except that in this case the actual location will matter less than the actual proximity physical proximity of a certain exchange (especially for unfair high frequency trading) .
• Virtual Networking:

• Hardware FPGA and GPGPU, ASIC :  Hardware acceleration can easily boost the performance of operations by a degree of magnitude. It has been successfully used by financial institution to do such thing as  XML processing, network routing, algorithmic trading , etc..  In the race to be the fastest, such piece of hardware can give an significant edge. However, future financial cloud providers will need to find a way to easily expose such piece of hardware. Creating a pool of hardware resource accessible through  I/O virtualization seems a potential solution (by using Hypertransport , Quickpath interconnect or PCIe over ethernet ).

•  Virtualization vs BareBone: Virtualization always comes at a price. You lose performance and I/O speed. But, contrary to the popular belief, cloud infrastructure does not preclude the use of non virtualized resources. Cloud Providers can easily provide both: virtual  and dedicated hardware resource  within a same cloud in order to satisfy the  various demands of its customers.
• PaaS , how to balance performance , flexibility and accessibility: 
• Langage:  C and C++  still represent a huge portion fo the core of financial apps framework. You can even see bits ASM !. However current PaaS languages of predilection are .Net, Java, Phyton or Ruby on Rails. These are often slower but much more easier  (and easy to secure ) to use to create a platform cloud computing engine. One way to go around this problem would be to design a custom language for trading algorithms. Or allow easy integration of external processing component. Which leads to the next aspect.
  
• Messaging and order routing : A similar problem arise  for messaging today cloud messaging  APIs are based on public REST, XML, and SOAP standards. While current trading platforms prefer the fast but proprietary (hence expensive ) ESBs or messaging platforms. It make it rather expensive to integrate cheaply with external component. Maybe ,if the PaaS vendor is also a IaaS  (servers ) and a SaaS (the ESB), the integration can be done or provided in a cost effective way.
  

• Security : challenges are multiple:
• VM security
• network
• Data (secure storage)
  
• Tracability
• Audit
• Authentication
• Non-repudiation
• Integration with third parties and customer owned solution
  
• Etc..
  
• Reliability and high availability are and order of magnitude higher than typical cloud apps.
  

This list is not extensive and I definitely know I have  missed some aspects. But i will try to dwelve deeper in this problem that constitute trading in the cloud. And, while providing cloud for customers wanting to run financial apps is relatively more difficult than "traditional cloud offer". The benefits of such offer can  be worth the efforts:

• Sharing exchange access point cost.
  
• Lowering the cost of cheating entering in High frequency game.
• Eliminating the cost of creating a trading platform.
  
• Lowering the TCO of trading platform.
• Lowering the cost of testing and validating trading algorithm. 
  
• Etc...