Thursday, January 31, 2013

Project Hecatonchire is LIVE !

I Finally got the green light from SAP to open source the project I started a little bit more than a year ago. What is Hecatonchire about ?

The context 

The cloud computing paradigm is fundamentally changing the expectations for how computing, storage, and networking resources get consumed. As these resources become fungible rather than self-managed, end users and service developers expect resources to be available on demand in response to their real time requirements. However, existing technologies to facilitate cloud infrastructures are inadequate, expensive and does not scale. Over time, server-centric infrastructure management systems have evolved to become a powerful but complex mesh of layered system functionalities that are both knowledge and labour intensive to setup, maintain, and most importantly, adapt to a constantly changing set of customer requirements.

 The Problem

As a result, renting a fixed combination of cloud resources does not and cannot fulfil the demands of cloud users for multiple reasons. One one hand, as the number of cores and amount of memory on servers continues to increase (over the next few years, we are expect to see servers with hundreds of cores and hundreds of gigabytes of memory per server commonly used), leasing an entire server equivalent may be too large for many customer’s needs with resources wasted. On the other hand, with the emergence of a broad class of high-end applications for analytics, data-mining, etc., the available amounts of memory and compute power on a single server may be insufficient. Moreover, leasing cloud infrastructure resources in a fixed combination of CPU, memory, etc. is only efficient when the customer load requirements are both known in advance and remains constant over time. As neither of these conditions are met for a majority of customers, the ability to dynamically mix-and-match different amounts of compute, memory, and I/O ressources is highly valuable.

Project Hecatonchire

Hecatonchire aim to  address these gaps by developing key enabling technologies that allow virtualization to go beyond the boundaries of the underlying hardware, breaking the barrier of using a single physical server for a virtual machine. With Hecatonchire, compute, memory, and I/O resources will be provisioned across multiple hosts and can also be consumed in dynamically changing quantities instead of in fixed bundles. This will effectively enable a fluid transformation of cloud infrastructures from commodity sized physical nodes to very large virtual machines to meet the rapidly growing demand for cloud-based services.

What can you do with Hecatonchire so far: 

  1. Application Memory extension : With Hecatonchire you can extend the reach of an applications to remote memory. effectively delivering a  scale-out solutions for the applications by leveraging LibHeca  APIs.
  2. Virtual Machine Memory pooling  : we have a modified  KVM  version  that supply reliable memory scale-out solutions that do not require changes in existing applications. Vm can simply tap into remote memory and also tier it ( to remote ram , remote compressed ram or flash, we also support memory deduplication)
  3. Flash Live migration : Consistent and high performance post copy live migration regardless of the workload and memory footprint ( we tested with TB+ while running in memory db under load)

Coming soon:

  1. Flash Cloning :   instantaneously clones a VM into multiple replicas running on different hosts.
  2. Applications and Virtual Machine Shared Memory :  Posix shared memory but across multiple node
  3. Memory Cloud : Complete integration of all the feature within one cloud platform (openstack integration). We aim eliminates memory segmentation by creating a pool of shared memory for the entire cluster. Hecatonchire will combines memory from dedicated appliances with memory from existing servers to create a large memory cache which can be transparently shared among any or all servers or data centers in a cluster. It provides a shared network memory resource large enough to accommodate huge data sets, significantly reducing storage access and improving processing performance by orders of magnitude. Basically you will be able to manipulate memory the same way you manipulate storage (I.e. : Tiering RAM ->Remote Ram-> Compressed Ram->Flash , Deduplication , etc..)

Further away: 

  1.  Compute Cloud: enable the aggregation of CPU resources across multiple COTS nodes providing VMs with a large number of vCPUs
  2. I/O Cloud: break the boundaries of the hypervisor, off-loading the I/O virtualization logic and aggregating all the I/O resources distributed across multiple servers to a dedicated server.

I will discuss in subsequent posts the different features , performance and roadmap of Hecatonchire. Stay Tuned !