[Links of the Day] 04/12/2018 : Matrix cookbook, Serverless Containers, C++ network coroutine lib

• The Matrix Cookbook : nothing about Keanu Reeves in a kitchen apron but a collection of facts about matrices and matters relating to them. A very well documented desktop reference.
• Firecracker : Secure and fast microVMs for serverless computing... This is really another AWS product that makes you ask why do we need Kubernete. To be honest I really see K8s now as another OpenStack. It's at the top of the hype cycle but I don't really see it going anywhere further anytime soon. This type of tech enables serverless containers. Making any of the k8s almost a moot point.
• Coro async : C++ coroutine-based networking library

A look at Google gVisor OCI runtime

Google released a new OCI container runtime: gVisor. This runtime aim at solving part of the security concerns associated with the current container technology stack. One of the big argument of the virtualisation crowd has always been that the lack of explicit partitioning and protection of resource can facilitate “leakage” from the containers to the host or other adjacent containers. 

This stem from the historical evolution of containers in Linux. Linux has no such concept of native containers. Not like BSD with Jails or Solaris with Zones. Containers in Linux are the result of the gradual emergence of a stack of various security and isolation technology that was introduced in the Linux kernel. As a result, Linux ended up with a highly broad technology stack that can be separately turned on / off or tuned. However, there is no such thing as a pure sandbox solution. The classic jack of all trade curse, it’s a master of none solution.

The Docker runtime (Containerd) package all the Linux kernel isolation and security stack (namespaces, cgroup, capabilities, seccomp, apparmor, SELinux) into a neat solution that is easy to deploy and use. 

It allows the user to restrict what the application can do such as which file it can access with which permission or limits resource consumption such as networks, disk I/O or CPU. It allows the applications to share resources without stepping on each other toes happily. Also, limits the risk any of their data being accessed while sitting on the same machine. 

With a correct configuration ( the default one is quite reasonable ) will allow blocking anything that is not authorised and in principle protect from any leak from a malicious or badly coded piece of code running in the container. 

However, you have to understand that Docker has some real limitation already. It has only limited support for user-namespace. The user-namespace allows applications to have UID 0 permission within the containers ( aka root ) while the containers and user running has a lower privilege level. As a result, each container would run under a different user ID without stepping on each other toes. 

All of these features rely on the reliability and security (as in no bugs) of the Linux kernel. Most of Docker advanced feature relies on kernel features. And getting new features is a multi-year effort, it took a while for good resource isolation mechanism percolates from the first RFC to the stable branch by example. As a result, Docker and current container ecosystem are directly dependent on the Linux kernel update inertia as well as its code quality. While being excellent, no system is entirely free of bug, not to mention the eternal race for patching them when they are discovered and fixed. 

Hence the idea is to, rather than having to share the kernel between all the containers, have one kernel per container. Explicitly limiting potential leakage, interference and reduce the attack surface. gVisor adopts this approach, which is not new as KataContainers already implemented something similar. Katacontainers is the result of the fusion of ClearContainer (intel) and runV (hyper). Katacontainers use KVM as a minimalistic kernel dedicated to the container runtime. But, you still need to manage the host machine to ensure fair resource sharing and their securitisation. This additional layer of indirection limits the attack surface as even if a kernel bug is discovered you will be challenged to exploit it to escape to another an adjacent container or underlying one as they are not shared. 

gVisor can use a KVM as kernel; however, it was initially and is still primarily designed around ptrace. User Mode Linux already used the same technique, which is to start a process in userspace for the subsystem that will be running on top. Similarly to a hypervisor model used by virtual machines. All the system calls will be executed using the permission of the userspace process on behalf of the subsystem via an interception mechanism. 

Now, how do you intercept these system calls which should be executed by the kernel? UML and gVisor divert ptrace primary goal ( which is debugging ) and inject a breakpoint in the executable code to intercept and stop for every system call execution. Once caught the new userspace kernel will execute the call on behalf the original process within userspace. It works well, but as you guessed, there is no free lunch. This method what heavily used by the first virtualisation solution. But rapidly, processor vendors realised that offering hardware-specific acceleration method would be highly beneficial ( and sell more at the same time).

KVM and other hypervisor leverage such accelerator. Now you even have AWS and Azure deploying completely dedicated coprocessor for handling virtualization related acceleration. Allowing VM to run almost that the same speed as a bare metal system. 

And like Qemu leveraging KVM, gVisor also offer KVM as underlying runtime environment. However, there is significant work to be done to enable any container to run on to of it. While ptrace allow to directly leverage existing Linux stack, with KVM you need need to reimplement a good chunk of the system to make it work. Have a look at Qemu code to understand the complexity of the task. This is the reason behind the limited set of supported applications as not all syscalls are implemented yet.

As is, gVisor is probably not ready yet for production. However, this looks like a promising solution providing a middle ground between the Docker approach and the Virtualization one while taking some of the excellent ideas coming from the unikernel world. I hope that this technology gets picked up, and the KVM runtime becomes the default solution for gVisor. It will allow the system to benefit from a rock-solid hardware acceleration with all the paravirtualisation goodies such as virtio. 

[Links of the day] 16/02/2016: VM density, VLDB15, Cassandra MQ, Eth roadmap

• VM density : a look at VM density and utilization profile in Pernixdata cloud. Seems that the customers preference goes for 16 core dual socket virtual system while hogging as much memory as possible.
• VLDB 2015 : Full VLDB 2015 program with papers attached. Notable Paper : Gobblin by linkedin crowd and Coordination Avoidance in Database Systems from Berkeley 
• Cassieq : Distributed queue built on top of Cassandra

Bonus: Ethernet roadmap

[Links of the day] 02/02/2016: Fosdem 2016

Fosdem 2016 just finished a couple of day ago, here are some selected bits:

• Data Replication Track : a lot of interest into clustering Mysql like DB this year: 
• Mysql Group Replication : Mysql native clustering with paxos protocol 
• Galera Replication demystified : While Galera does not provide snapshot isolation, it might still save your bacon
• Reliable Cluster Detection and Failover : how to reduce the hot switch time drastically
• Rearchitecting Linux I/O towards Petascale Storage : 
• RocksDB Storage Engine for MySQL : Facebook replace the underlying storage engine with a K/V DB
• Containers and Virtualization : some interesting use cases and thought on how to get the best of each worlds: Containerization of hypervisor and running opentsack service in containers
• From pets to cattle to flock of birds : lambda all the things 

Las but not least here are all the videos.

[Links of the day] 18/01/2016 : Post quantum crypto, ARM virtualization, Scalable C

• Post-quantum key agreement : First, 99% of people out their do not use encryption correctly, so you should not be worried about post quantum crypto because you are not protecting yourself in the pre-quantum era. Now if you are part of the 1%, bad news you key size just jump from a couple of Kbytes to Mbytes (and maybe GB)... Very good read explaining the challenges ahead and existing gap of crypto solution in the upcoming post quantum era.
• ARM virtualization extensions : In depth look at the ARM virtualization feature. Maybe we can see a glimpse of what can be done with the AMD ARM server push. 
• Scalable C : book on how to make C scalable by the founder of ZeroMQ. Some good things, some bad one, a lot of grief toward C++. I personally love the C language but sometimes pitting one language against another doesn't help without context. Pick the tools that suits best and sometimes yes it means picking the one that makes collaboration efficient rather than make the code efficient. 

Links of the day 16/11/2015: Named Data networking, VM scheduler, Asymmetric multi-core processors.

• Named Data : an effort to transform networking the same way object storage transformed well storage. By naming the data content for efficient transport rather than solely focusing on the point to point aspect. Really cool concept.
• BtrPlace : Virtual machine scheduler system. While really nice it suffer from a lack of integration with existing system. However from looking at the code it would be rather simple to integrate with Openstack or other cloud / virtualization system.
• Survey on Asymmetric Multicore Processors : extensive review of the literature out there on heterogeneous processors systems. 

Links of the day 10/11/2015: #containers, #Unikernel & #virtualization power consumption, security, and streamlined

• Qubes with Rumprun : Interesting application of unikernel to deliver greater security at speed. the goal is to reduce the possible attack surface while maximizing performance.
• Power Consumption of Virtualization Technologies: empirical Investigation that confirm a lot of what we knew already, virtualization power consumption is higher than container.
• Vagga : containerisation tool without daemons. Slim down streamline version of container. No deamons or other process to run to support it. Looks like a solid contender to docker and other vagrant.

Links of the day 08 - 07 - 2015

Links of the day 08/07/2015: Storage tipping point, SSL, reverse proxy, #Container with the virtualization goodies on top ( or under )

• The Storage Tipping Point -- the performance optimization technologies of the last decade – log structured file systems, coalesced writes, out-of-place updates and, soon, byte-addressable NVRAM – are conflicting with similar-but-different techniques used in SSDs and arrays. The software we use is written for dumb storage; we're getting smart storage; but smart+smart = fragmentation, write amplification, and over-consumption.
• s2n -- Amazon's open source ssl implementation.
• pushpin -- a reverse proxy server that makes it easy to implement WebSocket, HTTP streaming, and HTTP long-polling services. It communicates with backend web applications using regular, short-lived HTTP requests (GRIP protocol). This allows backend applications to be written in any language and use any webserver.
• Hyper : Make VM run like Container. Fast as Container, Isolated by VM. Intersting approach, Upon the RUN command, Hyper launches a new VM instance, instead of containers, and mount the specified image onto the instance. Inside to the VM, a minimalist Linux kernel, called HyperKernel, is booted. HyperKernel is built with a tiny Init service, called HyperStart, which creates a Pod, setup Mountnamespace, and launch apps from the loaded image.

Links of the day 08 - 06 - 2015

Today's links 08/06/2015: #SSD no panacea for VMs, Hardware X86 #security breach, #Bigdata stream engine

• SSD no panacea with VM : because of is operational model SSD tend to have poor performance in virtual environment when shared across multiple VM. The authors suggest to isolate and dedicate SSD for each VM in order to benefit from the speed benefit of SSD. Micro storage solution anyone ?
• X86 design flaw : Apparently there is a way to escalate from ring 0 to the high privileged state. This can be really interesting if this would enable VM or container to escape and gain access to the bare-metal system. : :
• Heron : the descendant of Storm by Twitter. The architecture seems nice and try to address some of Storm shortcoming.  

Links of the day 28 - 04 - 2015

Today's links 28/04/2015: #Rump Kernel Stack, Disque Distributed In Memory MQ, #FusionIO new PCIe product, Power level estimation of VM systems

• Ramp Stack : Nginx, MySQL, and PHP built on Rump Kernels without rearchitecting the application. Most of the work requires the app to be cross compile correctly (Nginx & MySQL). This implies that Unikernel-compatible unmodified POSIX C and C++ applications “just work” on top of Rump Kernels, provided that they can be cross- compiled.
• Disque :a distributed, in memory, message broker by Redis folk. Not production ready but a promising start.
• Pcie Flash : fusion IO is still kicking and deliver an interesting solution: up to 350,000 I/O operations per second (IOPS) on random reads and 385,000 IOPS on random writes (on the 3.2 TB model) with a 15k nanosecond write latency and 2.8 GB/sec of read bandwidth. .. However I still don't get why they don't want to use NVMe tech
• Process-level Power Estimation in VM-based Systems : the authors describe a fine-grained monitoring middleware providing real-time and accurate power estimation of software processes running at any level of virtualization in a system. 

Links of the day 13 - 03 - 2015

Today's links 13/03/2015: machine learning , Time-Series and Geospatial database, Write acceleration for virtual environment

• Clustered Fault Tolerant Write Acceleration in a Virtualized Environment : PernixData present a fault tolerant host-side flash write acceleration layer that seamlessly integrates with the virtualized environment while preserving dynamic resource management and high availability
• Machine Learning for Programming : Peter Norvig keynotes on using machine learning techniques to solve more general software problems, helping both the advanced programmer and the novice one.
• Time-Series Database Requirements : the author present its own ideas about what constitutes a good time-series database
• Why Are Geospatial Databases So Hard To Build? : the root of the problem is that the internals of a database engine designed for traditional “text and numbers” data models, which is virtually all of them, has very little in common with a database engine designed for real-time spatial data models.

Links of the day 12 - 03 - 2015

Today's links 12/03/2015: cryptocurrency landscape, vsphere memory state, hyperscale datacenter system, rowhammer bit flip

• Bitcoin Landscape : database and categorization of crypto based company [ database here ]
• Vsphere memory state : What technique vmware use to increase memory density depending of the system memory state . Bonus : memory management performance in vsphere 5.0
• HDS 8000 : Ericsson HDS 8000 (Hyperscale Datacenter System), built on Intel® Rack Scale Architecture. It uses optical interconnect. Combining a disaggregated hardware architecture with optical interconnect removes the traditional distance and capacity limitations of electrical connections.
• Project Zero - row hammer : how flipping bit in memory can allow you to gain root access to a physical machine. Basically by aggressively flipping 2 bits you can trigger the third one to switch. ECC can help prevent that to a certain extent but is not completely immune.

Links of the day 22 - 01 - 2015

Today's links 22/01/2015: NUMA & VMs,#openstack & messaging, continuous deliver

• Estimating memory locality for virtual machines on NUMA systems : using sampling techniques to 
• Scaling openstack - messaging : scaling is hard , especially when your messaging stack is already showing sign of weakness.
• Continuous Delivery at Scale: how CD evolve at Linkedin 

Links of the day 13 - 01 - 2015

Today's links 13/01/2015: #Rust Os, #virtualization Field day 4, Cpu history, Architecture design duality

• Rust OS : resources and documentation about operating system development with Rust. 
• Virtualization field day 4 : live streaming start on the 14th of January. You can have a preview of the companies presenting here. 
• What's New in CPUs Since the 80s : A very good overview of the evolution of CPUs ( x86 mainly) and how it affect programmers. 
• On the duality of operating system structures : why the current debate on microservices vs the rest, thread/locks vs events is moot and both views are equals / duals

M3 Duality by Jason-C on DeviantArt

Links of the day 10 - 12 - 2014

Today's links 10/12/2014: #openstack , VM placement , #Cloud , #HPC and transactional app delivery and update from #Ubuntu

• Snappy : Ubuntu Core, with a new mechanism for transactional app delivery and system updates ( "snappy" ). This is in direct competition with rpm-ostree from project atomic or NixOS
• Ironic + Crowbar for openstack : another step toward making bare metal provissioning as easy as provisioning VM
• Job packing in warehouse-scale computing : more VMs or jobs in more space .
• HPC aware VM placement : VMs packing for HPC use case

Links of the day 08 - 10 - 2014

Today's links 08/10/2014: Power 8 , NUMA, VM flash cloning , state of bitcoin

• Project Fargo / VMfork : flash cloning solution from VMWare, allow the very fast cloning of VM ( second vs minutes). VMware needs to push that if it wants to keep its hypervisor relevant compared to docker. 
• IBM Power 8 server : Impressive beast with proprietary NUMA interconnect enabling to reach up to 192 core and 16 TB of RAM by linking 4 nodes together. 
• State of Bitcoin Q3 2014 : adoption , usage, ATMs, investment is up but market cap is down ( probably due to the recent massive sell off ) 

Links of the day 10 - 09 - 2014

Today's links: #docker , #unikernel , #jit , #virtualization , #IDF14 , Business Model

• Just-In-Time Summoning of Unikernels : spawning CloudOs ( #OSv, #MirageOS , ..) on demand via forwarding DNS. Interesting concept but probably limited by how fast you can spawn these minions. 
• Freemium for Enterprise software : Should start-up and established company use the gaming industry freemium model? Not sure.. but interesting perspective. 
• Bare-metal, #Docker Containers, and #Virtualization: The Growing Choices for Cloud apps #IDF14 presentation which discuss discusses bare metal, containers, and virtualization as different options for running workloads with an interesting comments by Scott Lowe on its blog

Links of the day : 09 - 09 - 2014

Today's links : #Docker, #bittrorrent, #xeon, #intel, #CI

• Lastest Xeon: new virtualization technologies on the latest Intel® Xeon 
• Scaling CI : how Etsy scale continuous integration 
• Persistent Distributed Filesystems in #Docker : using bitTorrent to sync content within the docker image.

Hecatonchire Version 0.2 Released!

Version 0.2 of Hecatonchire has been released.

What's New:

1. Write Invalidate coherency model added for those who want to use Heca natively in their application as Distributed Shared Memory( more on that in a subsequent post)
2. Significant improvement in performance of page transfer as well as a numbres of bugs squashed.
3. Specific Optimisation for KVM.
4. Scale out memory mirroring
5. Hybrid Post copy live migration
6. Moved to linux Kernel 3.9 Stable
7. Moved to Qemu-kvm 1.4 stable
8. Added Test / Proof of concept tools ( specifically for the new coherency model)
9. Improved Documentation

Voila!

We are now focusing on Stabilizing the code as well as robustness ( we aim at making the code production ready by 0.4) . Also, we are starting significant work to transparently integrate Hecatonchire so it can be transparently leverage via a cloud stack and more specifically openstack.

You can download it here : http://hecatonchire.com/#download.html
You can see the install doc here: https://github.com/hecatonchire/heca-misc/tree/heca-0.2/docs
And finally the changelog  there : http://hecatonchire.com/#changelog-0.2.html
Or you can just pull the Master branch on github:  https://github.com/hecatonchire


Stay tuned for more in depth blog post on Hecatonchire.

KVM Post Copy Live Migration with Kernel RDMA transport

Overview:

Live Migration move running virtual machines from one physical server to another with no impact to end users. It allows you to  keeps your IT environment up and running, giving you unprecedented flexibility and availability to meet the increasing demands of your business and end users.

• Reduce IT costs and improve flexibility with server consolidation
• Decrease downtime and improve reliability with business continuity and disaster recovery
• Increase energy efficiency by running fewer servers and dynamically powering down unused servers with our green IT solutions

However, limitations of the current migration technology start to appear when they are applied on larger application systems such as SAP ERP or SAP ByDesign. Such systems consume a large amount of memory and cannot be transferred as seamlessly as smaller ones, creating service interruption. Limiting the impact and optimising migration becomes even more important with the generalisation of Service Level Agreement (SLA). This strand of research within the hecatonchire project aim at improving the live migration of VMs running large enterprise applications without severely disrupting their live services, even across the Internet.

How it works : 

Full post copy live migration

1. Stop the VM at the beginning
2. Sending all the CPU and device states to the destination including the memory
3. Send the RAM information and unmap the whole RAM memory region on Host B for RDMA connection 
4. .Immediately start KVM on Host B
5.  Host B will start page faulting and pull the page from Host A on demand ( + background prefetching)

Pre copy Vs Post copy live migration

Hybrid  post copy live migration 

Hybrid Post copy live migration provide a middle ground between the full post copy and the pre-copy approach. It limit the impact of the page faulting by enabling the pre copy phase while providing a deterministic with reduce performance impact during the overall execution of the live migration.

Hybrid Live migration

Architecture: 

Post copy live migration

  

If the VM touches a not-yet-transferred memory page, the VM page fault and initialise a memory request over RDMA using an In-kernel RDMA engine. This engine will  copy the content of the memory page from the source and resolve the page fault.

Prototype  / Demo :

We present the design, implementation, and evaluation of post-copy based live migration for virtual machines (VMs) across a Gigabit LAN. Post-copy migration defers the transfer of a VM's memory contents until after its processor state has been sent to the target host. This deferral is in contrast to the traditional pre-copy approach, which first copies the memory state over multiple iterations followed by a final transfer of the processor state. The post-copy strategy can provide a "win-win" by reducing total migration time while maintaining the liveness of the VM during migration.

The follwing Video demonstrate three different  post copy live migration scenario

1.  Full post copy
2.  Hybrid : 10 second timeout before switching to full post copy from standard live migration
3.  Hybrid 60 second time out : standard live migration finish  within allocated time ( however we don’t follow the standard process as there is no stop and copy , just stop  , send over cpu status and restart, missing page will be fetched on demand or by the background thread).

In comparison with the traditional approach we  demonstrated that post-copy improves several metrics including pages transferred, total migration time, and network overhead. it also provide a deterministic live migration features which is missing with traditional approach as the system administrator has no control of workload placement and transfer.





Comparison between Yabusame and RDMA kernel approach : 

Yabusame  rely on a special character device driver allows transparent memory page retrievals
from a source host for the running VM at the destination. however as shown in the diagram above this require a lot of communication between different part as well as context switching which tend to be less than optimal. With the aproach we are proposing we are able to eliminate most of the overhead associated with memory transfer while improving overall performance.
Also with  Yabusame the VM touches a not-yet-transferred memory page, it  pause the VM temporarily. while with our approach we make full usage fo the asynchronous page fault system allowing us to avoid as much as possible to pause the system.

Future Work : Flash Cloning :

Virtual Machine (VM) fork is a new cloud computing abstraction that instantaneously clones a VM into multiple replicas running on different hosts. All replicas share the sa e initial state, matching the intuitive semantics of statefull worker creation. VM fork thus enables the straightforward creation and efficient deployment of many tasks demanding swift instantiation of stateful workers in a cloud environment, e.g. excess load handling, opportunistic job placement, or parallel computing.
Lack of instantaneous stateful cloning forces users of cloud computing into ad hoc prac tices to manage application state and cycle provisioning. As a result, we aim to provides sub-second VM cloning, scales to hundreds of workers, consumes few cloud I/O resources, with negligible runtime overhead.