Links of the day 10/08/2015 : Cryptocurrency architecture, Distributed data stream processing and Rump kernels

• Architecture Of A Cryptocurrency : good overview of what make a cryptocurrency and the five pillars of its architecture : Network consensus, transaction Protocol and internal state
• High-throughput, low-latency, and exactly-once stream processing with Apache Flink : on the evolution of fault-tolerant streaming architectures and their performance
• On rump kernels and the Rumprun unikernel : a very good overview of the rump kernel . As stated, One of the key advantage of this type of unikernel is that it can run unmodified "legacy" applications. By that I means you can use your tested, running application that wasn't design for unikernel and slap it on top of rump. To a certain extend it allow a easy and smooth transition without the pain of re-coding everything from scratch. 

Links of the day 08 - 06 - 2015

Today's links 08/06/2015: #SSD no panacea for VMs, Hardware X86 #security breach, #Bigdata stream engine

• SSD no panacea with VM : because of is operational model SSD tend to have poor performance in virtual environment when shared across multiple VM. The authors suggest to isolate and dedicate SSD for each VM in order to benefit from the speed benefit of SSD. Micro storage solution anyone ?
• X86 design flaw : Apparently there is a way to escalate from ring 0 to the high privileged state. This can be really interesting if this would enable VM or container to escape and gain access to the bare-metal system. : :
• Heron : the descendant of Storm by Twitter. The architecture seems nice and try to address some of Storm shortcoming.  

Links of the day 09 - 03 - 2015

After a brief hiatus ( still enjoying the snow in Chamonix until the 15th ) here is Todays' links 09/03/2015: real time stream engines, high performance networking framework, RLE lib

• GearPump: Real-time Streaming Engine Using Akka [github] 
• Seastar : Really interesting framework for high-performance server applications on modern hardware. It leverage DPDK and single threaded model to enable fast zero copy communication, however like any higly specific framework model there is a certain learning curve. Applications using Seastar can run on Linux or OSv. [github] 
• TurboRLE : an Efficient and fastest Run Length Encoding library, nice real time stream engine using SQL semantics.
• Pulsar : Ebay start moving more aggressively into the opensource ecosystem game [ source ]

Links of the day 18 - 02 - 2015

Today's links 18/02/2015: Packet Sender and #Bigdata : Stream processing , Probabilistic methods, PCA

• Stream Processing and Probabilistic Methods: great introduction on how you can leverage probabilistic methods ( hyperloglog, blom filter, etc..) to handle data at scale
• Making sense of stream processing : excellent talk by Martin Kleppmann on how event streams can help make your application more scalable, more reliable and more maintainable. [transcript]
• Packet Sender : open source utility to allow sending and receiving TCP and UDP packets. Really practical when you need to debug network protocols.
• PCA : Principal component analysis (PCA) is a technique used to emphasize variation and bring out strong patterns in a dataset. It's often used to make data easy to explore and visualize.

Links of the Day 17 - 12 - 2015

Today's links 17/12/2014: HP's machine,  #datacenter, #bigdata, stream computing, #bitcoin dev guide

• Tigon : open-source, real-time, low-latency, high-throughput stream processing framework from Cask Data, Inc. and AT&T . Interesting to see how now company are all releasing open source components.. Open source projects are now just another economic weapon on the corporate arsenal. 
• Bitcoin Developer Guide : detailed information about the Bitcoin protocol and related specifications
• The Machine: HP datacenter scale computer based on memristor technology
• The HP Memristor Solution for Computing Big Data : Stanley Williams lecture on HP "machine"

Links of the day 8 - 11 - 2014

Today's links 8/11/2014: stream processing, documentation, #machinelearning , #bigdata

• Write the docs : a place where the art and science of documentation can be practiced and appreciated
• Mantis : Netflix's Event Stream Processing System
• ACID Stream Processing : transactional stream processing system that supports full ACID properties without compromising scalability and high throughput
• Machine learning : 
• when machine learning goes bad : when using ML to do preemptive maintenance end up being counter productive and really expensive.. 
• large scale deep learning : lecture deck from Google fame Jeff Dean

Links of the day 19 - 11 - 2014

Today's links 19/11/2014: #resiliency, #cloud , distributed system, #stream processing

• Resilient by Design : React SF 2014 talk by Ben Christensen on how Netflix use the application of reactive design principles to guarantee resiliency. 
• The Distributed Developer Stack : opensource Field Guide by O'reilly crowd 
• Kafka and Samza : Distributed stream processing slide deck

The network performance within the cloud, an hidden enemy

A lot of people talked about the latency issue when hosting services in the cloud . Recently amazon latency hiccup revealed a deeper problem, but seems to be rarely discussed. While most focus on the network access and consume services from the cloud. I realise that their is a big unknown concerning network performance inside the cloud.

Could provider  don't disclose their real infrastructure underlying their cloud offers. By doing so, cloud customers are completly left in the dark regarding the network linking their different instances. Leaving them with the false warm feeling that their are on top their own flat network.

What does it mean:

• You have no idea of  your  network or I/O performance for your instance. Your virtual interface is sharing a  physical (sometimes trunked) one(s) with  other tenants collocated on the same physical server and theycompete with you for a share of the network pipe.
  
• You have no idea of your network performance  between multiple instances within the same cloud:
• First your instances can be located in different branch of the infrastructure. Which means more network gears between them.
• Then, Virtualizated  network gears can also be thrown into the mix. Which add virtual switches and routers with sub optimal performance (remember they are software) but add greater flexibility.
• Finally, the network traffic generated by all the tenants makes it very difficult (and expensive) to guaranty QoS throughout the infrastructure. Not to mention that capacity planning , measurement and management becomes extremely difficult because it is impossible to predict  the(often asymmetric) bandwidth  network consumption of the instance.  A reason why cloud providers dream for hugely dense, multi-terabit, wire speed L2 switching fabrics.

As a consequence, there is not generally a published service level associated with throughput and latency  within cloud.  When oversubscription hit you, you often don't see it coming.  Maybe cloud will become similar to the home broadband  with  advertised "unilimited" offers but with content ratio.

All this, makes it extremely difficult to deploy  and guaranty the performance of  services that rely on low latency and/or high bandwidth architectures such as high performance computing, web and database clusters, storage access, seismic analysis, large scale data analytics, financial services and algorithmic trading platform.

I can think of  some solutions to these problems but this will be for another post.