Links of the day 2/10/2015 : ARM conference, Hashicorp Nomad, Kudu storage for Hadoop

• Linaro Connect : For those of you using ARM based system. [all the presentations][videos]
• Nomad : cluster manager and scheduler designed for microservices and batch workloads by hashicorp.
• Kudu : another storage solution for hadoop.

Links of the day 29/09/2015 : Time series in SQL, Chaos engineering, Distributed Systems Architecture

• Storing Time series in SQL : how to efficiently store time series data in postgres SQL or other SQL db
• Principles of Chaos Engineering : discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production.
• Architectural patterns of resilient distributed systems : while this is a good talk about resiliency I have a slight issue with the term itself. I have a preference for ductile rather than resilient. As often system that are designed to be resilient are extremely brittle when the reaching the breaking point [video] [slide-deck]

Links of the day 28/09/2015: SPDK , CAP for durability, NVM and In memory DB

• SPDK : Intel pendent of its DPDK toolkit specially aimed at storage and NVMe , As usual you can do some interesting stuff with it but it is also higly restrictive in term of Hardware and behavior. The device need to be purely dedicated to the library and cannot be shared with other system. Moreover it might be as big as a security risk as the DPDK libs (memory is accessible across DPDK consumer..) [more user friendly overview here]
• Impact of new Persistent Memory Tech on In memory DB: Nice overview of the impact of the new NVM tech. Too bad its still try to push the classic DB model instead of proposing an approach from scratch that would fully use the potential of these new technology. 
• CAP theorem for Durability : interesting effort to formulate the limitation of the durability problematic in distributed environment.  However it doesn't help curtail the acronyms soup from spreading. 

Links of the day 21/09/2015 : Encrypted DB, Curl and No/SQL

CryptDB : database system that can process SQL queries over encrypted data.
Curl Cheat Sheet : for the ones out there curling away.
How to do SQL with K/V : how to map SQL style table to a key / value store. Also know SQL in NoSQL :)

Links of the day 17/09/2015 : philosophy and donuts, GDB dashboard, Dweet API

• GDB dashboard : while i love DDD this is still really good when you don't have access to an X environment.
• Dweet : twitter style messaging for IoT. Reminded me of the time when I was using XMPP to do something similar to chef. 
• Philosophy Explained With Donuts : enough said...

The upcoming Storage API battle

There is an interesting trend within the storage ecosystem. We are witnessing a polarization of the offer. On one side, we are seeing the rise of high performance rack scale solution(DSSD, NVMe over fabric solution, etc..) . And on the other side we have the object storage solution which are more datacenter scale. While both leverage heavily non volatile memory they play a different different role within the ecosystem.

The rack scale storage target “very” high performance solution, delivering very low latency high bandwidth access time. Often in the 100 of usec or less. However often these solution come at a higher financial cost due to more expensive hardware (custom NVM), network fabric (IB+NVMe, PCIe + NVMe, Omnipath +NVMe, Pure PCIe, etc..) and significant power consumption (>2000W/5U for DSSD). Finally, these offer access via a specialized API that needs to be either accessed natively or adapted to other more standard one.

On the other side we have the object storage solution. Users access object storage through applications that will typically use a REST API. This makes object storage ideal for all online, Cloud environments. Moreover they tend to be a lot more cost efficient especially with the rise of Ethernet connected drives (up to 50% less TCO).

Stuck in the middle is the classic Filer / POSIX compliant solution that seems to slowly dwindle away. To a certain extent the rack scale solution should have a bright future in the niche (but still significant) market for enterprise that still consider that their application requires custom for what they think is a custom problem. On the other side the object storage is gaining momentum by riding the unstoppable cloud tide.

While, both technology can and should co-evolve, they both suffer from software limitation and to a certain extend hardware one as bandwidth and latency get dangerously close to what cpu are capable to handle. This requires a drastic shift on how applications are developed if user want to actually get any benefit from these solution. However, few company are willing to risk specializing their code using an API that can become obsolete when the next generation of storage solution pops up.

Storage startup/company out there start to discover that  API is playing a significant role success of their product and performance while still important will lose of its importance. It will either make rewriting applications to access your storage infinitely easier task or transform it into a painful experience by forcing to go through hoops and/or adaptation layers with the performance cost associated. 

The fight for the next generation storage API only started. There is and will be more push toward standardization which will be fueled by the customer tiredness with every revolving siloed point solution. People who use object storage one want it to behave more like POSIX storage, but they also want to keep the storage costs at an object level and improve the performance. On the other hand people using rack scale storage want to retain the performance but increase its simplicity and also want the price to come down. It is going to be extremely hard to deliver both but hopefully we might finally see a rationalization of the storage market as having an object storage system that allows byte range access is very appealing. 

Links of the day 16/09/2015 : neuroinformatics conference , power8 , server forum

• Neuroinformatics : congress 2015 videos of presentation
• Power 8 : inception, power optimization of the power 8 cpu 
• Server Forum : 2014 conference presentation . The interesting presentation is the Facebook DIMM interconnect one. It seems that they are looking on a way to allow server to share DIMM bank across server. Potentially I guess for their massive K/V store or memcache in order to reduce the access time. However i would be really curious to see how it works in practice.