[Links of the Day] 14/05/2018 : Concurrency and Paxos resources, PostgreSQL + docker streaming replication

• PostDock : This is an interesting project, It aims are delivering a Postgres streaming replication cluster for any docker environment. Sprinkle this with Kubernetes config and you would end up with an RDS equivalent. Even if I still think that on a long run CockroachDB / spanner solution are probably better for cloud deployment.
• awesome-consensus : Awesome list for Paxos and friends
• Seven Concurrency Models in Seven Weeks : more concurrency stuff. Excellent (free) book looking at all the important stuff: Threads & locks, functional programming, separating identity & state, actors, sequential processes, data parallelism, and the lambda architecture. 

[Links of the Day] 07/05/2018 : Cryptocurrency Consensus Algorithms , Fast18 conference, Google 2 real world project translation

• A Hitchhiker’s Guide to Consensus Algorithms: this post provides a quick and easy way to understand the classification of the various cryptocurrency consensus models. It's a gentle introduction to the concept of proof of work vs proof of stakes vs proof of authority vs ... Well, you got it many many more algorithm.
• Notes from FAST18 : a very good overview of the Storage conference. What is becoming obvious over the years is that a lot of the actual practical implementation of novel distributed storage solution is directly pushed into Ceph. Ceph is poised to become the defacto default private storage solution even if it has a long way to go in term of manageability and automation. I think it stems from the preconception that a lot of operations need a storage admin person. But the projects like Helm are helping it get there.
• xg2xg : a practical translation table of internal google tech and similar technology available to those that do not work in the chocolate factory. It is a very good list of production-ready project that can be leveraged in many devops (and non-devops) environment.

[Links of the Day] - 18/01/2018 : Stellar Cryptocurrency Consensus protocol, Optimizing linux server for high throughput and low latency, performance impact of meltdown patch on HPC Filesystem

• Stellar Consensus Protocol : from ripple for to full-blown rewrite. Stellar looks like an impressive protocol addressing many of the shortcoming and risk of Ripple. Also, the authors seem to be smart enough to avoid jumping to fast onto the smart contract aspect as it is a really tough nut to crack. Maybe, with all the mayhem surrounding cryptocurrency, the stellar approach seems to be rather measured. Worth keeping an eye on. 
• Optimizing web servers for high throughput and low latency : very good post on how to optimise your Linux system. A lot of it has already described many times, but it is never a bad thing to repeat them.
• The performance impact of Meltdown patches on HPC FS (Lustre) : no surprise here, IO intensive applications are the one the most heavily impacted. However, I wasn't expecting 40% performance penalty and up to 45% for large folders. 

[Links of the Day] 05/09/2017 : Patent Surviving Alice, Hot Cloud 17 conference papers

• 7 Post-Alice Patent Cases That Survived 101 Rejections : Alice US supreme court decision started a slaughter in the US patent office regarding IT related patent: 8400 applications dropped and 60k+ rejected. While courts invalidated the vast majority of patent litigation. However, it seems that there is a way to survive the onslaught, and it's quite simple. You just need your patent to satisfy the following criteria: novelty, enablement, non-obvious, and last but not least useful. it seems like a no brainer, but it seems that the USPTO allowed itself to be flooded by sub par applications that gamed the system. Not to mention that the agency financially gained from such practice to some extent also. 
• Hot cloud 17 : hot cloud conference just finished, here is a selection of interesting paper
• JavaScript for extending low-latency in-memory key-value stores : Adrian Colyer takes a look at in memory javascript engine using RamCloud. RamCloud project is entering the use case phase of the research project,  eyeing the comercialisation. Sadly most solution put forward are extremely niche. The risk is for great people to be stuck in a zombie startup if they try to run with it. Taking separately, the tech that came out of the RamCloud project is amazing. However, the solution as a whole doesn't really have a great killer app or any potential beyond some niche market.  [paper]
• Towards Index-based Global Trading in Cloud Spot Markets :  the authors propose to use an index based prediction model rather than per spot instance in order to obtain greater reliability at lower cost.
• DAL: A Locality-Optimizing Distributed Shared Memory System :  Different take on the whole in memory K/V system, the authors aggressively move the data to the computation rather than offering remote access. This allows great data reuse. We used something similar in hecatonchire. However, there is a certain risk when you have a high level of churn or serial data access and local caching of data generate a high level of eviction, effectively doubling the bandwidth usage. 
• Leader or Majority: Why have one when you can have both? : raft is a great consensus protocol ( and easier to understand). However, the over reliance on the leader is the main bottleneck for scalability of operations. The authors ( from cockroachdb ) propose a quorum based read operations that allow alleviating the load on the leader while still retaining strong consistency. This allows them to improve write by 4x write perf and increase throughput by 60%. Which is quite impressive. 
• DCCast: Efficient Point to Multipoint Transfers Across Datacenters : the authors proposed an efficient multipoint data transfer protocol allowing greater efficiency and bandwidth usage.

[Links of the Day] 10/03/2017 : User-space SysFS, Key Value consensus Algo, Cost efficient Big Data Serverless Framework

• ProcStat : Userspace equivalent of kernel SysFS. Really cool project by my friend Sasha. makes it really easy to expose internal counter and state of a process via FUSE
• Bizur : Key-value Consensus Algorithm using a nice solution where consensus is achieved on the key themselves rather than relying on a globally distributed log. The great aspect is that recovery and failure management is greatly simplified and streamlined. However, it implies that the progress and consensus on each key are independent of each other. As a result, you cannot rely on serialisation of state between key. Which can be limiting if you expect the state of Key A to be changed after the State of Key B by example. 
• PyWren : Framework that let you use serverless functions for cheap large-scale data analysis. [github]

[Links of the Day] 07/11/2016 : Baidu Open Source Repo(s), Wan Replicated DB

• Baidu : Baidu open source code on Github. It looks like it replicate a lot of service / feature that other hyperscale system use. Raft seems to be the default underlying consensus protocol for all applications. A lot of nice goodies in there, especially: 
• BFS : Baidu file system that provide the underlying persistence for Baidu real time application. Its a distributed multi datacenter using raft for metadata coherence and use a shared nothing approach for linear scalability. 
• Tera : Distributed database 
• Galaxy : mesos / kubernetes equivalent. 
• Paddle : Distributed machine learning 
• iNexus : Distributed K/V store . Looks similar to consul and it also use raft as the underlying consensus protocol
• Bedrock : Wan replicated distributed data (base). Designed to use SSD and other nice features. 

[Links of the Day] 31/08/2016 : Open Lambda and consensus algorithms

• Open Lambda : allow anybody to run a local lambda platform similar to AWS lambda or Azure Function.
• AllConcur : the authors propose in this paper a distributed system that provides agreement through a leaderless concurrent atomic broadcast algorithm. What is interesting is that the authors claims a 17x performance increase vs the leader based solution. However, there is a catch in there assumption : "We assume a model of reliable communication—messages cannot be lost (only delayed). This is a reasonable assumption if we consider a reliable protocol, such as TCP." I think that here we might hit a major issue as even with TCP we can have duplicated/ lost message. Paxos make not such assumption, Lets see if the idea can be adapted to an unreliable medium. 
• Flexible Paxos : in this paper authors provide a proof that majority agreement isn’t required by Paxos and the sets of nodes required to participate in agreement (known as quorums) do not even need to intersect with each other.

[Links of the day] 30/05/2016 : FPGA market extinction event, Consensus as cloud service , OSCON16

• FPGA extinction level event : article looking at the evolution of the FPGA vendor market. It seems that if Xilinx get aquired, 80% of the FPGA market vanished ( Altera was acquired recently by Intel). This as far reaching implication for the market as consolidation occurs and focus seems to be toward datacenter solution at the detriment of the rest of the market. 
• Filo : consolidated consensus as a cloud service. Really interesting paper looking at the possibility to offer a consensus system as a service within cloud. This would greatly help anybody out there relying on their zookeeper / consul / etc.. and allow them to focus even more on the business logic.
• OsCon : slides of the excellent OsCon are up . Lots of docker related stuff .. but if you look past it there is also some gems such as Netflix SSH Bastion talk or the Build to Lead talk.

[Links of the day] 11/04/2016: Rust Distributed K/V store, Consensus in Cloud and interactive service tail latency

After a short hiatus here are the links of the day back :

• Target-Driven Parallelism : Microsoft researcher look into using prediction and correction to reduce tail latency in interactive services.
• Consensus in the Cloud : a very good technical report on systems using Paxos and the advantage/disadvantage associated with its use.
• tikv : Distributed key value store written in Rust, Use Raft to deliver consistency and scalability coupled with a nice georeplication capability. Cherry on the top: distributed transaction are supported similar to google spanner. 

Links of the day 07/10/2015 : consensus, transaction and correctness

• Consensus in the wild : lot of way to achieve consensus out there but which one is the most efficient for your requirements. Answer : it depends and we have no consensus.
• IronFleet: Tool for proving corectness of practical distributed systems using TLA-style state-machine refinement and Hoare-logic verification. 
• AtlasDB : Transactional Distributed Database layer on top of K/V store.

Links of the day 26 - 06 - 2015

Today's links 26/06/2015: Lockless synchronization, #bigdata architecture papers, Raft consensus

• 100 open source Big Data architecture papers :  all in the title
• Two very good talk from the Paper we love meetup : 
• Raft : introduces the Raft consensus algorithm, which is an attempt to make a consensus algorithm that is easy to understand and easy to implement.
• Making Lockless Synchronization Fast : overview of common safe memory reclamation mechanisms and then explores their performance implications.

Links of the day 01 - 10 - 2014

Today's links 01/10/2014 : consensus, #startup, #bigdata, #machinelearning

• Understanding Consensus : nice overview and explanation of the consensus problem by Tyler Treat
• Startup Lecture 3: Counterintuitive Parts of Startups, and How to Have Ideas, third lecture of the excellent lecture series at Stanford by the YC crowd. 
• Intelligence cannot be commoditized : my own blog self promotion :) 

Links of the day 16 - 09 - 2014

Today's links  16/09/2014 : memcached, facebook, consensus

• MCrouter :Memcached Key value pair protocol router by facebook: ( github ) 
• A series of blog post on consensus in distributed system from Treode: 
• The Minitransaction: An Alternative to Multi-Paxos and Raft
• Why Treode Avoided Multi-Paxos
• Single-decree Paxos