Wednesday, August 31, 2016

[Links of the Day] 31/08/2016 : Open Lambda and consensus algorithms

  • Open Lambda : allow anybody to run a local lambda platform similar to AWS lambda or Azure Function.
  • AllConcur : the authors propose in this paper a distributed system that provides agreement through a leaderless concurrent atomic broadcast algorithm. What is interesting is that the authors claims a 17x performance increase vs the leader based solution. However, there is a catch in there assumption : "We assume a model of reliable communication—messages cannot be lost (only delayed). This is a reasonable assumption if we consider a reliable protocol, such as TCP." I think that here we might hit a major issue as even with TCP we can have duplicated/ lost message. Paxos make not such assumption, Lets see if the idea can be adapted to an unreliable medium. 
  • Flexible Paxos : in this paper authors provide a proof that majority agreement isn’t required by Paxos and the sets of nodes required to participate in agreement (known as quorums) do not even need to intersect with each other.

Tuesday, August 30, 2016

[Links of the Day] 30/08/2016: trends in infrastructure , HPC / supercomputer and memory

  • Algorithms for future emerging technologies : Jack Dongarra CRIM talk presenting trend in supercomputing [video]
  • Trends in - and the Future of - Infrastructure : One of the fathers of software-defined networking (SDN),) Martin Casado, gives its view on the future of infrastructure. 
  • Hot Chops - memory - 2016 : more persistent memory, more storage, but slower overall performance.  What is interesting is that the trend is more about filling the niche gap rather than raw performance or storage size improvement. It seems that memory like cpu is hitting a performance wall. The two big trend are NVDIMM as near term solution and PIM : process in memory. PIM literally package memory on top of the logic processor. It reduce data transfer cost, but the heat generated by the overall package quickly rise, it requires to support >95 C   . Power and heat is now the number limiting factor in HW 

Monday, August 29, 2016

Thursday, August 25, 2016

[Links of the Day] 25/08/2016 : Micro-services architecture best practices, Semantic hashing and principles of programming language book

  • Best Practices for Building a Microservice Architecture : Article providing an overview of the best practice when shifting complexity from a monolithic systems to a distributed one. Complexity doesn't disappear, it is bounded to the interactions between simple services. While this sound great, the complexity increase with the degree of interactions which can follow a power law. 
  • Semantic Hashing : paper describing a method using deep learning algorithms for generating fast hash of documents. This method allow to generate locality sensitive that permit to execute similarity search over a vast library of documents in a time independent of the size of the collection. 
  • Principles of Programming Languages :  book providing an introduction to the study of programming languages derived from Johns Hopkins University programming language courses.

Wednesday, August 24, 2016

[Links of the Day] 24/08/2016 : Berkley Data science texbook, IaaS pricing trends, Wargaming conference

  • IaaS Pricing Patterns and Trends : Interesting to see that Google is really aggressive on its pricing. 
  • Computational and Inferential Thinking :  Texbook for UC Berkley foundation of data science class.
  • Connections 2016 : Report of the connections 2016 conference on Wargaming. There is definitely more than a thing or two to be learned by corporation on how to leverage war-games for improving and testing various strategy and understand competition behavior. Sadly the slides deck are not available yet. I wish that there was also video recording of this event.

Tuesday, August 23, 2016

[Links of the day] 23/08/2016 : Adapting In memory database architecture for Storage class memory and Datacenter network congestion management

  • The implication of Storage Class Memory for In memory database architecture : 
    • SOFORT : The authors propose to modify traditional In memory database architecture in order to optimise its operation for upcoming storage class memory hardware. The idea is quite simple, get rid of the log mechanism and persist all data to NVM except for the index which needs to be maintained in RAM for performance requirement. SCM allow to drastically eliminate a lot of boiler plate architecture functionality by delivering fast byte addressable persistent storage. However, now the developers needs to be aware of the transnational model imposed by this new class of persistent memories. [Slides]
    • Instant Recovery for Main-Memory Databases : This paper build on top of SOFORT and looks at leveraging NVDIMM or SCM for speeding up crash recovery features. The idea is not only speed up the normal operation but also eliminate the recovery cost in case of application crash [Slides]
    • Note that both these paper have an author working for SAP, so my guess that we will start to see new dedicated feature in SAP Hana for supporting SCM.
  • Flowtune : It seems that we are going to see slowly a return of the ATM model in data-center for networking fabric. In this paper the author propose to combine a form of MPLS system with a centralized allocator for resources management and congestion avoidance.  Basically the system identify connection ( called flowlet )  establishment  and end . Using the existing and past information it derive an optimal path and resources allocation minimizing interference and congestion over the lifetime of the flowlet. Looks like SDN is finally enabling a simplified and more robust ATM model within and probably across data-centers. 

Monday, August 22, 2016

[Links of the day] 22/08/2016 : Kubernetes the hard way , GopherCon 2016, 3dxpoint DIMM

  • Kubernetes The Hard Way : The famed Keysley Higtower explain how to deploy and run a kubernetes system on google cloud the hard way ( no automatic installation cheat)
  • GopherCon 2016 : Videos of this year Gopher academy
  • Wicked Fast Storage and Beyond : Intel IDF 2016 talk on the future of storage , looking at 3dxpoint, and the Optane SSD . What is really exceiting is the upcoming Intel DIMM using 3DXpoint tech. This provide native PMEM capability , 2x storage vs RAM (but slower yes). I can seriously see future in memory database ( SAP HANA ... ) bypassing storage together and just stacking NVM DIMM. Using slower storage form ( over NVMe by example) for backup / snapshot, etc..

Thursday, August 04, 2016

[Links of the day] 04/08/2016 : Industrial Organisation Reading List, SQL migration, Quiescent States

  • Industrial Organization reading list : extensive fall reading list on industrial organistion. Covering : Sectors: finance, health care, others - Incentives - Production - Organization - Competition
  • gh-ost : triggerless online schema migration solution for MySQL. It is testable and provides pausability, dynamic control/reconfiguration, auditing, and many operational perks.[github]
  • Using Quiescent States to Reclaim Memory : Explanation on how to use Quiescent states to implement lock free algorithms. Similar to RCU in userspace.