Thursday, December 31, 2015

Links of the day 31/12/2015 : Blitzscaling Stanford class, CCC videos, Clock & interrupts & schedulers






Wednesday, December 30, 2015

Links of the day 30/12/2015: #destiny development process, vision of processor & memory systems and fault tree analysis


  • Vision of processor and memory systems : Excellent presentation by J. Thomas Pawlowski, Chief Technologist, Fellow at Mircon. It provides you with a very good overview of the memory landscape technology, trends and challenge. Pay special attention to the the power wall. Moreover, the automata processor product seems interesting as micron tries to move up the value chain with a different approach to the auxiliary processing unit => memory centric. While they have a vested interest in this approach, it seems opposite to the overall fabric trend (omnipath etc..). However, t might turn out complementary on the long term.
  • Lessons from the Core Engine Architecture of Destiny : cover the six-year development arc of the Destiny engine, from inception to ship. Some valuable lesson and a very deep and thoughtful and honest introspection of the process.
  • Fault Tree Analysis : very long but extremely complete overview on how Nasa use fault tree analysis to:
    • resolve the causes of system failure
    • quantify system failure probability
    • evaluate potential upgrades to a system
    • optimize resources in assuring system safety
    • resolve causes of an incident
    • model system failures in risk assessments

Tuesday, December 29, 2015

Links of the day 29/12/2015: Datavisualization, OS dev book and ICPE15 conf

  • Datavisualization.ch : Awesome collection of tools that the people behind Datavisualization.ch, work with on a daily basis and recommend warmly. This is not a list of everything out there, but instead a thoughtfully curated selection of our favorite tools that will make your life easier creating meaningful and beautiful data visualizations. Highly recommended!
  • The little book about OS development : ever wanted to make your own OS ? well this book will help you get kick started. And even if you don't want to build your own. This book will give you great insight on the nitty gritty detail of what make your OS tick ( and crash... )
  • ICPE 2015 : Technical Program and Conference Schedule, a year old but some nice paper and a lot of slide deck are available on the page. Checkout System-Level Characterization of Datacenter Applications paper or Adrian Cockroft presentation on cloud native optimization.

Monday, December 28, 2015

Links of the day 28/12/2015: #cloud trends, Game theory book, and Guide to handling bad data

  • Quartz guide to bad data : An exhaustive reference to problems seen in real-world data along with suggestions on how to resolve them. This is a must read for any data scientist out there. A lot of practical solution to a lot of tough and not so tough problem encountered in the wild. 
  • Game Theory: Open Access textbook on non-cooperative Game Theory with 165 solved exercises.
  • Cloud Trends : Adrian Cockroft talk on future cloud trend delivered November 2015 at Structure conference . A lot of docker, a lot of other stuff , some good insight if you don't really follow the cloud market evolution closely. 

Monday, December 21, 2015

Links of the day 21/12/2015 : super malloc, ISMM15 , Distributed system reading list

  • SuperMalloc : implementation of malloc(3) originally designed for X86 Hardware Transactional Memory (HTM). It turns out that the same design decisions also make it fast even without HTM. [github]
  • ISMM15 : ACM SIGPLAN International Symposium on Memory Management
  • Distributed Systems Seminar's reading list : list of papers used by the excellent Murat Demirbas for his distributed system seminar. A must read for anybody in the field.

Wednesday, December 16, 2015

Links of the day 16/12/2015 : curl | sh, Huawei NUWA and EU Mikelangelo project

  • curl | sh : People telling people to execute arbitrary code over the network. What harm can it do ?
  • Nuwa : Huawei micro-server with disaggregated architecture based on commercial chips, and the computing and storage nodes can be switched on demand.
  • Mikelangelo : EU project with some interesting bit of technology and consortium Member : Huawei, Cloudius (for OSv). Note the vRDMA concept where they aim to use shared memory approach over RDMA to speed up communication across VM.

Tuesday, December 15, 2015

Links of the day 15/12/2015 : GCHQ graph DB, Cloud Benchmark and racetrack memory

  • Gaffer : GHCQ graph engine , note that they release v1 and work on v2. This give you an idea of the technical gap between what we see and what they use.
  • Racetrack Memory : this technology has been promised for more than 15 years. It might need 15 more years before it comes to fruition and probably at that time be already obsolete. Lets hope not.
  • PerfKit Benchmarker : cloud benchmarker which supports 9 cloud providers, including AliCloud, Amazon Web Services, CloudStack, DigitalOcean, Google Cloud Platform, Kubernetes, Microsoft Azure, OpenStack, Rackspace, as well as the machine under your desk or in your datacenter. It fully automates 26 benchmarks covering compute, network and storage primitives, common applications like Tomcat and Cassandra, as well as cloud-specific services like object storage and managed MySQL.

Thursday, December 10, 2015

Links of the day 10/12/2015: Networks, Crowds, and Markets book, end to end encrypted DB, low level Datacenter devops

  • Networks, Crowds, and Markets : Books on the different scientific perspectives and approach to understanding networks and behavior. Drawing on ideas from economics, sociology, computing and information science, and applied mathematics, it describes the emerging field of study that is growing at the interface of all these areas, addressing fundamental questions about how the social, economic, and technological worlds are connected.
  • ZeroDB : An end-to-end encrypted database protocol.
  • Vapor : can't believe a company actually named themselves vapor .. anyway . While their hardware is semi interesting what really picked my interest is their Open Data Center Runtime Environment. It aims to offer capability to expose and manage low level DC capabilities such as Power and Temp. I am not sure if devops will really want to go that low but this might be a nice addition to orchestration framework for meta optimization. 

Wednesday, December 09, 2015

Links of the day 09/12/2015 : Intel telemetry framework, log search system and DIGITAL!

  • Snap : Telemetry framework by Intel. While this is yet another monitoring system it is a little bit different from what is out there. It offer a full secure clustering solution design to be not only scalable but also has a integrated flexible life-cycle management of its component. This might looks like a great framework for future telemetry systems and I can see it being adopted very quickly across the industry.
  • Designing a search system for log data : 3 part post on how to design from the ground a system for analyzing  logs at large scale.
  • Digital : because digital!


Tuesday, December 08, 2015

Bitcoin: the blockchain ecosystem lubricant

As mentioned in a previous post, Bitcoin is a rather attractive solution for self contained market ecosystem. What is really attractive and useful is the underlying blockchain technology. Recently, banking institution and other financial institutions have started looking into the possibility to deploy blockchain technology to implement a distributed ledger system. However, these institutions might want to create a fully private self contained blockchain system, probably without any virtual currency functionality.

Private blockchain

Financials institution have an history of seeking privacy and control over the tools and mechanism they rely on. As a result, they will have the natural tendency to try to deploy their own private blockchain platform. Essentially, instead of having a fully public and uncontrolled network and state machine secured by cryptoeconomics (eg. proof of work, proof of stake), they seek the creation of a system where access permissions are more tightly controlled, with rights to modify or even read the blockchain state restricted to a few users, while still maintaining many kinds of partial guarantees of authenticity and decentralization that blockchains provide. The obvious objective for a consortium or company running a private blockchain is to be able to change the rules of a blockchain, revert transactions, modify balances, etc.


Private does not implies trust

The parties to a transaction have to be 100% confident that nobody can change the transaction. However, since you can change the rules at will there's a risk of manipulation of the system. The potential collusion, and lack of transparency for a system that requires an arbitration system to guarantee fair execution should be concerning for potential customer. Naturally, there is many incentive to have a private systems such that the need to protect the secrecy of transaction. But, you can still deliver such feature by relying on hybrid system where you have dark transaction pool sanctioned by the public infrastructure. Moreover, without a virtual currency to provide an incentive for uninterested third party to participate in the system, accountability remains difficult to ascertain.


Blockchain system lubricant : Bitcoin 

Public blockchain system is tied to a virtual currency, which reinforce its trust-ability while allowing an incentive in selfish participation. The principle is that blockchain consumer recompense participant in a form of tit for tat approach, allowing a more heterogeneous ecosystem preventing the dominance of a single type of participant as well as peer pressure to keep the system trust-able. Without it, the value of the currency will simply vanish and as a result remuneration from participation.
This heterogeneity prevent collusion (think LIBOR scandal) and make the system more robust and accountable. This is where bitcoin is really needed and shine, not for common everyday operations.

Links of the day 08/12/2015 : CIA corporate sabotage manual, Cloud scalability patterns and 3D Ram technology

  • CIA’s Simple Sabotage Field Manual : well... it seems to be the internal operations playbook for a lot of company out there
  • Cloud Scalability Patterns : must watch for anybody moving applications to the cloud.
  • 3D ram stacking : Interesting development in the 3d chip business and its far reaching consequence in memory. It seems that we are bridging the bandwidth gap with CPU however Latency and remain the majors roadblock.

Monday, December 07, 2015

Links of the day 07/12/2015: Netflix Streaming architecture, Silicon Valley Satire, FLOSS doom checklist

  • 1/3 of internet Traffic : How Netflix Direct architecture was designed to route a third of the world internet traffic. 
  • How to tell if a FLOSS project is doomed to FAIL : quick check list to get an indication if an opensource project is doomed on not.
  • Iterating Grace : satire about a Silicon Valley programmer named Koons Crooks, who has a spiritual awakening that leads him on a pilgrimage to a volcano in Bolivia. Living in a yurt, he spends his days pondering the transcendent wisdom of tweets by top venture capitalists before apparently being trampled to death by vicuñas. Just read it.

Friday, December 04, 2015

Links of the day 04/12/2015 : Sub-NUMA patent, Fabric Emulation for HP Machine, Performance monitoring toolkit


  • Sub-NUMA : Numa is already hard to deploy and use efficiently, lets start bringing sub numa clustering. Sadly one of the consequence of the ever increasing core count. Rather than selling micro chip single core highly optimize CPU. Intel prefer to cram more feature into a larger package even if 50%+ of them are unused. Sometimes I really hope that ARM server will finally come to fruition and shake the whole damn market.
  • Fabric attached memory : want to play with HP "the machine" and its new shiny fabric but can't get early access to it or you don't have the finance. Fear not ! here is the emulation toolkit of Fabric-Attached Memory for The Machine.
  • LIKWID : Performance monitoring on Intel Haswell platforms easy to use but yet powerful performance tools for the GNU Linux operating system. Really cool set of tools for topology discovery , numa pinning, power monitoring etc... [github]

Wednesday, December 02, 2015

Links of the day 02/12/2015: fast linux perf analysis, Path to AI, Datacenter transport

  • Linux Performance Analysis in 60,000 Milliseconds : fast minimal performance analysis approach allowing to quickly narrow down where the issue might come from. 
  • Path to general AI : interesting essay on AI and why the current path for will not allow the emergence of a true intelligence as human understand it.
  • pHost : Distributed Near-Optimal Datacenter Transport Over Commodity Network Fabric. This is an improvement over the previous Fastpass transport as it allows end-hosts to directly make scheduling decisions, thus avoiding the overheads of centralized scheduler architecture

Tuesday, December 01, 2015

Links of the day 01/12/2015 : VISC, HP "machine" network fabric, Emu real time analytics server

  • VISC : Researcher are trying to improve the security, reliability, and performance implications of shipping all software in virtual instruction set form. They use the LLVM virtual instruction set as the shipping representation of code to enable install-time and run-time compilation of all such software. Probably a lot of potential there if coupled with uni-kernel tech. 
  • Communications fabric for The Machine : HP session presenting their own fabric, competing with Intel Omnipath, PCIe and others already out there. The risk is that it might be too specialized for an industry wide adoption unless they start to aggressively "give it way" the IBM openpower way. 
  • Emu : New type of server where the processing is moved to the data with a distributed shared memory model and high speed fabric. Seems like a revival of the connection machine for real time analytic solution. What is interesting is that they use a RapidIO fabric.