Monday, February 29, 2016

[Links of the day] 29/02/2016: AWS latency, EU H2020 HPC projects, Python Edge Fund

  • AWS latency : if you need to find the closest AWS region this tool is for you
  • 21 EU h2020 HPC project : here are the 21 project for High performance computing funded by EU. While I have my reserve regarding the real benefit of these project it might be useful to keep an eye on some of them: montblanc, exaNode and nextgenIO 
  • Quantopian : gaming based Interesting edge fund where people compete with their algorithm for a slice of the fund.

Friday, February 26, 2016

[Links of the day] 26/02/2016 : Usenix Fast 16 , FPGA liberouter and Event delivery at spotify

  • Fast 2016: all of Usenix Fast 2016 goodness available in one place.Interesting to see that we start to see the emergence of storage systems optimized for time series [BTrDB] . Also note the always interesting report on failure rate (this time for flash)
  • Liberouter : really cool project using FPGA to deliver hardware acceleration of network security and monitoring tools. 
  • Event Delivery at spotify : part 1 of a series of blog post on event monitoring and management system used at spotify.

Thursday, February 25, 2016

IoT next battlefield: test/dev platform commoditization

IoT term is past the point of no return and is now officially one of those buzzword that “serious” business need to brand themselves with in order to keep up with the time.
Lets have a look at the domain and do a quick mapping of the IoT value chain. What we discover is that a good portion of the value chain is already heavily commoditised but surprisingly some key aspects are not.

Current IoT Value Chain

First, we might ask why testing and dev platform are not as commoditized as the rest of the value chain. We have to remember that we are not developing software only. We need to validate if the a combination of hardware + software as well as its behavior within the overall IoT ecosystem.
The dev/test requirement for Internet of Things is a significant challenge. The scale of the systems is big because they contains large number of units spatially distributed. And the larger the scale, the harder it is to test within the software development lab for testing. Corporation do not have the financial capabilities to deploy large scale IoT testing like google does.
Moreover, as Nest CEO,Tony Fadell, explain in Stanford’s industrialist dilemma session : Testing software, testing procedures, test fixtures, (everything test) are critical IP that need to be protected in order to retain competitive advantage. As a result, the overall testing and, to a certain extend, dev remained closely guarded aspect of the value chain. Company kept these two at lower end of the shift toward commoditization in order to increase the barrier to entry.

IoT Value Chain Transition

However, with the increased demand, the ineluctable commoditization process will accelerate. And simulation will be at the heart of it by enabling companies to virtually deploy IoT devices over a virtual space and test their behavior. Using such environment, they will can model and test cheaply every aspect of the system, such as connectivity , disruptions, etc.. Instead of manually handling and maintaining hundreds of devices we will be able to do it via an API. While, we still need to do hardware testing we probably reduced the overall cycle by an order of magnitude and allow faster release of new product.
Future IoT Value Chain

Currently, most cloud player have offered IoT solution that revolve around analytic and have made almost no foray in the testing and dev platforms. However, they are poised to offer cloud IoT testing platform which will generate a shift in power within this field (as shown above). It is going to be great for smart object designer houses as it will steal away the leverage from bigger player ( Honeywell , GE, etc..) by breaking the barrier to competition while enabling an explosion of the IoT ecosystem.

To be honest, I wouldn’t be surprised if Amazon did a play in that field. They have already an significant interest in IoT and 3d printer. It would be a natural step for Amazon, enabling the company to lock down yet another market.

Wednesday, February 24, 2016

[Links of they day] 24/02/2016: cluster file system, Disks, death and datacenters

  • Disks for Data Centers : google research paper on the place of "legacy" storage system in current and future datacenter. Its all about the use case and RoI. 
  • How To Kill A Supercomputer : Dirty Power, Cosmic Rays, and Bad Solder
  • BeeGFS : parallel cluster file system formerly known as FhGFS is going full open source. An alternative to Lustre , however I wonder how it will really fair with Ceph improvement

Tuesday, February 23, 2016

[Links of the day] 23/02/2016: Intermittent Computing, Crude company bankruptcy, DC conference

Monday, February 22, 2016

[Links of the day] 22/02/2016: Quantum computing as a Service, Alexa - Amazon IoT platform, SNIA nvm videos

  • Quantum Computing as a Service : Stanford seminar on "Quantum Computing as a Service" by Matt Johnson & Randall Correll of QCWare. What is stand out from most of the talk is that quantum computing is really a solution looking for a problem. You can speed up certain problem solving by an order of magnitude but paying customers with this type of problems seems scares. It can just be an educational problem and it seems they are trying to tackle this aspect well. However, they didn't talk about the elephant in the room : breaking cryptography for fun and profit. Maybe to sensitive of a topic?
  • Alexa : Amazon tech talk on its IoT platform
  • SNIA nvm summit :  all video are uploaded , checkout the linux NVM talks and nvm over fabric one.

Friday, February 19, 2016

[Links of the day] 19/02/2016 : Machine Learning Intro, Fungal Network and Usenix Enigma

Wednesday, February 17, 2016

[Links of the day] 17/02/2016: IoT + Sensor day at Berkeley, Financial of Hybrid cloud, bigdata lib

Tuesday, February 16, 2016

[Links of the day] 16/02/2016: VM density, VLDB15, Cassandra MQ, Eth roadmap

  • VM density : a look at VM density and utilization profile in Pernixdata cloud. Seems that the customers preference goes for 16 core dual socket virtual system while hogging as much memory as possible.
  • VLDB 2015 : Full VLDB 2015 program with papers attached. Notable Paper : Gobblin by linkedin crowd and Coordination Avoidance in Database Systems from Berkeley 
  • Cassieq : Distributed queue built on top of Cassandra

Bonus: Ethernet roadmap

Is Amazon using Lumberyard to replicate its Video business model in Gaming?

Amazon recently launched its own gaming engine : Lumberyard. This should not seems as a surprise with the stream of high level investments they have been doing in the field over the past couple of years: Twitch.Tv or licensing Crytech engine (which form the basis of lumberyard) to name a few. Moreover, I will not extend on Amazon underlying strategic play as Simon Wardley already did brilliant job explaining it here and there.

A lot of discussions analyzing Amazon move have been centered around the long term strategic play in the AR/VR field. However, in the short term, Amazon might be aiming to accelerate the value chain shrinkage while potentially moving away from the traditional gaming industry business model to a service based approach and ultimately a complementary business model.

Historically, Work-for-hire & Royalty advance practices generated significant upfront fixed cost in the video game development business model which resulted in making publishers as the de facto main financial operator. Publisher typically mitigated these financial risks via portfolio management which exacerbate the reliance on franchise game (86% of the market). 

With the switch to digital distribution platform and the explosion of mobile gaming, the physical logistics needs drastically decreased while the barrier to entry vanished. This commoditization trend effectively shrinked the value chain significantly as show in the diagram below. 

Moreover technology evolution enabled an increased variety in revenue model : 
  1. Subscription : Subscribers pay periodically to get access to the game (ex: World of Warcraft)
  2. Utility : metering usage, i.e. a pay as you go approach. This model is widely used among MMOs in China. 
  3. Advertisement : sometime used in combination with other model in order to enhance revenue. Pure advertisement model are mainly found in mobile.
  4. Micro-transaction model : dominate Eastern markets 
  5. Licensed : historical revenue model
  6. Free to play : combination of other revenue models , ex Advertisement + micro transaction.
There is two other business model that are still nascent in the gaming industry: Service and Complementary. And this is where, I believe, Amazon is aiming all along with its gaming push.

If we look at the value chain above Amazon's plan seems extremely straightforward. By facilitating production systems via “free” access to lumberyard Amazon facilitate the emergence of gaming studio. This open platform with efficient underlying support system (AWS) and with great customer exposure ( will drive the commoditization of content creator and by transitivity content itself. This approach literally cut the grass under the foot of traditional gaming corporation that relied on a high barrier to entry ( via game engine licensing, distribution network, backend, etc..).

By analyzing beyond the pure technological aspect we can quickly theorize that Amazon might be aiming at pivoting the gaming revenue model completely. Amazon could push for a Netflix like service model. However, there is a greater chance that it will follow the same approach it used for Amazon Video. Amazon could start offering Video Game access (downloading via app store steam style first, streaming later) free as a complementary to Amazon Prime customers. Prime serving as an incentive and creating opportunities for more lucrative cross-sell and up-sell opportunities. The Gaming service attracts customers to Amazon store, where they can purchase the content which is not available for free, as well as other products from Amazon. Moreover the overall business model effect would be further reinforced through the broadcast platform. 

Obviously, to support and accelerate this model, Amazon will need to start producing its own games. It needs to offer an attractive gaming experience that cannot be easily replicated while co-opting the rest of the industry at the same time. 
One of the key element regarding the pace of change will be dependent of the commoditization of the hardware platform and co-optation of existing one. If Amazon is able to broker a deal with MSFT or Sony ( the later is more likely because they already run their services on AWS). They would be able to gain a foothold in the gamer market. However, by co-opting the “hardcore” PC market , TV causal (Fire TV) and mobile, Amazon should be able to squeeze out the competition. Even if the console put up a fight, they would be able to enshrine in concrete any market gain by enrolling top game studio and capturing gaming franchise.

Last but not least, the value of console hardware is dropping fast while console software value is increasing and already exceeding hardware. Similar relations are to be found for handheld devices with an even greater gap. Amazon, just has to wait for for the gap to reach a critical point and then wipe out the nascent video game streaming industry by leveraging its existing expertise from VDI (workspace). All of this would be a textbook replay of the Amazon Video strategy. 

The future of gaming is about to enter a new era. While the AR/VR future is exciting there is a gaming business model war looming that will hit way before these technologies reach maturity.

Monday, February 15, 2016

[Links of the day] 15/02/2016: Homomorphic cryptonet, Netflix OSS meetup S4E1, Jails & Zones

  • Cryponet : really cool application of homomorphic encryption by microsoft enabling deep learning on encrypted data whi high accuracy. While I still think we are a still a couple of years away from everyday use of the tech due to performance limitation, this type of encryption is really going to up the game for security of public cloud data.
  • Netflix Open Source Meetup Season 4 Episode 1 : netflix and OSS (co-)evolution and spinnaker [video]
  • Jails and Zones : Papers We Love presentation by Bryan Cantrill on jails and zone in Solaris OS

Wednesday, February 10, 2016

The Tower of Hanoi Fallacy

The drive behind pursuing upward value chain motion is commendable in the the eyes of the shareholders. However, it often result in a costly failure or worse becoming a zombie company (Yahoo...). Most of the time, such approach fail because corporations do not understand and satisfy the core requirement driving the Innovate - Leverage - Commoditise strategy via leveraging a real ecosystem play. As a result, companies end up deploying brittle strategy with very weak underlying gameplay which under perform in a highly competitive market.
One of the main reason behind the failure of such strategy is company's lack of customer's ecosystem understanding. Software vendors, by example, have the false sense that they understand the use of their competencies (database systems or ERP) beyond the customer horizon. They regularly misunderstand the higher level of the stack valued by customers. SalesForce customers do not care if the underlying DB is from Oracle or SAP Hana. In AWS case, nobody care they run Xen, KVM or VmWare. This is perfectly illustrated in the cloud value chain mapping bellow. The only important parts is the last layer exposed to the customer itself. Anything under it is just the submerged part of the cloud iceberg.

Additionally, companies do not only fall prey to the Stack Fallacy. Sometimes, rather than (just) trying to move up the stack, corporation try to also move laterally in a bid to absorb adjacent stack. Large software vendor (ex: Oracle & SAP) are quite attracted to this tower of Hanoi play as look to force their way into a parallel stack with alternative business models. Recently these very large software corporation embrace such strategy by trying compete directly against IaaS and PaaS solution rather than co-opting them
These large companies tend to have a seemingly good awareness of their competences based on the resources and knowledge that they gathered via their marketing departments, sales teams, R&D dept, etc.. This awareness allow them to maximise revenue from the software value chain (mapped below, interpretation and trimmed down version of the software value chain by Pussep and Al.) .

However, these companies have build enormous revenue base on this type of value chain. And, transitioning to a different consumption model generate a struggle originating from the financial implication of transforming from licensing to SaaS model combined with the self cannibalization of their existing revenue streams. Basically, they cannot (out)innovate competition from the market they transfer into due to internal financial tension.

Moreover, while they have a good understanding of their value chain. They often completely underestimate the new one they need to adopt. The assumption is that it is just another Tower of Hanoi play.

Rather than co-opting existing platform by building as a customer on top of the cloud value chain they suddenly need to expand their capabilities in order to internalize the new requirement. You can see in the diagram below the daunting challenge they are facing. Not to mention they still require to maintain their old revenue streams in order to transition. This bipolarity is not without reminding Gartner Bimodal strategy, which if often considered to be harmful.

As a result without situational awareness, it is easy to fall prey of the Tower of Hanoï fallacy as it is near impossible to achieve:
  • transition to a new value chain
  • acquisition of new technical skills/knowledge
  • expand new market and business model understanding
  • compete with ecosystem natives
  • manage revenue stream self cannibalization.

You might ask what type of play these behemoth (SAP - ORACLE) should adopt? Here is a very succinct possible strategy based off Wardley IBM vs Aws play.
  1. Target a proxy / platform play by leveraging the inherent inertia of their customers. Hint : lifespan of SAP ERP solution average 15 years.
  2. Provide a AWS/Azure/GooG cloud proxy service for customer and third party ISV. Their customer are looking for stable, robust and reassuring solution for the backbone of their enterprise. Providing an first safe path for extension of "legacy" on premise solution with external cloud service (even and especially not their own). And second a migration path for full blown cloud deployment.
  3. Play the platform play card but without recreating their own cloud IaaS. SAP & Oracle have already something in that domain which should help make the transition as long as ego don't get in the way. Then encourage the rest of their ecosystem to join while focusing on operational excellence. To be honest they have such a massive ecosystem it is surprising they didn't try to push this further already. Moreover, leveraging Cloud foundry is also another option, however this might be too late for this.
  4. Monitor the growing ecosystem in order to spot successful emerging services. Acquire them rather than copying them in order as in the beginning it is all about ecosystem expansion rather than commoditization. This will help create confidence in the solutions as well as attract newcomer with the potential promise of future acquisition.
  5. Offer a form of cloud insurance market solution for as a long term pay. They can leverage it to expand their data mining capabilities while satisfying the customer risk averse need.

[Links of the day] 10/02/2016: Cloudstack fork, AsyncSSH, Purely functional data structures


Tuesday, February 09, 2016

[Links of the day] 09/02/2016: Run openstack in containers, Flocon network security conf, Agent motion book

  • Kolla : run Openstack in containers for ease of management and operational flexibility.
  • Flocon 2016 presentation : Network Situational Awareness for large-scale network flow analytics.
  • Space & motion of communicating agents : CS thesis looking at leveraging bigraph to model the space , motion of communicating agents. This essential as for the pas decades system mainly focused on getting the itnernal communication of entity right. But with the upcoming Punctuated equilibrium of IoT we external communication systems in an non static world will become critical.

Monday, February 08, 2016

[Links of the day] 08/02/2016: EU cloud tender, Solo5 unikernel and microXchg 2016 microservice conf

Saturday, February 06, 2016

Guesstimating Private Cloud TCO

I decided to try out the fantastic GessTimate. Guesstimate is a spreadsheet for things that aren't certain. I recently started to gather informations on private cloud TCO and I decided to see if i could quickly sketch out a probabilistic private cloud TCO model with it.

First, a little bit of literature review. On one side, we have the public cloud with an open pricing which make cost calculation and comparison straightforward. On the other we have the fog of war of the private cloud. There is very few non-behind paywall reports or information out there and here is the result of a (very) quick search for serious source about private cloud TCO data :
While all the numbers and comparison are highly interesting, I needed to find something usable for building a probabilistic TCO model. Finally, I found that many reports and online information referenced Amazon's James Hamilton analysis of the breakdown of cloud cost. His model boil down to the following :
  • Servers: 57% 
  • Networking equipment: 8% 
  • Power distribution and cooling: 18% 
  • Power: 13% 
  • Other infrastructure: 4% 
For my own model I decided to use the following categories :
  • Cost of money : The interest that could be earned if the amount invested it. Yep money is not free. 
  • Software : Not everything is Open source, and even so you still need to deploy it. 
  • Hardware : Server, SSD, HDD, etc… 
  • Network :connectivity, switch, cable, router, etc... 
  • People/Support : How much do you spend on maintaining your environment and deal with the customer need. 
  • Aircon/power : Powering and cooling your cloud 
  • Building Racking : Well it’s not like you are running everything in the cloud :) 
Now that we have a model, let's plug that into guesstimate. You can see in the picture below the result of the exercise. Note, DO NOT pay attention to the numbers, they are completely fantasist as I couldn’t get access to the latest IDC datacenter cost report.
Screenshot from 2016-02-06 20:51:09.png
Private Cloud TCO Guesstimate

What I ended up with is a really neat model where you can plug your own numbers and come up with a quick estimation of your private cloud TCO. It will take more work to generate a more accurate model but its good enough for basic estimation. In the mean time, you can find the source here. If you extend/change it, don’t forget to give me a shout. I am interested to see what other private cloud TCO model are out there.

Update 07/02/2016: +Carlo Daffara pointed me to his extremely detailed CAPEX / OPEX spreadsheet model. I'm going to see if I can translate it to guesstimate as a more advanced exercise.

Friday, February 05, 2016

[Links of the day] 05/02/2016: Cloudlet server image, Shmoocon 2016, Multidimensional queries

  • Multidimensional Access Methods : 1996 paper showing how hard it is to resolve multidimensional queries . List the state of the art approach which are still relevant to date.
  • Cloudlets : I think I linked to cloudlets before, but I'll do it again jsut to remind myself to actually use them. Cloudlets are universal server images for the cloud. They're lightweight, version-controlled, and you can export them to any bootable format known to man: Xen, KVM, Amazon EC2, or just a plain bootable CD.
  • Shmoocon 2016 : excellent security conference , especially check out the post quantum crypto talk by Jean-Philippe Aumasson 

Bonus : video demonstrating CISE tool for proving distributed systems correctness

Thursday, February 04, 2016

[Links of the day] 04/02/2016 : Hierarchy are bad for human cooperation, Software defined memory and D&D alignement

  • Hierarchy is Detrimental for Human Cooperation : really good experimental study demonstrating something that a lot of people intuitively know. However, I have a slight concern regarding the interpretation of their finding. While they pretty much nailed it (see below), I suspect that a more accurate title would have been inequalitarian hierarchy is detrimental. If an organisation is able to deliver similar payoff throughout the layer of the hierarchy the detriment will inherently disappear. However, I have yet to see any organisation achieving this feat!
"""We have shown that achieving cooperation among humans is more difficult when there is an underlying hierarchical structure producing different ranks between people and therefore unequal payoffs for the participants. This result is driven by insufficient contributions from lower ranked individuals who cannot be confident that they will benefit from cooperating."""
  • Plexistor SDM : Plexistor Software-Defined Memory (please tell my why every single product out there has to be has to use a hype compliant description....) concept is a generic filesystem for storage, from NVDIMM to NVMe to SSD. They aim to offer a new POSIX filesystem that any application can use rather than trying to modify applications. In short they offer an under the hood tiering mechanism between the different underlying persistent layer.
  • Chaotic or Evil? : applying D&D characters alignement system to your project, team and teammate. Next hiring interview: ask all past characters sheet & compare with team (ps: never trust a rogue)

Wednesday, February 03, 2016

[Links of the day] 03/02/2016: Verifying distributed system and technical progress prediction

  • Predictability of technological progress : the authors present a technique that allows to make forecasts for any given technology. In layman terms, it can to a certain extend determine if the diffusion speed of a technology and hence its success. 
  • Psync : really nice DSL for fault-tolerant distributed algorithms using partially-synchronous communication-closed rounds. It allows to create verifiable code of distributed system straight from scala . [morning paper summary]
  • The Verification of a Distributed System : provide with an overview of most commonly used techniques for distributed system verification out there. 

AzureStack : Beyond hybrid cloud play, capturing the future "near shoring" cloud market

Microsoft released AzureStack last week. This solution, built on top of the forthcoming Window server 2016, enables customers to deploy a private cloud with hybrid capabilities. This product extend beyond limited subset of Azure features offered by Azure pack as Microsoft promised a full 1:1 code match and feature match of the Azure cloud.

Microsoft argue that the real value of this offer stem from the combination of scale, automation and app development capabilities that has been evolved from the Azure platform. And that Microsoft is bringing these to the enterprise through Azure Stack.

In reality, this offer is a private cloud for public cloud Trojan horse. From this venture, Microsoft will be able to gather immense value from capturing workload informations to via their APIs while being adjacent to customer data. Not only it allows Microsoft to tailor Azure in order to ease its cloud service adoption. But also allows them to gather invaluable information about potential services that could be internalized within its own cloud solution via an ecosystem play. To a certain extend, it permit to outplay Amazon by reaching directly into its customer premises without having to upfront the CAPEX.

Some may argue that that Microsoft is late in deploying this strategy and I tend to agree with the analysis. But, I would also argue that Microsoft might try to jump an evolution step in the history of computing. One of the possibility is that Microsoft is looking at capturing the future market of cloud near shoring via AzureStack.

What is cloud near shoring you might ask : it is the opposite of cloud bursting. It allows company to move critical workload and/or data closer to the end user, in this case literally within the company itself while retaining the majority of its operations within a public cloud.
You have to remember that we are still bounded by the law of physics. We cannot transfer information faster than the speed of light and as a result with have a physical restriction when it comes to bandwidth and latency. Moreover new workloads are emerging that will require the presence of closely geo-replicated assets which will bootstrap the need for such cloud usage patterns. Immersive VR is one example, another is the real time business analytic combine with machine intelligence. In the future, companies might be hosting 90% or more of their workload + data in the cloud while running part of the services closer to where it is needed.

The possibilities around this concept are quite vast:  you can offer on premise or near premise near shoring solution via a form of CDN for cloud workload : CWDN or you Microsoft can resell for you your unused private cloud ressource to local user by example. 
As you can see, the original intent of enabling private public cloud transition via an on premise cloud might just be a deceiving late move enabling the capture of the next cloud market use case for Microsoft.

Tuesday, February 02, 2016

[Links of the day] 02/02/2016: Fosdem 2016

Fosdem 2016 just finished a couple of day ago, here are some selected bits:

Las but not least here are all the videos.

Monday, February 01, 2016

[Links of the day] 01/02/2016: Linux internals, best paper and Spotify goes SDN

  • Best Papers : Best Paper Awards in Computer Science (since 1996)
  • Linux Internals : very good ebooks on the internal of linux kernel from boot to memory
  • SDN Internet Router [part2] : a very impressive demonstration of the implication of the SDN technology for company. It allowed spotify to replace routers that would have cost 1/2 $M each with a couple of SDN switches.