- ProcStat : Userspace equivalent of kernel SysFS. Really cool project by my friend Sasha. makes it really easy to expose internal counter and state of a process via FUSE
- Bizur : Key-value Consensus Algorithm using a nice solution where consensus is achieved on the key themselves rather than relying on a globally distributed log. The great aspect is that recovery and failure management is greatly simplified and streamlined. However, it implies that the progress and consensus on each key are independent of each other. As a result, you cannot rely on serialisation of state between key. Which can be limiting if you expect the state of Key A to be changed after the State of Key B by example.
- PyWren : Framework that let you use serverless functions for cheap large-scale data analysis. [github]
Friday, March 10, 2017
[Links of the Day] 10/03/2017 : User-space SysFS, Key Value consensus Algo, Cost efficient Big Data Serverless Framework
Wednesday, March 08, 2017
[Links of the Day] 08/03/2016 : Intel blockchain, Fast17 conference and papers, AWS cloud formation devops tool
After a small hiatus, here is the return of the links of the day.
- Sawtooth Lake: Intel distributed ledger system. It uses an interesting security mechanism to deliver secure consensus. Sadly it relies on Intel proprietary hardware encryption modules to deliver this feature.
- Fast17: File and Storage technology Usenix conference happened last month. There were a couple of interesting papers but one picked my interest: Redundancy Does Not Imply Fault Tolerance:Analysis of Distributed Storage Reactions toSingle Errors and Corruptions. The authors look at single file system fault impact on Redis, ZooKeeper, Cassandra, Kafka, RethinkDB, MongoDB, LogCabin, and CockroachDB. Turns out most systems are not able to handle these type of faults very well. It seems that a single node persistency layer error can have an adversarial ripple effect as distributed system seems to have put way to much trust in the reliability of this layer. Sadly they lack tools for recovering from errors or corruption emerging from file systems.
- Stacker : remind 101 tools for creating and updating AWS formation stacks. Looks like an interesting alternative to terraform.
Tuesday, January 24, 2017
- Numpy Cheat Sheet: all you need for data analysis in python with NumPy
- Mathematical Model of innovation patterns: Vittorio Loreto at the Sapienza University of Rome in Italy and al, created the first mathematical model that accurately reproduces the patterns that innovations follow.
- Persistent Memory Summit: SNIA NVM summit 2017. Finally, with the introduction of Intel 3dXpoint, we start to see more HW NVM solution out there. And with that software that uses it. Some really interesting talks:
- Nova file system demonstrates the benefit of NVM optimises storage solution.
- SAP Hana on NVM: interesting to see that they still require redundant copies as they fear data corruption on NvDimms. I wonder when we will start to see ECC NvDimms on the market?
- New interconnect: lots of hot new interconnects battling for the heterogeneous compute ecosystem domination. And this pass by offering persistent memory specific solution: Gen-z provide PM pooling, Open CAPI accelerate PM access and CCIX share PM
Wednesday, January 18, 2017
[Links of the Day] 18/01/2017 : Multi-tenant K/V cache, Http Tunnel , Google Infrastructure security
- Memshare: Multi-tenant in-memory key value store, the authors target specifically web caching use case. It is interesting to see that they are using log-structured for maximising memory usage and hit ratio. However, the really novel approach is that it allow each application to define its own eviction policy.
- Chisel: Interesting tunnel approach over Http, to some extent similar to coding but with a different approach. I really like that is provide something more akin to crowbar for firewall bypass and with out of the box encryption. Also, it seems to be a lot faster that other tunnel out there.
- Google Infrastructure Security Design: Google approach to security is really interesting. While it makes great use of hardware security feature it also leverages a more software-defined security approach allowing them to have multiple lines of defences stacked between each communicating component while eliminating a lot of the restriction that often scleroses highly secure infrastructure.
Wednesday, January 11, 2017
The list of self-driving vehicles and companies starting to offer services of these vehicles is ever growing. I recently started to be interested in the ancillary challenges brought by the deployment of a fleet of autonomous vehicle and the business opportunity that emerge.
There is two interesting business area with a certain potential: cleaning services and real estate.
Support service ecosystem:
Supports services are the main area of expansion created by companies like Uber, Google making a foray in the autonomous ride share business model. They will more likely outsource these operation to third parties as it these business has a low-profit margin and tends to be hard to automate (as in requiring manual labour).
I have three daughters under 5 and, let's face it, my car is a mess. It takes less than 2 rides to transform a clean spotless car interior in the equivalent of the Omaha beach d-day aftermath. And this is the same for taxi / uber drivers, the current best practice recommendation is to have cleaning implements and a throw-up bag at all time in the car in order to maintain high standard and rating. Not to mention the cleaning fee if things go really bad.
Now if you have an autonomous car, you will need to have it clean often as they will be providing ride 24/7.
Repair and maintenance requirements are obviously another areas that will need to be developed. By example, In Uber current model, the cleaning, repairing and refuelling is the responsibility of the owner of the car. However, when shifting to the autonomous ride, Uber will start to need to, either provide this service internally or outsource it.
Refuelling and recharging might be less of a problem as there is a clearer way of automating the process.
Real estate issue:
Another side effect is that for cleaning, recharging and repairing operations require real estate. You cannot deliver these service in the middle of the street. And this is another problem that corporations will have to solve. To some extent Google and Uber are trying to go around this issue by deploying their solution first in confined areas like college campuses, military bases or corporate office parks. As the owner of these private space will be able to provide space for free in order to benefit from the service. However, as they expand outside, this will become more problematic. Moreover, they might want to have buffer zone where the fleet of vehicles is at rest in off-peak periods.
Being able to deliver efficiently the logistic for support service while maximising resource efficiency will literally make or break the business model of autonomous rideshare. One possibility would be for these companies to contract, uber style, individual to offer their driveway and cleaning/refuelling services. Companies will be able to use the cleanliness rating made by the customer to evaluate the service quality of the individuals. This would partially solve the real estate issue until town planner starts to accommodate this new mode of transport. It will also allow to grow cheaply a widely distributed service point location. Enabling just in time servicing, hence maximising car usage efficiency.
To some extend the new business model deployed by the like of Uber couple with the commoditization of service create new business opportunity for ancillary support services. Unsurprisingly, these services can copy or adapt the same business model to scale while keeping cost down. However, it might be a little bit too early for these to blossom as we haven’t reached peak Uber fade and fleet of self-driving cars are a couple of years away. I would probably keep an eye instead on the less glamorous but potentially more lucrative self-driving trucks business instead.
Monday, January 09, 2017
[Links of the Day] 09/01/2017 : Incident Response process, Plain english Legal guide to start a startup, 33C3 videos
- Incident Response : pager duty incident response documentation. This is a very thorough and well documented process for handling incident, before during and after their occurrence. Probably not one size fit all by can be easily adapted to an company needs.
- Plain English legal guide on how to start a business : nice guide of the legal aspect of how to start a startup and the various options and pitfall associated with it. A little bit too US centric for my taste but still has some good insights.
- 33C3 :Chaos Computer Congress videos are now available. Chris Hager gave a great overview of the different talks. As always a lot of diversity and challenging presentations.
Friday, January 06, 2017
[Links of the Day] 06/01/2017 : System we love videos, Distributed programming book and all the code from NISP16 papers
- Systems we love : Video of the recent systems we love conference. Checkout the lessons from the cell one.
- Distributed programming book : yet another open source book on distributed systems.
- Code of NIPS16 : all code from the machine learning NIPS16 conference. Finally we start to see some traction to make code available alongside papers. I think all top conference should make mandatory that the code needs to be available on publication date. We don't care if its not production ready or nice.. We just want to see how these wonderful ideas translate in real code.