- Aphyr Scala Day 17 : Aphyr breaks database for a living and then talks about it :)
- Sia : a Blockchain-based marketplace for file storage, the really attractive thing is the cost comparison of SIA vs public cloud system. Which is between a tenth to a hundredth time cheaper than S3 or other similar solution. I would be curious to see the performance thought.
- How brains are built: High-level overview of principles of computational neuroscience.
Wednesday, April 26, 2017
Monday, April 24, 2017
[Links of the Day] 24/04/2017 : Data Center Perf Index, Physical Limits of Computing And DNS as Code
- Data Center Performance Index : Performance index aiming at providing a reliable idea of the performance and efficiency of the datacenter. It primarily focuses on Availability, Efficiency and Environmental impact. This effort is lead by Dean Nelson, Uber Had of computing.
- Physical Limits of Computing : A look at the limitation of compute from a physicist point of view. It seems that some limitations are fundamental and require a new and different approach in order to create compute device that goes around these limitations.
- DnsControl : system built to manage DNS systems. To some extent, it looks like a terraform for DNS where you can plug multiple DNS backend providers. This allows you to deploy and distribute your DNS infra in an agnostic way across multiple cloud provider. [github]
Friday, April 21, 2017
[Links of the Day] 21/04/2017 : HPC 2017 trends, Docker cheat-sheet, Incident response best practice
- Current Trends in High-Performance Computing and Challenges : Jack Dongarra annual HPC review, It's amazing how the chinese progressed. They literally took over the top 500 in less than ten years. And now they dominate using homegrown chips and network fabric. [slides]
- Docker cheatsheet : 'nough said.
- Increment - On-call : New magazine providing article on how to scale companies. Each edition focus on a different topic. For the inaugural issue, they focus on industry best practices around on-call and incident response.
Wednesday, April 19, 2017
[Links of the Day] 19/04/2017 : AMD ROCm GPU open platform, Weak Memory Models concurrency report, SSH server for distributed infrastruscture
- ROCm : this slide deck give an overview of the AMD ROCm open platform for GPU computing exploration. They are really pushing to become the open source standard for the GPU industry battling against NVIDIA supremacy in the domain. It looks like they are making really good progress and I would be curious to see how this progress when combining with their Ryzen CPU.
- Concurrency with Weak Memory Models : this is a really good report on the state of memory models in hardware and software. It provides a wide spectrum overview of Hardware and Software concurrency model and approaches as well as the future direction in the domain.
- Teleport 2 : a modern SSH server designed for teams managing distributed infrastructure. [github]
Monday, April 17, 2017
[Links of the Day] 17/04/2017 : Pedis Redis Clone, Serverless framework, Deep learning best practices
- Pedis : Redis Compatible NoSQL datastore using the Seastar Framework. It's interesting to see that on the single thread benchmark Redis and Pedis are on par while it Redis gets smoked on 8 thread benchmark. However on a side note, the author should probably have chosen another name for project.
- serverless : Serverless Framework with serverless architectures using AWS Lambda, Azure Functions, Google CloudFunctions [github]
- Best Practices for Applying Deep Learning to Novel Applications : this is pretty much a must read for machine learning expert using deep learning. This report decomposes deep learning project in phases and provides best practice for each phase.
Friday, April 14, 2017
- OpenFabrics Workshop 2017 : Some interesting talk this year at the open fabric conference:
- uRDMA : Userspace RDMA using DPDK. This opens up a certain amount of possibility, especially for object storage solution. [Video , Slides, github]
- Crail : Using urdma above to deliver accelerated storage solution for Apache big data projects [Slides, github]
- Remote Persistent Memory: I think this is the next killer app for RDMA. If Intel doesn't jump onto it and deliver a dpdk like solution. [Video, Slides]
- On Demand paging: slowly the tech is crawling its way up to upstream acceptance. While on-demand paging introduces a certain performance cost. It also allows a greater flexibility in consuming RDMA. One of the interesting aspects that nobody mentioned yet is how this feature could be used with persistent memory. I think that there is some good potential for p2p NVM storage solution.[Video, Slides]
- Containerd : Containerd move to github, the docker "industry standard" container runtime is also reaching its v.0.2.x release. [github]
- Category Theory : If you are into functional programming and Haskell. This is a must read book for you.
Wednesday, April 12, 2017
[Links of the Day] 12/04/2017: Linux Perf tools, libp2p and Contagion of Information in Social Media
- Perf Tools : miscellaneous collection of in-development and unsupported performance analysis tools for Linux ftrace and perf_events.
- Contagion of Information in Social Media : The authors look at how information spread on social media ( twitter ). The authors model contagion behaviour in the hope to create effective defences against "fake news" and other propaganda. However, to some extend the research can also be used to optimise the spread of such malicious information.
- libp2p : really cool network stack ( used by IPFS) that tackle a lot of the nitty gritty detail of p2p applications. It should allow devs to focus on the actual value of their p2p apps rather than the technical underlying problems of p2p itself. [github]
Monday, April 10, 2017
[Links of the Day] 10/04/2017 : Loopy , Distributed execution engine and Yet another distributed ledger algo
- Loopy : Fantastic tools for explaining and describing complex system interaction. It's easy to use and even easier to get the message across [github]
- Ray : experimental distributed execution engine replicating code across multiple workers. Written in python it leverages object store and distributed task execution to achieve parallelism. I really wonder if it wouldn't have been better to code Ray using AWS lambda and S3.
- Algorand : yet another distributed ledger. The approach proposes to eliminate the segregation of actors in the ledger system. No more miners and users, everybody is an equal participant. Moreover, it relies on a new for of Byzantine agreement ( need a TLA+ proof to really feel comfortable with that) and cryptographic selection algorithm for selecting leader (verifier) of the ledger process execution. This is a rather interesting paper and I will try to produce a short summary of it if I find the time.
Friday, April 07, 2017
[Links of the Day] 07/04/2017 : TensorFlow Example, Systems in the microseconds era and blockchain distributed direct democracy
- Naked Tensor : Google tensor flow bare-bone example.
- Attack of the Killer Microseconds : Hardware ( especially storage ) is entering the micro or sub-micro second era. This has far and wide ranging implication. And system designer needs to rethink the existing stack that was designed for the millisecond era. It looks like we are entering an era where the software stack is not the bottlenneck.
- Cicada : Distributed secure proof of work blockchain combined with a privacy guaranteed ID system. The creator of cicada aim at enabling distributed direct democracy and decentralised application platform. This is worthy goals, however, the creator forgot that direct democracy tend to fail as the majority of the population is not interested or knowledgeable enough in the problem they will be asked to vote on.
Wednesday, April 05, 2017
[Links of the Day] 05/04/2017 : Large network resilience, Distributed Systems, Machine Learning & Bayesian reasoning
- Understanding resilience in large networks : Thesis looking into the analysis of robustness in large interconnected networks. It looks at how disturbance propagates through nodes in networks and the characteristics that help these network to handle such shock and recuperate from it.
- Notes on Theory of Distributed Systems : Yale distributed system course note.
- Bayesian Reasoning and Machine Learning : Book that does an excellent job at teaching machine learning that doesn't assume an extended background in undergraduate calculus and linear algebra of the reader.
Monday, April 03, 2017
[Links of the Day] 03/04/2017 : Conway's Game of life Clock, Human-Bot social interaction, SQL time series DB
- Digital clock in Conway's Game of Life : I can't even start to comprehend how you can design this. But this is beyond cool.
- Online Human-Bot Interactions: Detection, Estimation, and Characterization : An analysis of bots on socials network (twitter). I think we need a reverse Turing test. When a robot can detect when they talk to a human.... Reverse captcha to weed out that pesky meat-bag from meddling from our robotic overlord affairs.
- Timescale : SQL compatible time series database. Another competitor for Influxdb. Let's just say that the clustering feature will make or break it as Influxdb has some serious issue there [github]
Friday, March 10, 2017
[Links of the Day] 10/03/2017 : User-space SysFS, Key Value consensus Algo, Cost efficient Big Data Serverless Framework
- ProcStat : Userspace equivalent of kernel SysFS. Really cool project by my friend Sasha. makes it really easy to expose internal counter and state of a process via FUSE
- Bizur : Key-value Consensus Algorithm using a nice solution where consensus is achieved on the key themselves rather than relying on a globally distributed log. The great aspect is that recovery and failure management is greatly simplified and streamlined. However, it implies that the progress and consensus on each key are independent of each other. As a result, you cannot rely on serialisation of state between key. Which can be limiting if you expect the state of Key A to be changed after the State of Key B by example.
- PyWren : Framework that let you use serverless functions for cheap large-scale data analysis. [github]
Wednesday, March 08, 2017
[Links of the Day] 08/03/2016 : Intel blockchain, Fast17 conference and papers, AWS cloud formation devops tool
After a small hiatus, here is the return of the links of the day.
- Sawtooth Lake: Intel distributed ledger system. It uses an interesting security mechanism to deliver secure consensus. Sadly it relies on Intel proprietary hardware encryption modules to deliver this feature.
- Fast17: File and Storage technology Usenix conference happened last month. There were a couple of interesting papers but one picked my interest: Redundancy Does Not Imply Fault Tolerance:Analysis of Distributed Storage Reactions toSingle Errors and Corruptions. The authors look at single file system fault impact on Redis, ZooKeeper, Cassandra, Kafka, RethinkDB, MongoDB, LogCabin, and CockroachDB. Turns out most systems are not able to handle these type of faults very well. It seems that a single node persistency layer error can have an adversarial ripple effect as distributed system seems to have put way to much trust in the reliability of this layer. Sadly they lack tools for recovering from errors or corruption emerging from file systems.
- Stacker : remind 101 tools for creating and updating AWS formation stacks. Looks like an interesting alternative to terraform.
Tuesday, January 24, 2017
- Numpy Cheat Sheet: all you need for data analysis in python with NumPy
- Mathematical Model of innovation patterns: Vittorio Loreto at the Sapienza University of Rome in Italy and al, created the first mathematical model that accurately reproduces the patterns that innovations follow.
- Persistent Memory Summit: SNIA NVM summit 2017. Finally, with the introduction of Intel 3dXpoint, we start to see more HW NVM solution out there. And with that software that uses it. Some really interesting talks:
- Nova file system demonstrates the benefit of NVM optimises storage solution.
- SAP Hana on NVM: interesting to see that they still require redundant copies as they fear data corruption on NvDimms. I wonder when we will start to see ECC NvDimms on the market?
- New interconnect: lots of hot new interconnects battling for the heterogeneous compute ecosystem domination. And this pass by offering persistent memory specific solution: Gen-z provide PM pooling, Open CAPI accelerate PM access and CCIX share PM
Wednesday, January 18, 2017
[Links of the Day] 18/01/2017 : Multi-tenant K/V cache, Http Tunnel , Google Infrastructure security
- Memshare: Multi-tenant in-memory key value store, the authors target specifically web caching use case. It is interesting to see that they are using log-structured for maximising memory usage and hit ratio. However, the really novel approach is that it allow each application to define its own eviction policy.
- Chisel: Interesting tunnel approach over Http, to some extent similar to coding but with a different approach. I really like that is provide something more akin to crowbar for firewall bypass and with out of the box encryption. Also, it seems to be a lot faster that other tunnel out there.
- Google Infrastructure Security Design: Google approach to security is really interesting. While it makes great use of hardware security feature it also leverages a more software-defined security approach allowing them to have multiple lines of defences stacked between each communicating component while eliminating a lot of the restriction that often scleroses highly secure infrastructure.
Wednesday, January 11, 2017
The list of self-driving vehicles and companies starting to offer services of these vehicles is ever growing. I recently started to be interested in the ancillary challenges brought by the deployment of a fleet of autonomous vehicle and the business opportunity that emerge.
There is two interesting business area with a certain potential: cleaning services and real estate.
Support service ecosystem:
Supports services are the main area of expansion created by companies like Uber, Google making a foray in the autonomous ride share business model. They will more likely outsource these operation to third parties as it these business has a low-profit margin and tends to be hard to automate (as in requiring manual labour).
I have three daughters under 5 and, let's face it, my car is a mess. It takes less than 2 rides to transform a clean spotless car interior in the equivalent of the Omaha beach d-day aftermath. And this is the same for taxi / uber drivers, the current best practice recommendation is to have cleaning implements and a throw-up bag at all time in the car in order to maintain high standard and rating. Not to mention the cleaning fee if things go really bad.
Now if you have an autonomous car, you will need to have it clean often as they will be providing ride 24/7.
Repair and maintenance requirements are obviously another areas that will need to be developed. By example, In Uber current model, the cleaning, repairing and refuelling is the responsibility of the owner of the car. However, when shifting to the autonomous ride, Uber will start to need to, either provide this service internally or outsource it.
Refuelling and recharging might be less of a problem as there is a clearer way of automating the process.
Real estate issue:
Another side effect is that for cleaning, recharging and repairing operations require real estate. You cannot deliver these service in the middle of the street. And this is another problem that corporations will have to solve. To some extent Google and Uber are trying to go around this issue by deploying their solution first in confined areas like college campuses, military bases or corporate office parks. As the owner of these private space will be able to provide space for free in order to benefit from the service. However, as they expand outside, this will become more problematic. Moreover, they might want to have buffer zone where the fleet of vehicles is at rest in off-peak periods.
Being able to deliver efficiently the logistic for support service while maximising resource efficiency will literally make or break the business model of autonomous rideshare. One possibility would be for these companies to contract, uber style, individual to offer their driveway and cleaning/refuelling services. Companies will be able to use the cleanliness rating made by the customer to evaluate the service quality of the individuals. This would partially solve the real estate issue until town planner starts to accommodate this new mode of transport. It will also allow to grow cheaply a widely distributed service point location. Enabling just in time servicing, hence maximising car usage efficiency.
To some extend the new business model deployed by the like of Uber couple with the commoditization of service create new business opportunity for ancillary support services. Unsurprisingly, these services can copy or adapt the same business model to scale while keeping cost down. However, it might be a little bit too early for these to blossom as we haven’t reached peak Uber fade and fleet of self-driving cars are a couple of years away. I would probably keep an eye instead on the less glamorous but potentially more lucrative self-driving trucks business instead.
Monday, January 09, 2017
[Links of the Day] 09/01/2017 : Incident Response process, Plain english Legal guide to start a startup, 33C3 videos
- Incident Response : pager duty incident response documentation. This is a very thorough and well documented process for handling incident, before during and after their occurrence. Probably not one size fit all by can be easily adapted to an company needs.
- Plain English legal guide on how to start a business : nice guide of the legal aspect of how to start a startup and the various options and pitfall associated with it. A little bit too US centric for my taste but still has some good insights.
- 33C3 :Chaos Computer Congress videos are now available. Chris Hager gave a great overview of the different talks. As always a lot of diversity and challenging presentations.
Friday, January 06, 2017
[Links of the Day] 06/01/2017 : System we love videos, Distributed programming book and all the code from NISP16 papers
- Systems we love : Video of the recent systems we love conference. Checkout the lessons from the cell one.
- Distributed programming book : yet another open source book on distributed systems.
- Code of NIPS16 : all code from the machine learning NIPS16 conference. Finally we start to see some traction to make code available alongside papers. I think all top conference should make mandatory that the code needs to be available on publication date. We don't care if its not production ready or nice.. We just want to see how these wonderful ideas translate in real code.
Wednesday, January 04, 2017
- ProxySQL : SQL proxy fir MySQL ( or any other fork like Percona and Mariadb) [Github]
- Smudge : Go library providing group member discovery, status dissemination and failure detection using the SWIM epidemic protocol. This is really cool as provide a building block for an equivalent to consul but with a very low footprint ressource and network wise.
- Hypervisor costs : interesting comparison of hypervisor solution costs. The surprising number is that contrary to popular belief VMware is not the most expansive but Hyperv is on a apple to apple comparison. However I am not sure that this hold true when you are talking for a complete solution.
Monday, January 02, 2017
- Cloud Cron : cool tool for executing cron job on cloud
- KSM : while the name might be confusing ( KSM also stand for kernel shared memory). KSM is a neat small hypervisor and support a lot of hardware feature.
- Technology Radar : ToughtWorks maintained technology landscape. Really useful to spot what up and coming tech are been baked by different startups. Also at what stage are the different technologies