CryptMPI: A Fast Encrypted MPI Library

 As more #HPC applications move to cloud infrastructure, securing and protecting HPC sensitive data in such an environment becomes critical.

But HPC solution tends to fall short when it comes to security. Security features tend to be perceived as detrimental to the performance of the applications.

By example, encrypted communication has always been seen as incurring very significant overheads when you are aiming for microsecond latency.

The author of the Crypt MPI paper demonstrates that you can ensure the privacy and integrity of sensitive data with minimal performance degradation using an enhance MPI library.

I hope to see this kind of feature integrated as standard in a future version of MPI.

Paper: https://arxiv.org/abs/2010.06471

Code: https://github.com/abu-naser/CryptMPI-A-Fast-Encrypted-MPI-Library

[Links of the Day] 12/12/2019 : Edge Neural Network attack surface survey, Multidimensional Indexes for K/V store, Deploy your own SaaS

• Survey of Attacks and Defenses on Edge-Deployed Neural Networks : As machine learning becomes more pervasive in everyday technology. We need to start looking into the attack surface area of these techs. And, ho boy... It's not only huge but there is a lot of hidden potential in there. The authors do a really good job of looking at the current state of play on edge deployed neural network security.
• Learned Multi-dimensional Indexes : The authors propose Flood, a multi-dimensional in-memory index that automatically adapts to the dataset and workload. It seems like a fantastic upgrade for standard k/v store. K/V store has always been terrible at range scan and other multidimensional operations. I wonder if this technique can be easily adapted to Cassandra / Scylladb. This might deliver a significant boost for these DB systems. 
• Deploy-your-own-SaaS : Deploy your own Saas, just for you. Cool Templates repo with tons of SaaS service for personal ( or family / small group ) use. Chat server, Music streaming, website, adblock, etc.. you name it. 

[Links of the Day] 14/02/2019: golang fast HTTP , virtio shared file-system, enclave application

• FastHTTP : like net/http but up to 10x faster . Zero memory allocation on a fast path. When you really really need to go fast and the standard golang lib is not enough.
• Virtio-fs : shared file system build on top of the virtio framework. Finally a good way of sharing files and data between the host and the guest VM or between guests [git].
• Asylo : I recently came across the concept of enclave applications. An enclave is a special execution context where code can run protected from even the OS kernel, with the guarantee that even a user running with root privileges cannot extract the enclave’s secrets or compromise its integrity. It relies on Intel SGX or ARM trustzone security hardware feature to protect the runtime operation of the process. This might be the next security level for highly regulated industries that want to deploy in a public cloud environment which will completely eliminate a lot of the conundrum that they currently face.

[Links of the Day] 31/01/2019 : Evidence based management, real time data streams analysis, proxy firewall with deep protocol analysis

• Evidence-Based Management Guide : agile meet KPI scorecards ... Some interesting concept, but I am wary of the current trends of anything can be solved by X + Z in management... 
• Confluo : real-time distributed analysis of multiple data streams. Confluo simultaneously supports high throughput concurrent writes, online queries at millisecond timescales, and CPU-efficient ad-hoc queries via a combination of data structures carefully designed for the specialized case of multiple data streams, and an end-to-end optimized system design.
• Zorp : open source proxy firewall with deep protocol analysis. If you want to be nice and cosy on a deep and meaningfull level with your traffic. This is for you. With Zorp you can control, inspect and modify traffic at the application layer.

[Links of the Day] 15/01/2019 : Incident Response best practice, Database Schema Crawler, Fingerprinting TLS

• ja3 : something I discovered recently. Apparently, you can fingerprint SSL and TLS session in order to identify the service being run behind the encrypted socket. Really awesome if you want to spot malware or bitcoin miner on your network. Or pretty much any other services as long as you have a fingerprint to compare with.
• SchemaCrawler : a cool tool for database schema discovery. This is a must when you have to take on board a legacy DB system that lacks clear documentation. 
• Incident response : pager duty open sourced they incident response process. This is a really great set of tools, process and best practice for incident response. What is even more eye-opening is the part the describe the incident resolution scenario that didn't work and point out some great anti-patterns. A must read for any SRE team out there and anybody else that has an on-call duty and their managers.

[Links of the Day] 10/01/2019 : High performance stream engine, Golang security links and a modern back orifice written in Go

• Trill: high-performance one-pass in-memory streaming analytics engine. This seems like a highly versatile and performant streaming engine. The team behind it is making some bold claims regarding its capability ( see table below). I think that this architecture is promising, however, the .net language might put some people off.
• Go Security link : loads of golang related security links 
• Merlin : a cross-platform command and control server and agent. If you have used back orifice in the 90s. You will know what this tool offers :)

[Links of the Day] 06/12/2018 : NLP summarisation, RPC protobuf framework, API security best practices

• Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting : I would have mentioned this paper on the sole basis that the authors provide a GitHub repo with all the code used. This should be mandatory for any publication in Computer science. Anyway, the summarization tech described in the paper is pretty cool too. [github]
• twirp : RPC framework with protobuf service definitions. If you don't want to go all gRPC, give this framework a serious look. I would consider twirp over gRPC for the sole reason that it uses the standard GO http server over the custom google one. Seriously google, why did you have to re-implement the HTTP server from your own language ?? 
• API Security Best Practices : documentations providing some good security practice when it comes GitHubhub usage. As well as an excellent leak management document describing a fairly efficient process.

[Links of the Day] 13/09/2018 : Kubernetes in Docker, Forensic Diffing AWS image, Consistent File system on top of S3

• kind : Kubernetes-in-Docker - A single node cluster to run your CI tests against that's ready in 30 seconds
• diffy : Diffy allows a forensic investigator to quickly scope a compromise across cloud instances during an incident, and triage those instances for followup actions.
• Snitch: Box created a Virtually Consistent FileSystem build on top of S3. An interesting solution that allows Box to prevent data loss by building a consistent solution using an eventually consistent storage. Sadly not open sourced... 

[Links of the Day] 05/07/2018 : Architecture of Open Source applications, Awesome CI , Searching for secrets in Git history

• Architecture of Open Source Applications : this is a fantastic set of books that go over the architecture of open sources application. From how programmers solve interesting problems in 500 lines of code (or less) to performance and architecture of open source project.
• Awesome-ci : awesome list of CI related articles, paper and tools. 
• gitleaks : a useful tool that searches your git repo history for secrets and keys.. Sanitize all the things !

A look at Google gVisor OCI runtime

Google released a new OCI container runtime: gVisor. This runtime aim at solving part of the security concerns associated with the current container technology stack. One of the big argument of the virtualisation crowd has always been that the lack of explicit partitioning and protection of resource can facilitate “leakage” from the containers to the host or other adjacent containers. 

This stem from the historical evolution of containers in Linux. Linux has no such concept of native containers. Not like BSD with Jails or Solaris with Zones. Containers in Linux are the result of the gradual emergence of a stack of various security and isolation technology that was introduced in the Linux kernel. As a result, Linux ended up with a highly broad technology stack that can be separately turned on / off or tuned. However, there is no such thing as a pure sandbox solution. The classic jack of all trade curse, it’s a master of none solution.

The Docker runtime (Containerd) package all the Linux kernel isolation and security stack (namespaces, cgroup, capabilities, seccomp, apparmor, SELinux) into a neat solution that is easy to deploy and use. 

It allows the user to restrict what the application can do such as which file it can access with which permission or limits resource consumption such as networks, disk I/O or CPU. It allows the applications to share resources without stepping on each other toes happily. Also, limits the risk any of their data being accessed while sitting on the same machine. 

With a correct configuration ( the default one is quite reasonable ) will allow blocking anything that is not authorised and in principle protect from any leak from a malicious or badly coded piece of code running in the container. 

However, you have to understand that Docker has some real limitation already. It has only limited support for user-namespace. The user-namespace allows applications to have UID 0 permission within the containers ( aka root ) while the containers and user running has a lower privilege level. As a result, each container would run under a different user ID without stepping on each other toes. 

All of these features rely on the reliability and security (as in no bugs) of the Linux kernel. Most of Docker advanced feature relies on kernel features. And getting new features is a multi-year effort, it took a while for good resource isolation mechanism percolates from the first RFC to the stable branch by example. As a result, Docker and current container ecosystem are directly dependent on the Linux kernel update inertia as well as its code quality. While being excellent, no system is entirely free of bug, not to mention the eternal race for patching them when they are discovered and fixed. 

Hence the idea is to, rather than having to share the kernel between all the containers, have one kernel per container. Explicitly limiting potential leakage, interference and reduce the attack surface. gVisor adopts this approach, which is not new as KataContainers already implemented something similar. Katacontainers is the result of the fusion of ClearContainer (intel) and runV (hyper). Katacontainers use KVM as a minimalistic kernel dedicated to the container runtime. But, you still need to manage the host machine to ensure fair resource sharing and their securitisation. This additional layer of indirection limits the attack surface as even if a kernel bug is discovered you will be challenged to exploit it to escape to another an adjacent container or underlying one as they are not shared. 

gVisor can use a KVM as kernel; however, it was initially and is still primarily designed around ptrace. User Mode Linux already used the same technique, which is to start a process in userspace for the subsystem that will be running on top. Similarly to a hypervisor model used by virtual machines. All the system calls will be executed using the permission of the userspace process on behalf of the subsystem via an interception mechanism. 

Now, how do you intercept these system calls which should be executed by the kernel? UML and gVisor divert ptrace primary goal ( which is debugging ) and inject a breakpoint in the executable code to intercept and stop for every system call execution. Once caught the new userspace kernel will execute the call on behalf the original process within userspace. It works well, but as you guessed, there is no free lunch. This method what heavily used by the first virtualisation solution. But rapidly, processor vendors realised that offering hardware-specific acceleration method would be highly beneficial ( and sell more at the same time).

KVM and other hypervisor leverage such accelerator. Now you even have AWS and Azure deploying completely dedicated coprocessor for handling virtualization related acceleration. Allowing VM to run almost that the same speed as a bare metal system. 

And like Qemu leveraging KVM, gVisor also offer KVM as underlying runtime environment. However, there is significant work to be done to enable any container to run on to of it. While ptrace allow to directly leverage existing Linux stack, with KVM you need need to reimplement a good chunk of the system to make it work. Have a look at Qemu code to understand the complexity of the task. This is the reason behind the limited set of supported applications as not all syscalls are implemented yet.

As is, gVisor is probably not ready yet for production. However, this looks like a promising solution providing a middle ground between the Docker approach and the Virtualization one while taking some of the excellent ideas coming from the unikernel world. I hope that this technology gets picked up, and the KVM runtime becomes the default solution for gVisor. It will allow the system to benefit from a rock-solid hardware acceleration with all the paravirtualisation goodies such as virtio. 

[Links of the Day] 18/01/2017 : Multi-tenant K/V cache, Http Tunnel , Google Infrastructure security

• Memshare: Multi-tenant in-memory key value store, the authors target specifically web caching use case. It is interesting to see that they are using log-structured for maximising memory usage and hit ratio. However, the really novel approach is that it allow each application to define its own eviction policy. 
• Chisel: Interesting tunnel approach over Http, to some extent similar to coding but with a different approach. I really like that is provide something more akin to crowbar for firewall bypass and with out of the box encryption. Also, it seems to be a lot faster that other tunnel out there. 
• Google Infrastructure Security Design: Google approach to security is really interesting. While it makes great use of hardware security feature it also leverages a more software-defined security approach allowing them to have multiple lines of defences stacked between each communicating component while eliminating a lot of the restriction that often scleroses highly secure infrastructure.  

[Links of th Day] 16/11/2016 : Open Whisper, Security via BPF + XDP, Why cloud fails

• X3DH : stand for Extended Triple Diffie-Hellman key agreement protocol. It allow asynchronous secure communication.
• Cilium : BPF start to emerge as the dominant tool for any network functionality out there. Google cilium which leverage BPF and XDP for enforcing dynamic security rules via eBPF. [github]
• Why Cloud Fails : Excellent review by Murat on the paper analyzing various cloud failures . Turn out that the vast majority are config related and upgrade related. Human failure only represent a very small portion of the overall numbers. [paper]

[Links of the Day] 09/11/2016 : Deep Neural Net Threats, Scaling Uber, Tcp over Sound

• Assessing Threat of Adversarial Examples on Deep Neural Networks : machine learning is the next frontier for hacker. And because of its inherent opacity it requires special capabilities to secure system that relies on this underlying technology. This paper show that for text driven classification, adversarial exemple are more an academic curiosity than a real threat. However, we need to see if this can be applied to other type of classification. 
• Lesson learns about scaling Uber : Many talk are about scaling, however most company and startup would love to have those problems. Often its not about scaling, its about having the right product market fit. Then you can enjoy the roller coaster of scaling problems. 
• Quiet : TCP over sound . This is really cool, it allows to pass data through speakers on android devices.

[Links of the day] 10/12/2016 : Full stack fest 16, Product Dev handbook and Stealing Machine Learning Models

• Full Stack Fest 2016 : Playlist of the video of Full Stack fest 2016 conference
• Product Aikido : handbook for an organisation’s Product Development Group.
• Stealing Machine Learning Models : the authors of the papers propose a technique that analyse response of system using Machine learning via their API in order to extract the model used. And as a result allow the attacker to determine the best response for manipulating the system.

[Links of the Day] 02/09/2016 : Row Hammer in the cloud , Usenix Security conf and Agile IT Book

• USENIX Security '16 : usenix security conference proceedings are out, there is two notable paper in there: 
• Off-Path TCP Exploits: Global Rate Limit Considered Dangerous : it seems that there is a flaw in current TCP sepcificatin and implementation that  allows a blind off-path attacker to infer if any two arbitrary hosts on the Internet are communicating using a TCP connection. This could allow large scale denial of service or worse . 
• One Bit Flips, One Cloud Flops: Cross-VM Row Hammer Attacks and Privilege Escalation : this one sound scarier than the precedent paper. However the inherent limitation of the approach reduce the potential scope of attack. It basically leverage Row Hammer procedure to attack neighbor VM within a cloud system. However, the caveat is that it requires that the cloud provider allow to run paravirtualised guest, can only target PV guest. And is easily defeated if the RAM used is non ECC. Which is the default RAM used in any decent data-center. 
• Agile IT Management: From Startup to Enterprise :  really good book providing a well documented set of observations on IT’s current challenges that can orient you for more effective decisions and actions in your journey toward IT excellence. But beware of buying too much of the hype of one solution fits all. Agile is just one part of the solution, old waterfall still has its place as well as intermediary one. The makes the difference between wisdom and knowledge. 

[Links of the day] 16/03/2016: userspace TCP stack, pebcak, neural net

• mTCP : Highly Scalable User-level TCP Stack for Multicore Systems, really cool project released back in 2014. The really interesting bit is that they now support DPDK natively which allow for massive performance improvement while bypassing the kernel stack altogether. [github]
• The problem is between the chair and the keyboard : how do you patch users ? A Taxonomy of Attacks and a Survey of Defence Mechanisms for Semantic Social engineering Attacks
• Neural Networks Demystified : series of video explaining neural networks

[Links of the day] 19/02/2016 : Machine Learning Intro, Fungal Network and Usenix Enigma

• Machine Learning : A gentle & structured introduction describing the types of learning, Machine Learning problems, recent examples/applications and methods for optimizing algorithms.
• Anti-jamming in a fungal transport network : fungal network can help design next generation dynamic network. 
• Enigma  : proceedings of the 2016 Usenix conference on emerging threats and novel attacks are available. As well as the videos

[Links of the day] 09/02/2016: Run openstack in containers, Flocon network security conf, Agent motion book

• Kolla : run Openstack in containers for ease of management and operational flexibility.
• Flocon 2016 presentation : Network Situational Awareness for large-scale network flow analytics.
• Space & motion of communicating agents : CS thesis looking at leveraging bigraph to model the space , motion of communicating agents. This essential as for the pas decades system mainly focused on getting the itnernal communication of entity right. But with the upcoming Punctuated equilibrium of IoT we external communication systems in an non static world will become critical.

[Links of the day] 05/02/2016: Cloudlet server image, Shmoocon 2016, Multidimensional queries

• Multidimensional Access Methods : 1996 paper showing how hard it is to resolve multidimensional queries . List the state of the art approach which are still relevant to date.
• Cloudlets : I think I linked to cloudlets before, but I'll do it again jsut to remind myself to actually use them. Cloudlets are universal server images for the cloud. They're lightweight, version-controlled, and you can export them to any bootable format known to man: Xen, KVM, Amazon EC2, or just a plain bootable CD.
• Shmoocon 2016 : excellent security conference , especially check out the post quantum crypto talk by Jean-Philippe Aumasson 

Bonus : video demonstrating CISE tool for proving distributed systems correctness

Links of the day 10/11/2015: #containers, #Unikernel & #virtualization power consumption, security, and streamlined

• Qubes with Rumprun : Interesting application of unikernel to deliver greater security at speed. the goal is to reduce the possible attack surface while maximizing performance.
• Power Consumption of Virtualization Technologies: empirical Investigation that confirm a lot of what we knew already, virtualization power consumption is higher than container.
• Vagga : containerisation tool without daemons. Slim down streamline version of container. No deamons or other process to run to support it. Looks like a solid contender to docker and other vagrant.