- Introduction to Apache Kafka : If you lived under a rock and don't know about kafka.
- Kafka at Scale : Interesting overview of running Kafka at Linkedin : 1100+ Kafka brokers - 675 Billion messages per day - 150 Terabytes In - 580 Terabytes Out - Peak Load – 10.5 Million messages/sec – 18.5 Gigabits/sec Inbound – 70.5 Gigabits/sec Outbound
- Fluent : videos of the 2015 Fluent Conference
- Vault : tool for securely managing secrets and encrypting data in-transit by HashiCorp
A blog about life, Engineering, Business, Research, and everything else (especially everything else)
Thursday, April 30, 2015
Links of the day 30 - 04 - 2015
Today's links 30/04/2015: Apache conf - kafka - more kafka - Fluent Conference,Vault keeping secret in the cloud
Labels:
cloud
,
conference
,
kafka
,
links of the day
,
secret
Wednesday, April 29, 2015
Links of the day 29 - 04 - 2015
Today's links 29/04/2015 : Concurrency vs parallelism, Reenix #Rust OS, Pulsar Real Time Analytics, Business Town
- Concurrency is not parallelism : very good talk by Rob Pike on the misconception that often occur when people hear the word concurrency they often think of parallelism, a related but quite distinct concept. In programming, concurrency is the composition of independently executing processes, while parallelism is the simultaneous execution of (possibly related) computations. Concurrency is about dealing with lots of things at once. Parallelism is about doing lots of things at once.
- Reenix : unix-like operating system kernel in rust. I like seeing this type of Rust project, it show the vitality of the language and its versatility. [github]
- Pulsar : Ebay Open-source real-time analytics platform and stream processing framework. Some nice characteristic for real time large scale CEP solution [github]
- Business Town : when real life cliche meets cartoonist .. long live captain moonshot !
Labels:
analytic
,
bigdata
,
cartoon
,
concurrency
,
ebay
,
links of the day
,
os
,
parallelism
,
rust
Tuesday, April 28, 2015
Links of the day 28 - 04 - 2015
Today's links 28/04/2015: #Rump Kernel Stack, Disque Distributed In Memory MQ, #FusionIO new PCIe product, Power level estimation of VM systems
- Ramp Stack : Nginx, MySQL, and PHP built on Rump Kernels without rearchitecting the application. Most of the work requires the app to be cross compile correctly (Nginx & MySQL). This implies that Unikernel-compatible unmodified POSIX C and C++ applications “just work” on top of Rump Kernels, provided that they can be cross- compiled.
- Disque :a distributed, in memory, message broker by Redis folk. Not production ready but a promising start.
- Pcie Flash : fusion IO is still kicking and deliver an interesting solution: up to 350,000 I/O operations per second (IOPS) on random reads and 385,000 IOPS on random writes (on the 3.2 TB model) with a 15k nanosecond write latency and 2.8 GB/sec of read bandwidth. .. However I still don't get why they don't want to use NVMe tech
- Process-level Power Estimation in VM-based Systems : the authors describe a fine-grained monitoring middleware providing real-time and accurate power estimation of software processes running at any level of virtualization in a system.
Labels:
distributed system
,
flash
,
fusionio
,
links of the day
,
message queue
,
pcie
,
power
,
rump kernel
,
virtualization
Monday, April 27, 2015
Links of the day 27 - 04 - 2015
Today's links 27/04/2015: #Postgresql on #FPGA and with #Kafka , #NVM impact on storage and Database
- Bottled Watter : Real time integration of Postgresql and Kafka.
- How Will Emerging Memory Technologies Impact Computing and Storage Architectures : looks at existing architectures and how they might adapt to the unique qualities of emerging memory technologies.
- Let’s Talk About Storage & Recovery Methods forNon-Volatile Memory Database Systems : look at how many "legacy" DBMS components are unnecessary and will degrade the performance of data intensive applications with the advance of NVM technology use.
- PostgreSQL on FPGA : interesting project trying to deliver basic infrastructure in order to help PostgreSQL developers have their "own" FPGA hacking projects.
Labels:
database
,
kafka
,
links of the day
,
nvm
,
storage
Thursday, April 23, 2015
Links of the day 24 - 04 - 2015
Today's links 24/04/2015: iOS fail, Memory Snooping Protection in HW, NVMe/PCIe fabric for storage, High performance server application framework
- Back to the Future With C++ and Seastar : Meetup notes on Seastar, open source server application framework written in C++ that presents a future/promise based API with 7 million requests per second served on a single machine. [slides]
- No iOS zone: PoC of attack : within WiFi hotspot range iOS devices are rendered unstable / unusable by constant reboots.
- Cloud security reaches silicon : Hardware implementation of method for thwarting memory snooping or inference across VM attacks by disguising memory-access patterns
- NVMe/PCIe as a Storage Interface : an very good overview of the market, future solution and form factors.
Labels:
apple
,
framework
,
High Performance
,
ios
,
links of the day
,
memory
,
nvme
,
pcie
,
security
Links of the day 23 - 04 - 2015
Today's links 23/04/2015: HPC MQ, Ethernet Connected Drives, MongoDB consistency issue, Memory Errors
- CoralMQ : full-fledged, ultra-low-latency, high-reliability, software-based middle ware for the development of distributed systems based on asynchronous messages. Developed for high speed trading and FX applications. Its always interesting to see that a lot of high frequency trading apps are written in Java while the (often flawed) perception is that it should be avoid for HPC solution.
- Ethernet Connected Drives : I really think that this type of storage is going to be the next generation of jack of all trade cloud storage solution. Massive potential scale out potential with better simplicity and maintainability ( not to mention lower TCO) [slides][Q&A]
- MongoDB stale reads : well not all good news if you really want to have consistent transaction.. MongoDB can return old / stall read value and stripe heavily relies on it is even more puzzling. This all scream run to me..
- Memory Errors in Modern Systems : with more memory in system ( and exascale HPC) more errors will occur and with the current state of things probably be silently ignored.. The risk is to what extend these error will damage the system and how to detect and mitigate them.
Labels:
error
,
ethernet
,
links of the day
,
memory
,
message queue
,
storage
Wednesday, April 22, 2015
Links of the day 22 - 04 - 2015
Today's links 22/04/2015: #bitcoin alternative, Nested OS , #Intel MXP, State of Probabilistic Programming
- Stellar : Stanford professor David Mazières thinks he has a faster, more flexible, and more secure alternative to bitcoin. The new cryptocurrency protocol, called SCP, is being adopted by a nonprofit called Stellar to replace a Bitcoin-inspired system meant to make financial services cheaper and more widely accessible. [white paper]
- Nested Kernel Operating System Architecture : old idea coming back ( ring protection) [paper][presentation]
- Intel MXP : Intel's MPX extension offers much of the old segment-level protection, but is opt-in for software and needs tooling support (compiler, linker, loader, etc). This is being worked on, e.g., Gcc started adding support for it.
- The State of Probabilistic Programming : good overview of probabilistic programming and where its at ( if you can see past the Deep learning craze)
Labels:
bitcoin
,
intel
,
links of the day
,
os
,
probabilistic
,
security
Tuesday, April 21, 2015
Links of the day 21 - 04 - 2015
Today's links 21/04/2015: #NVM architecture, #HPC storage fabric, Cluster sqlite, Just in time #Unikernel Quic protocol
- Architecture-level Designs using Emerging Non-volatile Memories : describe architecture techniques necessary for building efficient and reliable systems with NVMs.
- Jitsu : Just-In-Time Summoning of Unikernels using a Xen toolstack that satisfies the demands of secure multi-tenant isolation on resource-constrained embedded ARM devices.
- Fortissimo : A3CUBE Fortissimo Foundation, computing/storage converged solution and data engine, provides a very fast and scalable platform for storing and retrieving data items with 100s of gigabytes/s of throughput and Mega IOPS. But as always hte catch is that it requires a proprietary fabric to do so which limits its adoption [Architecture] [Benchmark]
- rqlite : Cluster of sqlite using Raft protocol, really cool implementation. I wonder if it could be used as a for of fast ACID compliant distributed cache for SQL DB [github]
- quic : Google pushing for their udp based protocol as IETF standard
Labels:
architecture
,
cluster
,
HPC
,
links of the day
,
network fabric
,
nvm
,
protocol
,
sqlite
,
unikernel
Monday, April 20, 2015
Links of the day 20 - 04 - 2015
Today's links 20/04/2015: NVMe over RDMA, Trademark , Pub-Sub for geo-replicated services , multicore middleware
- Mangstor : demonstration of "NVMe over RDMA" shows over 2-Million NVMe 4KB Random read and ~1.7M random write low latency IOPs, fully saturating dual 40Gb ehternet ports.
- Trademark : somebody trademarked "THE DATA CENTER IS THE COMPUTER "
- Wormhole : Reliable Pub-Sub to Support Geo-replicated Internet Services at Facebook
- MDTM : Multicore-Aware Data Transfer Middleware Project aims to accelerate data movement at multiore systems. It addresses inefficiencies in existing data movement tools when running on multicore systems by harnessing multicore parallelism to scale data movement on end systems.
Labels:
distributed
,
links of the day
,
message queue
,
multicore
,
numa
,
nvme
,
rdma
,
trademark
Friday, April 17, 2015
Links of the day 17 - 04 - 2015
Today's Links 17/04/2015: Cluster management, Causal consistency , Lock-free architecture and when server memory get big
- Borg : Large-scale cluster management at Google with Borg. It achieves high utilization by combining admission control, efficient task-packing, over-commitment, and machine sharing with process-level performance isolation. It supports high-availability applications with runtime features that minimize fault-recovery time, and scheduling policies that reduce the probability of correlated failures.
- Efficient Virtual Memory for Big Memory Servers : interesting paper, too bad the premise are bad. If you have that type of application you should size and configure the server to avoid swapping. Moreover Transparent Huge Page is the must in these scenario. All in all , nice academic exercise but does not represent real production scenario.
- Causal Consistency : nice post on the usage of causal consistency by the Jan Lindstrom of the mariaDB crowd.
- lock-free architecture : built on queues allowing tasks to control their parallelism level easily from 1(synchronized) to N(parallel).
Labels:
architecture
,
causal consistency
,
links of the day
,
lock free
,
memory
Wednesday, April 15, 2015
Links of the day 15 - 04 - 2015
Today's links 15/04/2015: distributed system, #containers security breach, #cloud and #NVM storage
- Parkeeper : a ZooKeeper service which seamlessly bridges all requests to etcd or consul. Nice if you want to use other service like Kafka etc..
- Another containers security breach : exploiting automatic crash analysis frameworks for fun and profit
- Yahoo Cloud Object Store : nice CEPH based cloud storage at exabyte scale.
- Mojim : A Reliable and Highly-Available Non-Volatile Memory System
Labels:
cloud
,
containers
,
links of the day
,
nvm
,
storage
Tuesday, April 14, 2015
Links of the day 14 - 04 - 2015
Today's links 14/04/2015: @Intel Storage Accel Lib.,@netlix monitoring system, why opensource startups #fail
- Storage Acceleration Library : open source version of Intel library collection of low-level functions such as contains fast erasure codes that implement a general Reed-Solomon type encoding for blocks of data and expanded set of functions used for data protection, hashing, encryption, etc. [fancy video explaining it here]
- Netflix Built Its Own Monitoring System : Roy Rapoport shares some of the lessons Netflix learned building a monitoring system, the challenges, pitfalls and opportunities encountered along the way.
- Why opensource startup fail : a intersting perspective on opensource based startup and why often the lack of clear public engagment with the project you relies one leads to a death spiral.
Labels:
library
,
links of the day
,
monitoring
,
open source
,
storage
Friday, April 10, 2015
Links of the day 10 - 04 - 2015
Today's links 10/04/2015: out of core #Bitcoin transaction, cluster mgmt @google, DC memory & storage
- Lightning Networks : Summary of the 4 part blog post by Rusty Russell. The concept is really interesting however it also require a "soft fork" of bitcoin. And that won't be easy.
- Cluster Management at Google : John Wilkes shares lessons learned managing clusters at the scale of Google.
- Datacenter memory and storage: very good overview of server primary system memory and secondary storage technologies spanning the past, present, and future.
Labels:
bitcoin
,
cluster
,
links of the day
,
memory
,
storage
Monday, April 06, 2015
Links of the day 06 - 04 - 2015
Today's links 06/04/2015: HPC cache, architecture pattern, virtual RDMA, Queue
- cachelot : High-performance cache library and distributed caching server. Memcached compatible.
- Software ArchitecturePatterns : Understanding Common Architecture Patterns and When to Use them by Mark Richards
- Virtual RDMA : presentation on virtual RDMA device using SR-IOV for virtual environment.
- libtorrent alert queue :new architecture of libtorrent alert queue using an heterogeneous queue
Labels:
architecture
,
cache
,
links of the day
,
rdma
Friday, April 03, 2015
Links of the day 03 - 04 - 2015
Today's links 03/04/2015: NVM trends and loads of conferences - Hepix, CernVm, Lambda Days
- Emerging Non Volatile Memory (NVM) Technology & Market Trends: 2014 Report by Yole Developpement
- HEPiX workshop : lots of very good talk in there.
- CernVM workshop : ecosystem providing all the building blocks for a virtualized analysis environment for High Energy Physics experiments.
- Lambda 2015 : Functional Programming Conference (26-27 FEBRUARY 2015)
Labels:
cern
,
cloud
,
conference
,
HPC
,
lambda
,
links of the day
Thursday, April 02, 2015
Links of the day 02 - 04 - 2015
Today's links 02/04/2015: Billing & payment, #Bitcoin Lightning transactions, Compression benchmark, #unikernel
- Billing & Payments Engineering Meetup II : lots of very interesting talk about the difficulty of processing payements at scale while maintaining compliance and security [first meetup]
- Lightning Networks Part I - Revocable Transactions : very good break down by Rusty Russel of the Lightning Network paper. The promise of this work is exceptional: instant reliable transactions across the bitcoin network.
- Squash Benchmark : compression benchmark with 28 datasets, each of which is tested against 23 plugins containing 39 codecs at every compression level they offer—the number varies by codec, but there are 178 in total, yielding 4,984 different settings. The benchmark is currently run on 8 different machines (with more coming soon!) for a current grand total of 39,872 configurations.
- A VM for every URL: interesting experiment where the author tries to run a single unikernel per URL, not really anything significant can be derived from the experiment but it shows the direction the industry is going for.
Labels:
benchmark
,
billing
,
bitcoin
,
compression
,
links of the day
,
payment
,
unikernel
Wednesday, April 01, 2015
Links of the day 01 - 04 - 2015
Today's links 01/04/2015: Messaging, WS #Tor Routing, #Bitcoin time stamping, HPC programming model
- Exactly once messaging : more rant on the subject, basically guarantee anything , especially message behavior is hard. Its even harder without formal proof and most system out there do not have any.
- Hammer Protocol : an implementation of Tor Routing for Web Services
- Decentralized Trusted Timestamping using the Crypto Currency Bitcoin : interesting, however it beg the question, how do you garantee that the timestamp generated in the first place is accurate. Unless the time at each peer signing it is accurate we can have huge variation.
- High-Performance and Scalable Designs of Programming Models for Exascale Systems :focus on programming models and their designs for upcoming exascale systems with millions of processors and accelerators such as current status and future trends of MPI and PGAS (UPC and OpenSHMEM) programming models.
Labels:
bitcoin
,
HPC
,
links of the day
,
Messaging
,
tor
Subscribe to:
Posts
(
Atom
)