The Operating System is the Target

I recently had to solve a small problem that involved collecting some information from around 1000 machines, do some computation on the information and then do some work on each machine to fix something. It wasn't anything fancy, just more machines than I could do by hand. I could have used a language like Python or Ruby to drive the whole thing, but I decided I should just use the tools that come with the OS (in this case, FreeBSD). I used ssh to connect to each machine, but doing that in serial would take too long, so I used xargs to do it in parallel. I needed to store some information about each host, so I used the file system as a key-value database where the name of the file was the machine and the contents were the information. With sort and uniq I could do a range of interesting set operations. And it all worked just great. Using the file system as my database, rather than an in-memory data structure let me do a lot of debugging easily as each phase was on disk and could be inspected. Going in parallel was just a matter of how many ssh processes I was willing to run at once. I could have solved it in Python, figuring out a library for doing SSH, hoping it's async or using a bunch of threads and using in-memory data structures to store the data. But parallelizing the whole thing was a matter of adding -P10 to the xargs command, almost too easy. I had some edge cases in what the data looked like, so having the data from each phase on disk made debugging and doing the next phase easy. By embracing the OS, I actually got a smoother experience than I likely would have otherwise.

That's a simple example, though, and what I was doing was a one-off. A long-term solution would probably not be implemented that way. Or would it? What is the limit in this solution? A reasonable file system can handle at least an order of magnitude more files in a directory and there are some well known tricks to making directories small by sharding the data across more directories. Using ssh with xargs probably starts to feel some serious pain coming up on 10k machines. sort and uniq can work pretty well across gigabytes of data (which would be a lot of hosts). It seems like the limiting factor here is ssh, one process per connection might just be a bit too much. Ok, so we can replace the ssh and xargs with some kind of real program that handles that but keep the rest. Testing is not bad in this setup either, unit tests could just be creating a directory structure and running the phase of the system on that structure and see what happens. And we can mock things just by playing with the PATH and making a mock, for example, ssh.

Maybe just the idea of using these shell scripts and connecting them via pipes or the file system is off putting, though. It certainly doesn't feel professional. Modern programmers don't use their OS. They abstract it away so their code can run anywhere. Java has probably done the most to push this, having it as an explicitly stated goal. The Java developer does not write code for the OS they are on, they write it for the JVM. The JVM even goes as far as to provide its own DNS cache, something you'd leave to the surrounding OS infrastructure, which has some odd behaviour and worse default settings.

Using the OS as the programming API isn't new, by a long shot. The first experience I had with this was through qmail which lets the OS enforce security for it. Rather than being implemented as a single monolithic application which has access to everything and attempts to enforce permissions itself, qmail breaks itself into many different processes, each of which runs as a different user and communicate over IPC. The OS then becomes the security model.

Joyent's Manta takes a similar approach of utilizing the OS. Manta is a map-reduce framework that uses ZFS and Solaris Zones. Unlike Hadoop, Manta lets one distribute map-reduce jobs over thousands of machines where the job is any program, rather than having to be implemented in a specific framework. The demo app is implementing word count using wc and awk. By using the primitives the OS provides, the existing ecosystem of programs are already usable. Running multi-tenant involves setting up the file system permissions and creating the zones correctly, two things which may not be perfect but are well understood and possible to reason about.

The model that qmail uses is successful from a security standpoint, but qmail itself isn't all that popular these days. Postfix and OpenSMTPD are easier to administrate. It's possible that qmail was a bit too early and would be more popular if configuration managers like Ansible or Puppet existed, reducing the burden on setting it up. But one limitation of qmail is that most OS's are not designed to scale to thousands or millions of users, which qmail uses for its own user management. There exists a parallel universe, though, where a kernel developer decided to make users a more fluid and dynamic concept such that applications could outsource user management to the OS. In this universe, logging into your GMail account spawns your own service process running under your own OS user, maybe even in a jail.

Abstracting the OS away isn't inherently bad. Writing a program to run across multiple OS's will need to unify the semantics of each system somehow. But making more use of the OS where appropriate can simplify a program while making it more robust and debuggable. OS's already provide a wide range of tools to inspect what they are doing and by using the OS you get that introspection for free. Maybe next time you see yourself writing a service or tool, take a moment and see where the OS might be able to help you solve it.

UPDATE: I was reminded of Ted Dziuba's post on Taco Bell Programming which says a very similar thing to this post as well. Taco Bell Programming is about having a toolbox of as few tools as possible that can be connected in different ways to achieve a complex range of functionality. This post is about embracing the operating system as that toolbox. Ted says it best here: "functionality is an asset, but code is a liability."