Docker, Microservices, Miscellaneous

A step towards the future of configuration and infrastructure management with Nix

Recently, I’ve been experimenting with the Nix ecosystem. It offers an interesting and more integrated alternative than traditional configuration and infrastructure management systems such as Puppet, (Docker) containers and Terraform. I have looked into Nix, nixpkgs, NixOS and NixOps. Nix is a purely functional package management language, which aims to make the process of packaging reproducible. Nixpkgs is a set of curated packages that can be installed, described in Nix. You can use Nixpkgs on top of any modern Linux distribution. On OS X, it can be used as an alternative to homebrew. The Nix documentation contains instructions to install it on any Linux distribution and OS X. NixOS is a linux distribution based on nixpkgs, and NixOps is a tool to deploy NixOS on other (virtual) machines.

We’ll start by comparing Nix with some alternatives, after which I’ll dive into a example of how to set up a local marathon cluster. We wrap this post up with a conclusion.

A step towards the future of configuration and infrastructure management with Nix

Nix vs Converging Configuration Management Systems vs Docker

Converging Configuration Management Systems (CCMS), like Puppet, Chef and Ansible, work by modifying the system to match the desired state, by e.g. adding or removing packages or files. A typical problem that I have encountered in the past, is that removing packages or files does not happen automatically when you remove the install-this-package instruction from the CCMS' code. You explicitly have to tell the CCMS to remove the package, converge the system, and then delete the remove command from the CCMS’ code. Other issues include that identical machines that run the CCMS at two different points in time could very well end up in a different state, if you don’t take care to prevent this from happening by pinning e.g. version numbers. Note that it is possible to prevent it, it just takes some additional discipline and effort.

Build model of CCMS' - A step towards the future of configuration and infrastructure management with Nix

Start with a base image, run the CCMS to apply the changes on each VM. If no great care is taken to prevent divergence, the machines could diverge over time, and may diverge into different states.

This not only makes provisioning potentially divergent, rolling back to a previous version is not trivial; you need to undo your changes by writing code more code.

CCMS Deployment Model - A step towards the future of configuration and infrastructure management with Nix

The deployment of CCMS' consist of running the CCMS against the machines, making changes to them. Once a change is made, rolling back to a previous version is not trivial.

There are various workarounds, such as using “golden images” that you build once using a CCMs. When you'd want to update your machines, you'd start again from this golden image, and run the CCMS again.

Docker solves these problems by using a “golden image” as it’s default modus operandi, and by having file system layers. Consider the following Dockerfile:

FROM ubuntu
# ...
RUN apt-get install -yq gcc
RUN apt-get install -yq vim mc
RUN apt-get install -yq big package
RUN some-expensive-operation
# ...

Each line in this Dockerfile results in a new file system layer. If you no longer want to have the package mc installed, you can simply modify the line in which you install this package, rebuild the image and it is gone! The disadvantage is that you changed a file system layer, all depending layers must be re-build, although you as a human know that the change should have no effect at all on the depending layers.

The new docker file looks like this:

FROM ubuntu
# ...
RUN apt-get install -yq gcc
RUN apt-get install -yq vim
RUN apt-get install -yq big package
RUN some-expensive-operation

Docker Build Model - A step towards the future of configuration and infrastructure management with Nix

Docker offers a branching-but-no-merges model. If you modify a Dockerfile halfway, all dependent layers must be rebuilt.

Deployment consists of pulling the correct container image from a registry, and starting the container. Rolling back to a previous version is just a question of pulling and running the correctly tagged version of your container image.

Docker Deployment Model - A step towards the future of configuration and infrastructure management with Nix

Docker provides identical runtime environments, by separating building a service and deploying it. First, a container is build from a Dockerfile, afer which diffferent machines can pull it from a registry. In the end, both are running exactly the same version of software.

NixOS does things differently. You can configure the state of the system by writing a “nix expression”; a description of how the system should look like.

As I mentioned in the introduction, Nix only has pure functions. This implies that a function always returns the same result for the same parameters. The function f(x) = x+1 is pure, but g(x)=x+time_of_day() is not.

Packages in nixpkgs, and your system configuration are implemented as pure functions. Therefore, building your system twice from the same input results in exactly the same state.

Nix Build Model - A step towards the future of configuration and infrastructure management with Nix

Expressions in Nix are pure functions. Hence, building the same expression on two distinct machines results in an identical state.

This has some interesting consequences! Given the two definitions in the figure below, if you build Expression 1, and then Expression 2 that depend on one shared packge P2, it can simply reuse package P2; rebuilding it would result in exactly the same package.

Nix Deployment Model - A step towards the future of configuration and infrastructure management with Nix

Deployment in nix consist of writing an expression that results in a dependency graph. Any dependency that is already present, can and will be shared.

There two P3's. They represent different versions of the same package. The P3 referenced by Expression 1 is an older version than the one referenced by Expression 2.

Nix offers atomic deployment of a new version of the systems configuration, by using the fact that updating symlinks is an atomic operation on UNIX. There is a global symlink pointing to some expression that models the systems configuration. We’ll get into the details of this in this blog post.

The point of all this, is that nix makes rollbacks trivial; it’s just a matter of updating one symlink. In our example, this implies that rolling back to Expression 1 would make P1 unavaible (but not remove it!) and downgrade P3.

If after some time, Expression 1 is not deemed to be needed anymore, it’s Garbage Collection root can be removed. By running nix-collect-garbage after removing this GC root, all unnecessary dependencies, in our example, P1, and the old P3, will be collected and removed.

NixOps vs Terraform

NixOps builds on top of NixOS, and offers a completely integrated environment to define (virtual) machines and the software that should be installed on them.

Terraform's documentation states:

"Terraform is not a configuration management tool, and it allows existing tooling to focus on their strengths: bootstrapping and initializing resources." (source)

NixOps offers this, but more. It manages changes in both infrastructure and installed software. It’s a combination of a configuration management tool and an infrastructure management tool.

We show an example of how NixOps works in the example below.

Extra components needed

When you manage a system with a CCMS, your infrastructure generally consists of VM’s with some Linux distribution, and your CCM’s repository that uses external libraries to manage the software on the VM’s.

If you’re using Docker, you are depending less on the features provided by the OS distribution. As shown by for example CoreOS, only a small core operating system with some basic utilities is needed. Rancher takes this to an even more extreme level, where the only purpose of the linux distribution is to run Docker containers that manage the system itself. However, a tool to manage the Containers themselves is still needed.

By using Nix, nixpkgs and NixOps, we stay can use the same abstraction mechanism to work on both the infrastructure, the installed software and its configuration.

Comparing CCMS’, Docker and Nix

The table below contains a rough comparison of CCMS’s, Docker and Nix.

  CCMS Docker Nix
No divergence between machines With effort Safe Safe
Repreducable builds With effort With effort Yes
Rollback With effort Yes Yes
Components OS + CCMS OS + Docker + Container Manager Nix
Incremental builds “Golden image” Linear history Dependency graph
Difficulty in usage Harder Easy Harder

Example: Setting up a Mesos and Marathon cluster

If you want to run the examples yourself (both on Linux and OS X), you should install Nix first from their download page. Note that it only touches files in /nix, so uninstalling nix is just a question of removing all those files.

To provision multiple machines with Nix, we can use NixOps. Once you have installed Nix, NixOps can be installed by running nix-env -i nixops. Make sure that you have installed VirtualBox, and created a hostonly network with the name vboxnet0, and a default NAT network. Some instructions on how to do this on OSX can be found here.

Note: In the next blogpost, we dive deeper in the Nix-language, and we’ll dissect the code below. For now, just take a quick glance at the code below

  # The colon states that this is a function, with an argument named id.
  makeMaster = id: {
    name = "mesos-master-${toString id}";
    value = { config, pkgs, lib, nodes, ... }: {
      deployment.targetEnv = "virtualbox";
      deployment.virtualbox.memorySize = 512;
      deployment.virtualbox.headless = true;
      networking.firewall.enable = false;
      services.zookeeper = {
        enable = true;
        id = id;
        servers = let 
            zk_servers = map zk_server_line masterServers;
            zk_server_line = zk: (let
                zkId = toString nodes.${};
              in "server.${zkId}=${}:2888:3888");
            builtins.concatStringsSep "\n"  zk_servers;
      services.mesos.master = {
        enable = true;
        zk = mesosZkString;
        quorum = (nrMasterServers / 2 ) + 1; # Ensure a quorum of at least half + 1.
      services.marathon = {
        enable = true;
        zookeeperHosts = (s: + ":2181") masterServers;
        user = "root";
  makeSlave = id: {
    name = "mesos-slave-${toString id}";
    # So 'value' is a function that takes an attribute set (Dictionary in python speak) as an input, and returns a new attribute set.
    value = { config, pkgs, lib, ... }: {
      deployment.targetEnv = "virtualbox";
      deployment.virtualbox.memorySize = 1024 * 2;
      deployment.virtualbox.headless = true;
      networking.firewall.enable = false;
      virtualisation.docker.enable = true;
      services.mesos.slave = {
        enable = true;
        withDocker = true;
        master = mesosZkString;
  nrMasterServers = 3;
  nrSlaveServers = 2;
  # This generates a list [ { name = "mesos-master-0", value =  } { name = "mesos-master-1”; value =  } ]
  masterServers = builtins.genList makeMaster nrMasterServers;
  slaveServers = builtins.genList makeSlave nrSlaveServers;
  mesosZkString = let 
                   zkServers = builtins.concatStringsSep "," ( (s: + ":2181") masterServers);
  { network.description = "Marathon test"; }
    // builtins.listToAttrs masterServers
    // builtins.listToAttrs slaveServers
# The listToAttrs function builds an attribute set from a list of attribute sets with some { name = "attr_name"; value = "attr_value"; }. It returns a set { attr_name = "attr_value" }

Each master server uses 512MB of ram, and each slave uses 2GB’s. The current setup has three masters and two slaves, so it uses 5.5 GB. The minimum setup is to have just one master, and one slave, which would take 2.5GB of ram, you can set the nrMasterServers and nrSlaveServers to 1 in the marathon.nix file, if you do not have 5.5GB available.

You’ve got your marathon.nix ready? Let’s create a deployment named marathon, by running:

nixops create -d marathon marathon.nix
(out) created deployment ‘b8393c45-2673-11e6-9e82-208984d3285c’
(out) B8393c45-2673-11e6-9e82-208984d3285c’

This generates a unique ID to identify the deployment / stack you’ve created.

To actually create the VirtualBox VM’s and provision them, run:

nixops deploy -d marathon

NOTE: From time to time, this fails with the error "unable to active new configuration", caused by some virtualbox guest driver that has not been started properly. If this happens, you can just abort (CTRL+C) nixops, and try to deploy again. It always worked the second time I tried. I’ve only seen these error messages when working with the virtualbox backend. GCE posed no problems.

If you’d destroy this deployment (by running nixops destroy -d marathon) and redeploy it, you’ll see that only a very small part of the system configuration must be rebuild; the part that contains dynamic settings like IP addresses, which would be the service definitions that depend on the IP addresses and the overall OS configuration that uses these service definitions.

Let’s run some demo application on top of marathon!

You can inspect the state of the deployment by running nixops info:

nixops info -d marathon
(out)Network name: marathon
(out)Network UUID: b8393c45-2673-11e6-9e82-208984d3285c
(out)Network description: Marathon test
(out)Nix expressions: /home/maarten/cs/blog/nix/marathon.nix
(out)| Name           |      Status     | Type       | Resource Id                                                | IP address     |
(out)| mesos-master-0 | Up / Up-to-date | virtualbox | nixops-b8393c45-2673-11e6-9e82-208984d3285c-mesos-master-0 | t |
(out)| mesos-master-1 | Up / Up-to-date | virtualbox | nixops-b8393c45-2673-11e6-9e82-208984d3285c-mesos-master-1 | |
(out)| mesos-master-2 | Up / Up-to-date | virtualbox | nixops-b8393c45-2673-11e6-9e82-208984d3285c-mesos-master-2 | |
(out)| mesos-slave-0  | Up / Up-to-date | virtualbox | nixops-b8393c45-2673-11e6-9e82-208984d3285c-mesos-slave-0  | |
(out)| mesos-slave-1  | Up / Up-to-date | virtualbox | nixops-b8393c45-2673-11e6-9e82-208984d3285c-mesos-slave-1  | |

Tip: install jq using nix-env -i jq, and abuse the fact that nixops can output JSON.

export MARATHON_ENDPOINT=http://`nixops export -d marathon | jq -r '.[].resources["mesos-master-0"].privateIpv4'`:8080

Create a marathon application file named nginx.json with the following content:

  "id": "nginx",
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "mesosphere/simple-docker",
      "network": "BRIDGE",
      "portMappings": [
         { "hostPort": 0, "containerPort": 80, "protocol": "tcp"}
  "instances": 1,
  "cpus": 0.1,
  "mem": 64

And submit it to marathon:

curl -X POST $MARATHON_ENDPOINT/v2/apps -d @nginx.json -H "Content-type: application/json"

Cool, let’s see if we can find the IP and port of the service we just created.

curl -s $MARATHON_ENDPOINT/v2/apps/nginx | jq '.app.tasks'
(out)  {
(out)    "id": "nginx.1a2ac376-2717-11e6-aee5-0800271e2037",
(out)    "host": "mesos-slave-1",
(out)    "ipAddresses": [],
(out)    "ports": [
(out)      31072
(out)    ],
(out)    "startedAt": "2016-05-31T10:04:46.994Z",
(out)    "stagedAt": "2016-05-31T10:04:45.812Z",
(out)    "version": "2016-05-31T10:04:45.691Z",
(out)    "slaveId": "10b80303-74f0-4b81-8619-7b969724ee89-S1",
(out)    "appId": "/nginx"
(out)  }

Unfortunately, the hostname does not resolve from your host machine. Use nixops info to manually look this up, or run the following commands to let your computer do the work:

export NGINX_HOST=`curl -s $MARATHON_ENDPOINT/v2/apps/nginx | jq -r '.app.tasks[].host'`
export NGINX_PORT=`curl -s $MARATHON_ENDPOINT/v2/apps/nginx | jq -r '.app.tasks[].ports[]'` 
export NGINX_IP=`nixops export | jq -r ".[].resources[\"$NGINX_HOST\"].privateIpv4"`

Cool, now we know on which slave and on which port the nginx instance is running. Let's see if we can read from it.

(out)    <html>
(out)        <body>
(out)        <h1> Hello brave new world! </h1>
(out)        </body>
(out)    </html>

Yay, success!

Wrapping up

As you saw, with just 65 lines of Nix expression, we can describe arbitrary large marathon clusters. This is a testament to the power of abstraction offered by the nix tools. Using nix has some disadvantages though; nixos is a relative young distribution that is not backed by a tech giant yet. However, the same can be said for distributions like Gentoo or Debian. There are several companies using NixOS in production settings. Someone on Reddit suggests that Nix is used by Intel to verify their SSD’s. Over time, NixOS has the potential to grow in one of the big distributions and I guess it only takes some time before companies start to offer Nix support.

At the moment, I’m suggesting to take a look at Nix, and it’s model. I believe it’s a next step in the evolution of managing computer systems.

Looking forward

In the next blogpost, we’ll take a look at how nix-the-language works and we will dissect the marathon.nix file from this post, and show how we can modify this expression to create identical clusters on GCE and EC2.

Leave your Comment