NixOS Planet

August 02, 2018

Sander van der Burg

Automating Mendix application deployments with Nix

As explained in a previous blog post, Mendix is a low-code development platform -- the general idea behind low-code application development is that instead of writing (textual) code, you model an application, such as the data structures and the corresponding views. One of the benefits of Mendix is that it makes you more productive as a developer, for certain classes of applications.

Although low-code development is conceptually different from a development perspective compared to more "traditional" development approaches (that require you to write code), there is one particular aspect a Mendix application lifecycle has in common. Eventually, you will have to deploy your app to an environment that makes your application available to end users.

For users of the Mendix cloud portal, deploying an application is quite convenient: with just a few simple mouse clicks your application gets deployed to a test, acceptance or production environment.

However, managing on-premise application deployments or actually managing applications in the cloud is all but a simple job. There all all kinds of challenges you need to cope with, such as:

  • Making sure that all dependencies of an app are present, such as a database for storage.
  • Executing all relevant deployment activities to make an app available for use.
  • Upgrading is risky and difficult -- it may break the application and introduce downtime.

There are a variety of deployment solutions available to manage deployment processes. However, no solution is perfect -- every tool has its strengths and weaknesses and no tool is a perfect fit. As a result, we still have to develop custom solutions that automate missing parts in a deployment process and we have many kinds of additional complexities that we need to cope with.

Recently, I investigated whether it would be possible to deploy Mendix applications, with my favourite class of deployment utilities from the Nix project, and I gave an introduction to the Nix project to the R&D department at Mendix.

Using tools from the Nix project


For readers not familiar with Nix: the tools in the Nix project solve many configuration management problems in their own unique way. The basis of all the tools is the Nix package manager that borrows concepts from purely functional programming languages, such as Haskell.

To summarize Nix in just a few sentences: deploying a package with Nix is the same thing as invoking a pure function that constructs a package from source code and its build-time dependencies (that are provided as function parameters). To accomplish purity, Nix composes so-called "pure build environments", in which various restrictions are imposed on the build script to ensure that the outcome will be (almost) identical if a package is built with the same build inputs.

The purely functional deployment model has all kinds of benefits -- for example, it provides very strong dependency completeness and reproducibility guarantees, and all kinds of optimizations (e.g. a package that has been deployed before does not have to be built again, packages that have no dependency on each other can be built in parallel, builds can be downloaded from a remote location or delegated to another machine).

Another important property that all tools in the Nix project have in common is declarative deployment -- instead of describing the deployment activities that need to be carried out, you describe the structure of your system that want to deploy, e.g. the packages, a system configuration, or a network of machines/services. The deployment tools infer the activities that need to be carried out to get the system deployed.

Automating Mendix application deployments with Nix


As an experiment, I investigated how Mendix application deployments could fit in Nix's vision of declarative deployment -- the objective is to take a Mendix project created by the modeler (essentially the "source code" form of an application), write a declarative deployment specification for it, and use the tools from the Nix project to get a machine running with all required components to make the app run.

To bring a Mendix application in a running state, we require the following ingredients:

  • We must obtain the Mendix runtime that interprets the Mendix models. Packaging the Mendix runtime in Nix is fairly straight forward -- simply unzipping the distribution, and moving the package contents into the Nix store, and adding a wrapper script launches the runtime suffices.
  • We must produce a Mendix Deployment Archive (MDA file) that creates a Zip container with all artifacts required to run a Mendix app by the runtime. An MDA file can be produced from a Mendix project by invoking the MxBuild tool. Since MxBuild is required for this, I had to package it as well. Packaging mxbuild is a bit trickier, as it requires mono and Node.js.

Building an MDA file with Nix


The most interesting part is writing a new function abstraction for building MDA files with Nix -- in a Nix builder environment, (almost) any build tool can be used albeit with restrictions that are imposed on them to make builds more pure.

We can also create a function abstraction that invokes mxbuild in a Nix builder environment:


{stdenv, mxbuild, jdk, nodejs}:
{name, mendixVersion, looseVersionCheck ? false, buildInputs ? [], ...}@args:

let
mxbuildPkg = mxbuild."${mendixVersion}";
extraArgs = removeAttrs args [ "buildInputs" ];
in
stdenv.mkDerivation ({
buildInputs = [ mxbuildPkg nodejs ] ++ buildInputs;
installPhase = ''
mkdir -p $out
mxbuild --target=package \
--output=$out/${name}.mda \
--java-home ${jdk} \
--java-exe-path ${jdk}/bin/java \
${stdenv.lib.optionalString looseVersionCheck "--loose-version-check"} \
"$(echo *.mpr)"
mkdir -p $out/nix-support
echo "file binary-dist \"$(echo $out/*.mda)\"" > $out/nix-support/hydra-build-products
'';
} // extraArgs)

The above expression is a function that composes another function that takes common Mendix parameters -- the application name, the version of MxBuild that we want, and whether we want to use a strict or loose version check (it is possible to compile a project developed for a different version of Mendix, if desired).

In the body, we create an output directory in the Nix store, we invoke mxbuild to compile to MDA app and put it in the Nix store, and we generate a configuration file that makes it possible to expose the MDA file as a build product, when Hydra: the Nix-based continuous integration service is being used.

With the build function shown in the code fragment above, we can write a Nix expression for a Mendix project:


{ pkgs ? import { inherit system; }
, system ? builtins.currentSystem
}:

let
mendixPkgs = import ./nixpkgs-mendix/top-level/all-packages.nix {
inherit pkgs system;
};
in
mendixPkgs.packageMendixApp {
name = "conferenceschedule";
src = /home/sander/SharedWindowsFolder/ConferenceSchedule-main;
mendixVersion = "7.13.1";
}

The above expression (conferenceschedule.nix) can be used to build an MDA file for a project named: conferenceschedule, residing in the /home/sander/SharedWindowsFolder/ConferenceSchedule-main directory using Mendix version 7.13.1.

By running the following command-line instruction, we can use Nix to build our MDA:


$ nix-build conferenceschedule.nix
/nix/store/nbaa7fnzi0xw9nkf27mixyr9awnbj16i-conferenceschedule
$ ls /nix/store/nbaa7fnzi0xw9nkf27mixyr9awnbj16i-conferenceschedule
conferenceschedule.mda nix-support

In addition to building an MDA, Nix will also download the dependencies: the Mendix runtime and MxBuild, if they have not been installed yet.

Running a Mendix application


Producing an MDA file is an important ingredient in the deployment lifecycle of a Mendix application, but it is not entirely what we want -- what we really want is a running system. To get a running system, additional steps are required beyond producing an MDA:

  • We must unzip the MDA file into a directory with write permissions.
  • We must create writable state sub directories, e.g. data/tmp, data/files.
  • After starting the runtime, we must configure the admin interface, to send instructions to the runtime to initialize the database and start the app:

    $ export M2EE_ADMIN_PORT=9000
    $ export M2EE_ADMIN_PASS=secret
  • Finally, we must communicate over the admin interface to configure, initialize the database and start the app:

    curlCmd="curl -X POST http://localhost:$M2EE_ADMIN_PORT \
    -H 'Content-Type: application/json' \
    -H 'X-M2EE-Authentication: $(echo -n "$M2EE_ADMIN_PASS" | base64)' \
    -H 'Connection: close'"
    $curlCmd -d '{ "action": "update_appcontainer_configuration", "params": { "runtime_port": 8080 } }'
    $curlCmd -d '{ "action": "update_configuration", "params": { "DatabaseType": "HSQLDB", "DatabaseName": "myappdb", "DTAPMode": "D" } }'
    $curlCmd -d '{ "action": "execute_ddl_commands" }'
    $curlCmd -d '{ "action": "start" }'

These deployment steps cannot be executed by Nix, because Nix's purpose is to manage packages, but not the state of a running process. To automate these remaining parts, we generate scripts that execute the above listed steps.

NixOS integration


NixOS is a Linux distribution that extends Nix's deployment facilities to complete systems. Aside from using the Nix package manage to deploy all packages including the Linux kernel, NixOS' main objective is to deploy an entire system from a single declarative specification capturing the structure of an entire system.

NixOS uses systemd for managing system services. The systemd configuration files are generated by the Nix package manager. We can integrate our Mendix activation scripts with a generated systemd job to fully automate the deployment of a Mendix application.


{pkgs, ...}:

{
...

systemd.services.mendixappcontainer =
let
runScripts = ...
appContainerConfigJSON = ...
configJSON = ...
in {
enable = true;
description = "My Mendix App";
wantedBy = [ "multi-user.target" ];
environment = {
M2EE_ADMIN_PASS = "secret";
M2EE_ADMIN_PORT = "9000";
MENDIX_STATE_DIR = "/home/mendix";
};
serviceConfig = {
ExecStartPre = "${runScripts}/bin/undeploy-app";
ExecStart = "${runScripts}/bin/start-appcontainer";
ExecStartPost = "${runScripts}/bin/configure-appcontainer ${appContainerConfigJSON} ${configJSON}";
};
};

The partial NixOS configuration shown above defines a systemd job that runs three scripts (as shown in the last three lines):

  • The undeploy-app script removes all non-state artefacts from the working directory.
  • The start-appcontainer script starts the Mendix runtime.
  • The configure-appcontainer script configures the runtime, such as the embedded Jetty server and the database, and starts the application.

Writing a systemd job (as shown above) is a bit cumbersome. To make it more convenient to use, I captured all Mendix runtime functionality in a NixOS module, with an interface exposing all relevant configuration properties.

By importing the Mendix NixOS module into a NixOS configuration, we can conveniently define a machine configuration that runs our Mendix application:


{pkgs, ...}:

{
require = [ ../nixpkgs-mendix/nixos/modules/mendixappcontainer.nix ];

services = {
openssh.enable = true;

mendixAppContainer = {
enable = true;
adminPassword = "secret";
databaseType = "HSQLDB";
databaseName = "myappdb";
DTAPMode = "D";
app = import ../../conferenceschedule.nix {
inherit pkgs;
inherit (pkgs.stdenv) system;
};
};
};

networking.firewall.allowedTCPPorts = [ 8080 ];
}

In the above configuration, the mendixAppContainer captures all the properties of the Mendix application that we want to run:

  • The password for communicating over the admin interface.
  • The type of database we want to use (in this particular case an in memory HSQLDB instance) and the name of the database.
  • Whether we want to use the application in development (D), test (T), acceptance (A) or production (P) mode.
  • A reference to the MDA that we want to deploy (deployed by a Nix expression that invokes the Mendix build function abstraction shown earlier).

By writing a NixOS configuration file, storing it in /etc/nixos/configuration.nix and running the following command-line instruction:


$ nixos-rebuild switch

A complete system gets deployed with the Nix package manager that runs our Mendix application.

For production use, HSQLDB and directly exposing the embedded Jetty HTTP is not recommended -- instead a more sophisticated database, such as PostgreSQL should be used. For serving HTTP requests, it is recommended to use nginx as a reverse proxy and use it to serve static data and provide caching.

It is also possible to extend the above configuration with a PostgreSQL and nginx system service. The NixOS module system can be used to retrieve the properties from the Mendix app container to make the configuration process more convenient.

Conclusion


In this blog post, I have investigated how Mendix applications can be deployed by using tools from the Nix project. This resulted in the following deployment functionality:

  • A Nix function that can be used to compile an MDA file from a Mendix project.
  • Generated scripts that configure and launch the runtime and the application.
  • A NixOS module that can be used to deploy a running Mendix app as part of a NixOS machine configuration.

Future work


Currently, only single machine deployments are possible. It may also be desirable to connect a Mendix application to a database that is stored on a remote machine. Furthermore, we may also want to deploy multiple Mendix applications to multiple machines in a network. With Disnix, it is possible to automate such scenarios.

Availability


The Nix function abstractions and NixOS module can be obtained from the Mendix GitHub page and used under the terms and conditions of the Apache Software License version 2.0.

Acknowledgements


The work described in this blog post is the result of the so-called "crafting days", in which Mendix supports its employees to experiment completely freely two full days a month.

Furthermore, I have given a presentation about the functionality described in this blog post and an introduction to the Nix project:


and I have also written an introduction-oriented article about it on the Mendix blog.

by Sander van der Burg (noreply@blogger.com) at August 02, 2018 09:51 PM

Graham Christensen

an EPYC NixOS build farm

EPYC vs m1.xlarge.x86

Nix is a powerful package manager for Linux and other Unix systems that makes package management reliable and reproducible. It provides atomic upgrades and rollbacks, side-by-side installation of multiple versions of a package, multi-user package management and easy setup of build environments.

The Nix community has collected and curated build instructions (expressions) for many thousands of packages in the Nix package collection, Nixpkgs. The collection is a large GitHub repository, which receives over a thousand pull requests each month. Some of these changes can sometimes cause all of the packages to rebuild.

To test changes to Nixpkgs and release updates for Nix and NixOS, we necessarily created our own build infrastructure. This allows us to give better quality guarantees to our users.

The NixOS infrastructure team runs several types of servers: VMs on AWS, bare metal, macOS systems, among others. We build thousands of packages a day, sometimes reaching many tens of thousands per day.

Some of our builds depend on unique features like KVM which are only available by using bare metal servers, and all of them benefit from numerous, powerful cores.

For over a year now, Packet.net (where you can natively deploy NixOS by the way!) has been generously providing bare metal hardware build resources for the NixOS build farm, and together we were curious how the new EPYC from AMD would compare to the hardware we were already using.

For this benchmark we are comparing Packet’s m1.xlarge.x86 against Packet’s first EPYC machine, c2.medium.x86. Hydra already runs a m1.xlarge.x86 build machine, so the comparison will be helpful in deciding if we should replace it with EPYC hardware.

AMD EPYC has the chance to reduce our hardware footprint, reduce our need for our AWS scale-out, and improve our turnaround time for time-sensitive security patches.

System Comparison:

  m1.xlarge.x86 c2.medium.x86 (EPYC)
NixOS Version 18.03.132610.49a6964a425 18.03.132610.49a6964a425
Cost per Hour $1.70/hr $1.00/hr
CPU 24 Physical Cores @ 2.2 GHz 24 Physical Cores @ 2.2 GHz
  2 x Xeon® E5-2650 v4 1 x AMD EPYC™ 7401P
RAM 256 GB of ECC RAM 64 GB of ECC RAM

Benchmark Methods

All of these tests were run on a fresh Packet.net server running NixOS 18.03 and building Nixpkgs revision 49a6964a425.

For each test, I ran each build five times on each machine with --check which forces a local rebuild:

checkM() {
  nix-build . -A "$1"
  for i in $(seq 1 5); do
    rm result
    nix-collect-garbage
    /run/current-system/sw/bin/time -ao "./time-$1" \
      nix-build . -A "$1" --check
  done
}

Benchmark Results

Kernel Builds

NixOS builds a generously featureful kernel by default, and the build can take some time. However, the compilation is easy to spread across multiple cores. In what we will further see is a theme, the EPYC beat the Intel CPU with by about five minutes, or about 35% speed-up.

nix-build '<nixpkgs>' -A linuxPackages.kernel

Trial m1.xlarge.x86 (seconds) EPYC (seconds) Speed-up
1 823.77 535.30 35.02%
2 821.27 536.94 34.62%
3 824.92 538.45 34.73%
4 827.74 537.79 35.03%
5 827.37 539.98 34.74%

NixOS Tests

The NixOS release process is novel in that package updates happen automatically as an entire cohesive set. We of course test that specific software compiles properly, but we are also able to perform fully automatic integration tests. We test that the operating system can boot, desktop environments work, and even server tests like validating that our MySQL server replication is still working. These tests happen automatically on every release and is a unique benefit to the Nix build system being applied to an operating system.

These tests use QEMU and KVM, and spawn one or more virtual machines running NixOS.

Plasma5 and KDE

The Plasma5 test looks like the following:

  1. First launches a NixOS VM configured to run the Plasma5 desktop manager.

  2. Uses Optical Character Recognition to wait until it sees the system is ready for Alice to log in and then types in her password:
    $machine->waitForText(qr/Alice Foobar/);
    $machine->sendChars("foobar\n");
    
  3. Waits for the Desktop to be showing, which reliably indicates she has finished logging in:
    $machine->waitForWindow("^Desktop ");
    
  4. Launches Dolphin, Konsole, and Settings and waits for each window to appear before continuing:
    $machine->execute("su - alice -c 'dolphin &'");
    $machine->waitForWindow(" Dolphin");
    $machine->execute("su - alice -c 'konsole &'");
    $machine->waitForWindow("Konsole");
    $machine->execute("su - alice -c 'systemsettings5 &'");
    $machine->waitForWindow("Settings");
    
  5. If all of these work correctly, the VM shuts down and the tests pass.

Better than the kernel tests, we’re pushing a 40% improvement.

nix-build '<nixpkgs/nixos/tests/plasma5.nix>'

Trial m1.xlarge.x86 (seconds) EPYC (seconds) Speed-up
1 185.73 115.23 37.96%
2 189.53 116.11 38.74%
3 191.88 115.18 39.97%
4 189.38 116.05 38.72%
5 188.98 115.54 38.86%

MySQL Replication

The MySQL replication test launches a MySQL master and two slaves, and runs some basic replication tests. NixOS allows you to define replication as part of the regular configuration management, so I will start by showing the machine configuration of a slave:

    {
      services.mysql.replication.role = "slave";
      services.mysql.replication.serverId = 2;
      services.mysql.replication.masterHost = "master";
      services.mysql.replication.masterUser = "replicate";
      services.mysql.replication.masterPassword = "secret";
    }
  1. This test starts by starting the $master and waiting for MySQL to be healthy:
    $master->start;
    $master->waitForUnit("mysql");
    $master->waitForOpenPort(3306);
    
  2. Continues to start $slave1 and $slave2 and wait for them to be up:
    $slave1->start;
    $slave2->start;
    $slave1->waitForUnit("mysql");
    $slave2->waitForUnit("mysql");
    $slave1->waitForOpenPort(3306);
    $slave2->waitForOpenPort(3306);
    
  3. It then validates some of the scratch data loaded in to the $master has replicated properly to $slave2:
    $slave2->succeed("
         echo 'use testdb; select * from tests' \
             | mysql -u root -N | grep 4
    ");
    
  4. Then shuts down $slave2:
    $slave2->succeed("systemctl stop mysql");
    
  5. Writes some data to the $master:
    $master->succeed("
         echo 'insert into testdb.tests values (123, 456);' \
             | mysql -u root -N
    ");
    
  6. Starts $slave2, and verifies the queries properly replicated from the $master to the slave:
    $slave2->succeed("systemctl start mysql");
    $slave2->waitForUnit("mysql");
    $slave2->waitForOpenPort(3306);
    $slave2->succeed("
         echo 'select * from testdb.tests where Id = 123;' \
             | mysql -u root -N | grep 456
    ");
    

Due to the multiple VM nature, and increased coordination between the nodes, we saw a 30% increase.

nix-build '<nixpkgs/nixos/tests/mysql-replication.nix>'

Trial m1.xlarge.x86 (seconds) EPYC (seconds) Speed-up
1 42.32 29.94 29.25%
2 43.46 29.48 32.17%
3 42.43 29.83 29.70%
4 43.27 29.66 31.45%
5 42.07 29.37 30.19%

BitTorrent

The BitTorrent test follows the same pattern of starting and stopping a NixOS VM, but this test takes it a step further and tests with four VMs which talk to each other. I could do a whole post just on this test, but in short:

  • a machine serving as the tracker, named $tracker.
  • a client machine, $client1
  • another client machine, $client2
  • a router which facilitates some of the incredible things this test is actually doing.

I’m going to gloss over the details here, but:

  1. The $tracker starts seeding a file:
    $tracker->succeed("opentracker -p 6969 &");
    $tracker->waitForOpenPort(6969);
    my $pid = $tracker->succeed("transmission-cli \
         /tmp/test.torrent -M -w /tmp/data");
    
  2. $client1 fetches the file from the tracker:
    $client1->succeed("transmission-cli \
         http://tracker/test.torrent -w /tmp &");
    
  3. Kills the seeding process on tracker so now only $client1 is able to serve the file:
    $tracker->succeed("kill -9 $pid");
    
  4. $client2 fetches the file from $client1:
    $client2->succeed("transmission-cli \
         http://tracker/test.torrent -M -w /tmp &");
    

If both $client1 and $client2 receive the file intact, the test passes.

This test sees a much lower performance improvement, largely due to the networked coordination across four VMs.

nix-build '<nixpkgs/nixos/tests/bittorrent.nix>'

Trial m1.xlarge.x86 (seconds) EPYC (seconds) Speed-up
1 54.22 45.37 16.32%
2 54.40 45.51 16.34%
3 54.57 45.34 16.91%
4 55.31 45.32 18.06%
5 56.07 45.45 18.94%

The remarkable part I left out is Client1 uses UPnP to open a port of the firewall of the router which Client2 uses to read from Client1.

Standard Environment

Our standard build environment, stdenv, is at the deepest part of this tree and almost nothing can be built until after it is completed. stdenv is like build-essential on Ubuntu.

This is an important part of the performance story for us: Nix builds are represented by a tree, where Nix will schedule as many parallel builds as possible as long as its parents are done building. Single core performance is the primary factor impacting how long these builds take. Shaving off even a few minutes means our entire build cluster is able to get to share the work sooner.

For this reason, the stdenv test is the one exception to the methodology. I wanted to test a full build from bootstrap to a working standard build environment. To force this, I changed the very root build causing everything “beneath” it to require a rebuild by applying the following patch:

diff --git a/pkgs/stdenv/linux/default.nix b/pkgs/stdenv/linux/default.nix
index 63b4c8ecc24..1cd27f216f9 100644
--- a/pkgs/stdenv/linux/default.nix
+++ b/pkgs/stdenv/linux/default.nix
@@ -37,6 +37,7 @@ let

   commonPreHook =
     ''
+
       export NIX_ENFORCE_PURITY="''${NIX_ENFORCE_PURITY-1}"
       export NIX_ENFORCE_NO_NATIVE="''${NIX_ENFORCE_NO_NATIVE-1}"
       ${if system == "x86_64-linux" then "NIX_LIB64_IN_SELF_RPATH=1" else ""}

The impact on build time here is stunning and makes an enormous difference: almost a full 20 minutes shaved off the bootstrapping time.

nix-build '<nixpkgs>' -A stdenv

Trial m1.xlarge.x86 (seconds) EPYC (seconds) Speed-up
1 2,984.24 1,803.40 39.57%
2 2,976.10 1,808.97 39.22%
3 2,990.66 1,808.21 39.54%
4 2,999.36 1,808.30 39.71%
5 2,988.46 1,818.84 39.14%

Conclusion

This EPYC machine has made a remarkable improvement in our build times and is helping the NixOS community push timely security updates and software updates to users and businesses alike. We look forward to expanding our footprint to keep up with the incredible growth of the Nix project.

Thank you to Packet.net for providing this hardware free of charge for this test through their EPYC Challenge.

and thank you Gustav, Daiderd, Andi, and zimbatm for their help with this article

August 02, 2018 12:00 AM

July 26, 2018

Sander van der Burg

Layered build function abstractions for building Nix packages

I have shown quite a few Nix expression examples on my blog. When it is desired to write a Nix expression for a package, it is a common habit to invoke the stdenv.mkDerivation {} function, or functions that are abstractions built around it.

For example, if we want to build a package, such as the trivial GNU Hello package, we can write the following expression:


with import <nixpkgs> {};

stdenv.mkDerivation {
name = "hello-2.10";

src = fetchurl {
url = mirror://gnu/hello/hello-2.10.tar.gz;
sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
};

meta = {
description = "A program that produces a familiar, friendly greeting";
longDescription = ''
GNU Hello is a program that prints "Hello, world!" when you run it.
It is fully customizable.
'';
homepage = http://www.gnu.org/software/hello/manual/;
license = "GPLv3+";
};
}

and build it with the Nix package manager as follows:


$ nix-build
/nix/store/188avy0j39h7iiw3y7fazgh7wk43diz1-hello-2.10

The above code fragment does probably not look too complicated and is quite easy to repeat for build other kinds of GNU Autotools/GNU Make-based packages. However, stdenv.mkDerivation {} is a big/complex function abstraction that has many responsibilities.

Its most important responsibility is to compose a so-called pure build environments, in which various restrictions are imposed on the build scripts to provide better guarantees that builds are pure (meaning: that they always produce the same (nearly) bit-identical result if the dependencies are the same), such as:

  • Build scripts can only write to designated output directories and temp directories. They are restricted from writing to any other file system location.
  • All environment variables are cleared and some of them are set to default or dummy values, such as search path environment variables (e.g. PATH).
  • All build results are made immutable by removing the write permission bits and their timestamps are reset to one second after the epoch.
  • Running builds as unprivileged users.
  • Optionally, builds run in a chroot environment and use namespaces to restrict access to the host filesystem and the network as much as possible.

In addition to purity, the stdenv.mkDerivation {} function has many additional responsibilities. For example, it also implements a generic builder that is clever enough to build a GNU Autotools/GNU Make project without specifying any build instructions.

For example, the above Nix expression for GNU Hello does not specify any build instructions. The generic builder automatically unpacks the tarball, opens the resulting directory and invokes ./configure --prefix=$out; make; make install with the appropriate parameters.

Because stdenv.mkDerivation {} has many responsibilities and nearly all packages in Nixpkgs depend on it, its implementation is very complex (e.g. thousands of lines of code) and hard to change.

As a personal exercise, I have developed a function abstraction with similar functionality from scratch. My implementation can be decomposed into layers in which every abstraction layer gradually adds additional responsibilities.

Writing "raw" derivations


stdenv.mkDerivation is a function abstraction, not a feature of the Nix expression language. To compose "pure" build environments, stdenv.mkDerivation invokes a Nix expression language construct -- the derivation {} builtin.

(As a sidenote: derivation is strictly speaking not a builtin, but an abstraction built around the derivationStrict builtin, but this is something internal to the Nix package manager. It does not matter for the scope of this blog post).

Despite the fact that this low level function is not commonly used, it is also possible to directly invoke it and compose low-level "raw" derivations to build packages. For example, we can write the following Nix expression (default.nix):


derivation {
name = "test";
builder = ./test.sh;
system = "x86_64-linux";
person = "Sander";
}

The above expression invokes the derivation builtin function that composes a "pure" build environment:

  • The name attribute specifies the name of the package, that should appear in the resulting Nix store path.
  • The builder attribute specifies that the test.sh executable should be run inside the pure build environment.
  • The system attribute is used to tell Nix that this build should be carried out for x86-64 Linux systems. When Nix is unable to build the package for the requested system architecture, it can also delegate a build to a remote machine that is capable.
  • All attributes (including the attributes described earlier) are converted to environment variables (e.g. strings, numbers and URLs are converted to strings and the boolean value: 'true' is converted to '1') and can be used by the builder process for a variety of reasons.

We can implement the builder process (the test.sh build script) as follows:


#!/bin/sh -e

echo "Hello $person" > $out

The above script generates a greeting message for the provided person (exposed as an environment variable by Nix) and writes it to the Nix store (the output path is provided by the out environment variable).

We can evaluate the Nix expression (and generate the output file with the Hello greeting) by running:


$ nix-build
/nix/store/7j4y5d8rx1vah5v64bpqd5dskhwx5105-test
$ cat result
Hello Sander

The return value of the derivation {} function is a bit confusing. At first sight, it appears to be a string corresponding to the output path in the Nix store. However, some investigation with the nix repl tool reveals that it is much more than that:


$ nix repl
Welcome to Nix version 2.0.4. Type :? for help.

when importing the derivation:


nix-repl> test = import ./default.nix

and describing the result:


nix-repl> :t test
a set

we will see that the result is actually an attribute set, not a string. By requesting the attribute names, we will see the following attributes:


nix-repl> builtins.attrNames test
[ "all" "builder" "drvAttrs" "drvPath" "name" "out" "outPath" "outputName" "person" "system" "type" ]

It appears that the resulting attribute set has the same attributes as the parameters that we passed to derivation, augmented by the following additional attributes:

  • The type attribute that refers to the string: "derivation".
  • The drvAttrs attribute refers to an attribute set containing the original parameters passed to derivation {}.
  • drvPath and outPath refer to the Nix store paths of the store derivation file and output of the build. A side effect of requesting these members is that the expression gets evaluated or built.
  • The out attribute is a reference to the derivation producing the out result, all is a list of derivations of all outputs produced (Nix derivations can also produce multiple output paths in the Nix store).
  • In case there are multiple outputs, the outputName determines the name of the output path that is the default.

Providing basic dependencies


Although we can use the low-level derivation {} function to produce a very simple output file in the Nix store, it is not very useful on its own.

One important limitation is that we only have a (Bourne-compatible) shell (/bin/sh), but no other packages in the "pure" build environment. Nix prevents unspecified dependencies from being found to make builds more pure.

Since a pure build environment is almost entirely empty (with the exception of the shell), the amount of things we can do in an environment created by derivation {} is very limited -- most of the commands that build scripts run are provided by executables belonging to external packages, e.g. commands such as cat, ls (GNU Coreutils), grep (GNU Grep) or make (GNU Make) and should be added to the PATH search environment variable in the build environment.

We may also want to configure additional environment variables to make builds more pure -- for example, on Linux systems, we want to set the TZ (timezone) environment variable to UTC to prevent error messages, such as: "Local time zone must be set--see zic manual page".

To make the execution of more complex build scripts more convenient, we can create a setup script that we can include in a every build script that adds basic utilities to the PATH search environment variable, configures these additional environment variables, and sets the SHELL environment variable to the bash shell residing in the Nix store. We can create a package named: stdenv that provides a setup script to accomplish this:


{bash, basePackages, system}:

let
shell = "${bash}/bin/sh";
in
derivation {
name = "stdenv";
inherit shell basePackages system;
builder = shell;
args = [ "-e" ./builder.sh ];
}

The builder script of the stdenv package can be implemented as follows:

set -e

# Setup PATH for base packages
for i in $basePackages
do
basePackagesPath="$basePackagesPath${basePackagesPath:+:}$i/bin"
done

export PATH="$basePackagesPath"

# Create setup script
mkdir $out
cat > $out/setup <<EOF
export SHELL=$shell
export PATH="$basePackagesPath"
EOF

# Allow the user to install stdenv using nix-env and get the packages
# in stdenv.
mkdir $out/nix-support
echo "$basePackages" > $out/nix-support/propagated-user-env-packages

The above script adds all base packages (GNU Coreutils, Findutils, Diffutils, sed, grep, gawk and bash) to the PATH of builder and creates a script in $out/setup that exports the PATH environment variable and the location to the bash shell.

We can use the stdenv (providing this setup script) as a dependency for building a package, such as:


{stdenv}:

derivation {
name = "hello";
inherit stdenv;
builder = ./builder.sh;
system = "x86_64-linux";
}

In the corresponding builder script, we include the setup script in the first line and we, for example, invoke various external commands to generate a shell script that says: "Hello world!":


#!/bin/sh -e
source $stdenv/setup

mkdir -p $out/bin

cat > $out/bin/hello <<EOF
#!$SHELL -e

echo "Hello"
EOF

chmod +x $out/bin/hello

The above script works because the setup script adds GNU Coreutils (that includes cat, mkdir and chmod) to the PATH of the builder.

Writing more simple derivations


Using a setup script makes writing build scripts somewhat practical, but there are still a number inconveniences we have to cope with.

The first inconvenience is the system parameter -- in most cases, we want to build a package for the same architecture as the host system's architecture and preferably we want the same architecture for all other packages that we intend to deploy.

Another issue is the shell. /bin/sh is, in a sandbox-enabled Nix installations, a minimal Bourne-compatible shell provided by Busybox, or a reference to the host system's shell in non-sandboxed installations. The latter case could be considered an impurity, because we do not know what kind of shell (e.g. bash, dash, ash ?) or version of a shell we are using (e.g. 3.2.57, 4.3.30 ?). Ideally, we want to use a shell that is provided as a Nix package in the Nix store, because that version is pure.

(As a sidenote: in Nixpkgs, we use the bash shell to run build commands, but this is not a strict requirement. For example, GNU Guix (a package manager that uses several components of the Nix package manager) uses both Guile as a host and guest language. In theory, we could also launch a different kind of interpreter than bash).

The third issue is the meta parameter -- for every package, it is possible to specify meta-data, such as a description, license and homepage reference as an attribute set. Unfortunately, attribute sets cannot be converted to environment variables. To deal with this problem, the meta attribute needs to be removed before we invoke derivation {} and be readded to the return attribute set. (IMO I believe this ideally should be something the Nix package manager could solve by itself).

We can hide all these inconveniences by creating a simple abstraction function that I will call: stdenv.simpleDerivation that can be implemented as follows:


{stdenv, system, shell}:
{builder, ...}@args:

let
extraArgs = removeAttrs args [ "builder" "meta" ];

buildResult = derivation ({
inherit system stdenv;
builder = shell; # Make bash the default builder
args = [ "-e" builder ]; # Pass builder executable as parameter to bash
setupSimpleDerivation = ./setup.sh;
} // extraArgs);
in
buildResult //
# Readd the meta attribute to the resulting attribute set
(if args ? meta then { inherit (args) meta; } else {})

The above Nix expression basically removes the meta argument, then invokes the derivation {} function, sets the system parameter, uses bash as builder and passes the builder executable as an argument to bash. After building the package, the meta attribute gets readded to the result.

With this abstraction, we can reduce the complexity of the previously shown Nix expression to something very simple:


{stdenv}:

stdenv.simpleDerivation {
name = "hello";
builder = ./builder.sh;
meta = {
description = "This is a simple testcase";
};
}

The function abstraction is also sophisticated enough to build something more complex, such as GNU Hello. We can write the following Nix expression that passes all dependencies that it requires as function parameters:


{stdenv, fetchurl, gnumake, gnutar, gzip, gcc, binutils}:

stdenv.simpleDerivation {
name = "hello-2.10";
src = fetchurl {
url = mirror://gnu/hello/hello-2.10.tar.gz;
sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
};
inherit stdenv gnumake gnutar gzip gcc binutils;
builder = ./builder.sh;
}

We can use the following builder script to build GNU Hello:


source $setupSimpleDerivation

export PATH=$PATH:$gnumake/bin:$gnutar/bin:$gzip/bin:$gcc/bin:$binutils/bin

tar xfv $src
cd hello-2.10
./configure --prefix=$out
make
make install

The above script imports a setup script configuring basic dependencies, then extends the PATH environment variable with additional dependencies, and then executes the commands to build GNU Hello -- unpacking the tarball, running the configure script, building the project, and installing the package.

The run command abstraction


We can still improve a bit upon the function abstraction shown previously -- one particular inconvenience that remains is that you have to write two files to get a package built -- a Nix expression that composes the build environment and a builder script that carries out the build steps.

Another repetitive task is configuring search path environment variables (e.g. PATH, PYTHONPATH, CLASSPATH etc.) to point to the appropriate directories in the Nix store. As may be noticed by looking at the code of the previous builder script, this process is tedious.

To address these inconveniences, I have created another abstraction function called: stdenv.runCommand that extends the previous abstraction function -- when no builder parameter has been provided, this function executes a generic builder that will evaluate the buildCommand environment variable containing a string with shell commands to execute. This feature allows us to rewrite the first example (that generates a shell script) to one file:


{stdenv}:

stdenv.runCommand {
name = "hello";
buildCommand = ''
mkdir -p $out/bin
cat > $out/bin/hello <<EOF
#! ${stdenv.shell} -e

echo "Test"
EOF
chmod +x $out/bin/hello
'';
}

Another feature of the stdenv.runCommand abstraction is to provide a generic mechanism to configure build-time dependencies -- all build-time dependencies that a package needs can be provided as a list of buildInputs. The generic builder carries out all necessary build steps to make them available. For example, when a package provides a bin/ sub folder, then it will be automatically added to the PATH environment variable.

Every package can bundle a setup-hook.sh script that modifies the build environment so that it knows how dependencies for this package can be configured. For example, the following partial expression represents the Perl package that bundles a setup script:


{stdenv, ...}:

stdenv.mkDerivation {
name = "perl";
...
setupHook = ./setup-hook.sh
}

The setup hook can automatically configure the PERL5LIB search path environment variable for all packages that provide Perl modules:


addPerlLibPath()
{
addToSearchPath PERL5LIB $1/lib/perl5/site_perl
}

envHooks+=(addPerlLibPath)

When we add perl as a build input to a package, then its setup hook configures the generic builder in such a way that the PERL5LIB environment variable is automatically configured when we provide a Perl module as a build input.

We can also more conveniently build GNU Hello, by using the buildInputs parameter:


{stdenv, fetchurl, gnumake, gnutar, gzip, gcc, binutils}:

stdenv.runCommand {
name = "hello-2.10";
src = fetchurl {
url = mirror://gnu/hello/hello-2.10.tar.gz;
sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
};
buildInputs = [ gnumake gnutar gzip gcc binutils ];
buildCommand = ''
tar xfv $srcb
cd hello-2.10
./configure --prefix=$out
make
make install
'';
}

Compared to the previous GNU Hello example, this Nix expression is much simpler and more intuitive to write.

The run phases abstraction


We can improve the ease of use for build processes even further. GNU Hello, and many other GNU packages and other system software used for Linux are GNU Autotools/GNU Make based and follow similar conventions including the build commands you need to carry out. Likewise, many other software projects use standardized build tools that follow conventions.

As a result, when you have to maintain a collection of packages, you probably end up writing the same kinds of build instructions over and over again.

To alleviate this problem, I have created another abstraction layer, named: stdenv.runPhases making it possible to define and execute phases in a specific order. Every phase has a pre and post hook (a script that executes before and after each phase) and can be disabled or reenabled with a do* or dont* flag.

With this abstraction function, we can divide builds into phases, such as:


{stdenv}:

stdenv.runPhases {
name = "hello";
phases = [ "build" "install" ];
buildPhase = ''
cat > hello <<EOF
#! ${stdenv.shell} -e
echo "Hello"
EOF
chmod +x hello
'';
installPhase = ''
mkdir -p $out/bin
mv hello $out/bin
'';
}

The above Nix expression executes a build and install phase. In the build phase, we construct a script that echoes "Hello", and in the install phase we move the script into the Nix store and we make it executable.

In addition to environment variables, it is also possible to define the phases in a setup script as shell functions. For example, we can also use a builder script:


{stdenv}:

stdenv.runPhases {
name = "hello2";
builder = ./builder.sh;
}

and define the phases in the builder script:


source $setupRunPhases

phases="build install"

buildPhase()
{
cat > hello <<EOF
#! $SHELL -e
echo "Hello"
EOF
chmod +x hello
}

installPhase()
{
mkdir -p $out/bin
mv hello $out/bin
}

genericBuild

Another feature of this abstraction is that we can also define exitHook and failureHook parameters that will be executed if the builder succeeds or fails.

In the next sections, I will show abstractions built on top of stdenv.runPhases that can be used to hide implementation details of common build procedures.

The generic build abstraction


For many build procedures, we need to carry out the same build steps, such as: unpacking the source archives, applying patches, and stripping debug symbols from the resulting ELF executables.

I have created another build function abstraction named: stdenv.genericBuild that implements a number of common build phases:

  • The unpack phase generically unpacks the provided sources, makes it content writable and opens the source directory. The unpack command is determined by the unpack hook that each potential unpacker provides -- for example, the GNU tar package includes a setup hook that untars the file if it looks like a tarball or compressed tarball:


    _tryUntar()
    {
    case "$1" in
    *.tar|*.tar.gz|*.tar.bz2|*.tar.lzma|*.tar.xz)
    tar xfv "$1"
    ;;
    *)
    return 1
    ;;
    esac
    }

    unpackHooks+=(_tryUntar)
  • The patch phase applies any patch that is provided by the patches parameter uncompressing them when necessary. The uncompress file operation also works with setup hooks -- uncompressor packages (such as gzip and bzip2) provide a setup hook that uncompresses the file if it is of the right filetype.
  • The strip phase processes all sub directories containing ELF binaries (e.g. bin/ and lib/) and strips their debugging symbols. This reduces the size of the binaries and removes non-deterministic timestamps.
  • The patchShebangs phase processes all scripts with a shebang line and changes it to correspond to a path in the Nix store.
  • The compressManPages phase compresses all manual pages with gzip.

We can also add GNU patch as as base package for this abstraction function, since it is required to execute the patch phase. As a result, it does not need to be specified as a build dependency for each package.

This function abstraction alone is not very useful, but it captures all common aspects that most build tools use, such as GNU Make, CMake or SCons projects.

I can reduce the size of the previously shown GNU Hello example Nix expression to the following:


{stdenv, fetchurl, gnumake, gnutar, gzip, gcc, binutils}:

stdenv.genericBuild {
name = "hello-2.10";
src = fetchurl {
url = mirror://gnu/hello/hello-2.10.tar.gz;
sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
};
buildInputs = [ gnumake gnutar gzip gcc binutils ];
buildCommandPhase = ''
./configure --prefix=$out
make
make install
'';
}

In the above expression, I no longer have to specify how to unpack the download GNU Hello source tarball.

GNU Make/GNU Autotools abstraction


We can extend the previous function abstraction even further with phases that automate a complete GNU Make/GNU Autotools based workflow. This abstraction is what we can call stdenv.mkDerivation and is comparable in terms of features with the implementation in Nixpkgs.

We can adjust the phases to include a configure, build, check and install phase. The configure phase checks whether a configure script exists and executes it. The build, check and install phases will execute: make, make check and make install with appropriate parameters.

We can also add common packages that we need to build these projects as base packages so that they no longer have to be provided as a build input: GNU Tar, gzip, bzip2, xz, GNU Make, Binutils and GCC.

With these additional phases and base packages, we can reduce the GNU Hello example to the following expression:


{stdenv, fetchurl}:

stdenv.mkDerivation {
name = "hello-2.10";
src = fetchurl {
url = mirror://gnu/hello/hello-2.10.tar.gz;
sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
};
}

The above Nix expression does not contain any installation instructions -- the generic builder is able to figure out all steps on its own.

Composing custom function abstractions


I have shown several build abstraction layers implementing most features that are in the Nixpkgs version of stdenv.mkDerivation. Aside from clarity, another objective of splitting this function in layers is to make the composition of custom build abstractions more convenient.

For example, we can implement the trivial builder named: writeText whose only responsibility is to write a text file into the Nix store, by extending stdenv.runCommand. This abstraction suffices because writeText does not require any build tools, such as GNU Make and GCC, and it also does not need any generic build procedure executing phases:


{stdenv}:

{ name # the name of the derivation
, text
, executable ? false # run chmod +x ?
, destination ? "" # relative path appended to $out eg "/bin/foo"
, checkPhase ? "" # syntax checks, e.g. for scripts
}:

stdenv.runCommand {
inherit name text executable;
passAsFile = [ "text" ];

# Pointless to do this on a remote machine.
preferLocalBuild = true;
allowSubstitutes = false;

buildCommand = ''
target=$out${destination}
mkdir -p "$(dirname "$target")"

if [ -e "$textPath" ]
then
mv "$textPath" "$target"
else
echo -n "$text" > "$target"
fi

[ "$executable" = "1" ] && chmod +x "$target" || true
'';
}

We can also make a builder for Perl packages, by extending: stdenv.mkDerivation -- Perl packages also use GNU Make as a build system. Its only difference is the configuration step -- it runs Perl's MakeMaker script to generate the Makefile. We can simply replace the configuration phase for GNU Autotools by an implementation that invokes MakeMaker.

When developing custom abstractions, I basically follow this pattern:


{stdenv, foo, bar}:
{name, buildInputs ? [], ...}@args:

let
extraArgs = removeAttrs args [ "name" "buildInputs" ];
in
stdenv.someBuildFunction ({
name = "mypackage-"+name;
buildInputs = [ foo bar ] ++ buildInputs;
} // extraArgs)

  • A build function is a nested function in which the first line is a function header that captures the common build-time dependencies required to build a package. For example, when we want to build Perl packages, then perl is such a common dependency.
  • The second line is the inner function header that captures the parameters that should be passed to the build function. The notation allows an arbitrary number of parameters. The parameters in the { } block (name, buildInputs) are considered to have a specific use in the body of the function. The remainder of parameters are non-essential -- they are used as environment variables in the builder environment or they can be propagated to other functions.
  • We compose an extraArgs variable that contains all non-essential arguments that we can propagate to the build function. Basically, all function arguments that are used in the body need to be removed and function arguments that are attribute sets, because they cannot be converted to strings.
  • In the body of the function, we set up important aspects of the build environment, such as the mandatory build parameters, and we propagate the remaining function arguments to the builder abstraction function.

Following this pattern also ensures that the builder is flexible enough to be extended and modified. For example, by extending a function that is based on stdenv.runPhases the builder can be extended with custom phases and build hooks.

Discussion


In this blog post, I have derived my own reimplementation of Nixpkgs's stdenv.mkDerivation function that consists of the following layers each gradually adding functionality to the "raw" derivation {} builtin:

  1. "Raw" derivations
  2. The setup script ($stdenv/setup)
  3. Simple derivation (stdenv.simpleDerivation)
  4. The run command abstraction (stdenv.runCommand)
  5. The run phases abstraction (stdenv.runPhases)
  6. The generic build abstraction (stdenv.genericBuild)
  7. The GNU Make/GNU Autotools abstraction (stdenv.mkDerivation)

The features that the resulting stdenv.mkDerivation provides are very similar to the Nixpkgs version, but not entirely identical. Most notably, cross compiling support is completely absent.

From the experience, I have a number of improvement suggestions that we may want to implement in Nixpkgs version to improve the quality and clarity of the generic builder infrastructure:

  • We could also split the implementation of stdenv.mkDerivation and the corresponding setup.sh script into layered sub functions. Currently, the setup.sh script is huge (e.g. over 1200 LOC) and has many responsibilities (perhaps too many). By splitting the build abstraction functions and their corresponding setup scripts, we can separate concerns better and reduce the size of the script so that it becomes more readable and better maintainable.
  • In the Nixpkgs implementation, the phases that the generic builder executes are built for GNU Make/GNU Autotools specifically. Furthermore, the invocation of pre and post hooks and do and dont flags are all hand coded for every phase (there is no generic mechanism that deals with them). As a result, when you define a new custom phase, you need to reimplement the same aspects over and over again. In my implementation, you only have to define phases -- the generic builder automatically executes the coresponding pre and post hooks and evaluates the do and dont flags.
  • In the Nixpkgs implementation there is no uncompressHook -- as a result, the decompression of patch files is completely handcoded for every uncompressor, e.g. gzip, bzip2, xz etc. In my implementation, we can delegate this responsibility to any potential uncompressor package.
  • In my implementation, I turned some of the phases of the generic builder into command-line tools that can be invoked outside the build environment (e.g. patch-shebangs, compress-man). This makes it easier to experiment with these tools and to make adjustments.

The biggest benefit of having separated concerns is flexibility when composing custom abstractions -- for example, the writeText function in Nixpkgs is built on top of stdenv.mkDerivation that includes GNU Make and GCC as dependencies, but does not depend on it. As a result, when one of these packages get updated all generated text files need to be updated as well, while there is no real dependency on it. When using a more minimalistic function, such as stdenv.runCommand this problem will go away.

Availability


I have created a new GitHub repository called: nix-lowlevel-experiments. It contains the implementation of all function abstractions described in this blog post, including some test cases that demonstrate how these functions can be used.

In the future, I will probably experiment with other low level Nix concepts and add them to this repository as well.

by Sander van der Burg (noreply@blogger.com) at July 26, 2018 09:58 PM

July 25, 2018

Matthew Bauer

Beginner’s guide to cross compilation in Nixpkgs

1 What is cross compilation?

First, compilation refers to converting human-readable source code into computer-readable object code. Usually the computer you are building the code for is the same as the computer you are running on1. In cross compilation, however, that is not the case. We can build code for any computer that our compiler supports!

Cross-compilation is not a new idea at all. GCC and Autoconf are ancient tools that we use internally in Nixpkgs. But, getting those ideas to work well with Nix’s functional dependency model has taken years and years of work from the Nix community. We are finally to the point where an end user can easily start cross compiling things themselves.

2 Unstable channel

/If you do not have Nix installed, you can install it available at NixOS.org. The rest of the guide will assume that Nix is already installed./

Much work has gone into bringing cross compilation support to Nixpkgs. While Nixpkgs has had some support for cross compiling for awhile, recent changes have made cross compilation much easier and more elegant. These changes will be required for this guide. We plan to have a stable version of this ready for 18.09. Before that though, you will need to use the unstable channel. Things can be built with the unstable channel fairly easily with Nix 2.0. For instance, to build the hello program,

nix build -f channel:nixos-unstable hello

The rest of the guide will use nixos-unstable as the channel. However, once 18.09 is released, you should be able to also use the stable channel.

3 Building things

One of the important principles of cross compilation in Nixpkgs is handling native and cross compilation identically. This means that it should be possible to cross-compile any package in Nixpkgs with little to no modification at all. If your derivation specifies its dependencies correctly, Nix/Nixpkgs can figure out how to build it.

So now it’s time to show what we can do with Nixpkgs cross compilation framework2. I’ve compiled a short list of cross package sets along with their corresponding attribute names.

  • Raspberry Pi (pkgsCross.raspberryPi)
  • x86_64 Musl (pkgsCross.musl64)
  • Android (pkgsCross.aarch64-android-prebuilt)
  • iPhone (pkgsCross.iphone64)
  • Windows (pkgsCross.mingwW64)

So, if you are familiar with Nixpkgs, you would know that if you wanted to build Emacs for your native computer you can just run,

$ nix build -f channel:nixos-unstable pkgs.emacs

Likewise, if you wanted to build Emacs for a Raspberry Pi, you can just run,

$ nix build -f channel:nixos-unstable pkgsCross.raspberryPi.emacs

The built package will be in the same ARM machine code used by the Raspbery Pi. The important thing to notice here is that we have the power to build in any package in Nixpkgs for any of the platforms listed above. Of course, many of these will have issues due to not being portable, but with time we can make both Nixpkgs & the free software world better at handling cross compilation. Any of the software listed in ‘nix search’ should be possible to cross compile through the pkgsCross attribute.

Some more examples of things that I have worked on,

  1. Windows

    $ nix build -f channel:nixos-unstable pkgsCross.mingw32.hello
    $ nix run -f channel:nixos-unstable wine -c ./result/bin/hello.exe
    Hello, world!
    
  2. Android

    $ nix build -f channel:nixos-unstable \
          pkgsCross.aarch64-android-prebuilt.curl
    
  3. iPhone3

    $ nix build -f channel:nixos-unstable \
          pkgsCross.iphone64.haskell.packages.jq
    

Notice that the pkgsCross attribute is just sugar to a more powerful & composable interface to Nixpkgs. This can be specified from the command line with,

$ nix build -f channel:nixos-unstable \
      --arg crossSystem '{ config = "<arch>-<vendor>-<kernel>-<environment>"; }'

For instance you may want to cross-compile Firefox for ARM64 Linux. This is as easy as4:

$ nix build -f channel:nixos-unstable \
      --arg crossSystem '{ config = "arm64-unknown-linux-gnu"; }'

You can be much more specific with what you want through crossSystem. Many more combinations are possible, but they all revolve around that four-part string config listed. It corresponds to <arch>-<vendor>-<kernel>-<environment> and is commonly called the LLVM triple5. The LLVM triple has become the standard way to specify systems accross many free software toolchains including GCC, Binutils, Clang, libffi, etc. There is more information that can be specified in crossSystem & localSystem within Nixpkgs but this is not covered here as they are heavily dependent on the specific toolchain being used.

4 When things break

While the fundamentals of cross compiling in Nixpkgs are very good, individual packages will sometimes be broken. This is sometime because the package definition in Nixpkgs is incorrect. There are some common mistakes that occur that I want to cover here. First, the difference between ‘build-time’ vs ‘runtime’ dependencies6.

  • build-time dependencies: tools that will be run on the computer doing the cross compiling
  • runtime dependencies: libraries and tools that will run on the computer we are targeting.

In Nixpkgs, build-time dependencies should be put in nativeBuildInputs. Runtime dependencies should be put in buildInputs. Currently, this distinction has no effect on native compilation but it is crucial for correct cross-compilation. There are proposals to Nixpkgs to enforce the use of buildInputs as nativeBuildInputs even on native builds but this is yet to be agreed on7.

Sometimes your package will pull in a dependency indirectly so that dependency is not listed in buildInputs or nativeBuildInputs. This breaks the package splicing that goes on behind the scenes to make pick up the package set to get each package. To fix it, you will have to splice the package yourself. This is fairly straightforward. For examples, let’s say that your package depends on the pkgs.git git executable to be available through the GIT_CMD variable, which means it is not listed in nativeBuildInputs. In this case, you should instead refer to git as pkgs.buildPackages.git. This will pick up the build package set instead of the target package set.

There are a few more things that can go wrong within Nixpkgs. If you need to conditionally do something only when cross compiling (say a configure flag like --enable-cross-compilation), you should use stdenv.hostPlatform != stdenv.buildPlatform. If you want to check, for instance, that the platform you are building for is a Windows computer, just use stdenv.hostPlatform.isWindows, in the same way that you can also check for Linux with stdenv.hostPlatform.isLinux. These cases are often necessary, but remember they should only be used when absolutely needed. The more code we share between platforms, the more code is tested.

Sometimes packages are just not written in a cross-friendly way. This will usually happen just because the software author has not thought of how to handle cross compilation8. We want to work with software authors to make this process easier & contribute to the portability of free software. This takes time but we are definitely making progress. Contributions are always encouraged to the Nixpkgs repo.

5 Further reading

The concepts introduced here are also available in the Nixpkgs manual. These are the relevant sections/chapters:

GNU Automake also has a section on build vs. host vs. target. This will help clarify some of the naming conventions in Nixpkgs:

Footnotes:

1

This is referred to as native compilation.

2

All examples are provided by the file lib/systems/examples.nix in Nixpkgs.

3

Cross-compilation to iPhone, unfortunately, requires that you download the unfree XCode environment. This is a consequence of Apple’s choices regarding what toolchains they allow.

4

In fact, each of these correspond to a value for crossSystem listed in lib/systems/examples.nix.

5

Of course there are 4 of them, so LLVM quadruple seems like a better name.

6

Like a few other parts of this article, this is somewhat of a simplification. There are many other types of dependencies but they all revolve around the build-time vs runtime distinction.

7

See strictDeps in pkgs/stdenv/generic/setup.sh.

8

Or even worse, they have thought about cross-compilation, but embraced many antipatterns that break with Nixpkgs’ cross-compilation framework.

July 25, 2018 12:00 AM

July 11, 2018

Graham Christensen

cache.nixos.org, now more local!

I’m delighted to be able to announce that users all around the world will now have a great experience when fetching from the NixOS cache.

I heard several times from users in Hong Kong and Singapore that the cache was “slow”, but I didn’t know it was this slow! After working closely with a team of Nix users in Bangalore, I experienced first-hand just how eye-wateringly slow it could be.

The NixOS cache is now being served from all of AWS CloudFront edge locations, significantly reducing latency for users in Asia, Africa, South America, and Oceania.

By expanding the cache’s distribution settings to include all of the edge locations, performance has been substantially improved build time:

    Sydney
GHC
117.06 MiB
Before 178.491s
  After
Cold Cache
73.612s
  After
Hot Cache
15.707s
     
Graphical ISO closure
1,660.95 MiB
Before 2,326.957s
  After
Cold Cache
376.014s
  After
Hot Cache
25.328s

Experiments in Tokyo and Hong Kong produced similar results.

NixOS’s cache is stored in AWS S3, and distributed using AWS CloudFront. This combination gives us the excellent durability guarantees of S3 combined with the large geographical distribution of CloudFront.

Until today, the NixOS cache was only served through edge nodes in the United States, Canada, and Europe.

A big thank-you to Amine Chikhaoui, and Eelco Dolstra for their help in researching this change and turning on such a massive improvement.

July 11, 2018 12:00 AM

June 01, 2018

Domen Kozar

Announcing Cachix - Binary Cache as a Service

In the last 6 years working with Nix and mostly in last two years full-time, I've noticed a few patterns.

These are mostly direct or indirect result of not having a "good enough" infrastructure to support how much Nix has grown (1600+ contributors, 1500 pull requests per month).

Without further ado, I am announcing https://cachix.org - Binary Cache as a Service that is ready to be used after two months of work.

What problem(s) does cachix solve?

The main motivation is to save you time and compute resources waiting for your packages to build. By using a shared cache of already built packages, you'll only have to build your project once.

This should also speed up CI builds, as Nix can take use of granular caching of each package, rather than caching the whole build.

Another one (which I personally consider even more important) is decentralization of work produced by Nix developers. Up until today, most devs pushed their software updates into the nixpkgs repository, which has the global binary cache at https://cache.nixos.org.

But as the community grew, fitting different ideologies into one global namespace became impossible. I consider nixpkgs community to be mature but sometimes clash of ideologies with rational backing occurs. Some want packages to be featureful by default, some prefer them to be minimalist. Some might prefer lots of configuration knobs available (for example cross-compilation support or musl/glib swapping), some might prefer the build system to do just one thing, as it's easier to maintain.

These are not right or wrong opinions, but rather a specific view of use cases that software might or might not cover.

There are also many projects that don't fit into nixpkgs because their releases are too frequent, they are not available under permissive license, are simpler to manage over complete control or maintainers simply disagree with requirements that nixpkgs developers impose on contributors.

And that's fine. What we've learned in the past is not to fight these ideas, but allow them to co-exist in different domains.

If you're interested:

Domen (domen@enlambda.com)

by Domen Kožar at June 01, 2018 10:00 AM

May 13, 2018

Matthew Bauer

Channel Changing with Nix

1 Introduction to channels

One of the many underappreciated feature of Nix is its ability to travel back in time. Functional dependencies mean that you can easily pull in old releases of NixOS & Nixpkgs without changing your environment at all! It’s surprisingly easy in Nix 2.0 with its support for Import From Derivation.

First, I will provide some code to get us started. This Nix script is what I use as my “channel changer”. It bootstraps the use of old channels. In Nix-world, channels are just what we call the CI-tested branch of NixOS/Nixpkgs1. The NixOS maintainers have been making releases consistently since 2013, so there is a lot of interesting history.

2 Channel changing

Here is my script that I will refer to later on in the post as “channels.nix” (be sure to try it out yourself!)2

let mapAttrs = f: set: builtins.listToAttrs (
      map (attr: { name = attr; value = f set.${attr}; })
    (builtins.attrNames set));
    channels = {
      aardvark    = "13.10";
      baboon      = "14.04";
      caterpillar = "14.12";
      dingo       = "15.09";
      emu         = "16.03";
      flounder    = "16.09";
      gorilla     = "17.03";
      hummingbird = "17.09";
      impala      = "18.03";
    };
in mapAttrs (v:
     import (builtins.fetchTarball
       "https://nixos.org/channels/nixos-${v}/nixexprs.tar.xz") {})
   channels

As you can see from the script there have been 9 releases in total. We use a different letter of the alphabet for each release, starting with A for Aardvark. We are now up to I for Impala3. New releases happen every 6 months with Aardvark released in December 2013. The releases are versioned as YY.MM which is a common practice for Linux distros.

3 ‘nix run’ magic

In my Nix script, I have created an “attribute” for each version that has been released. With Nix 2.0, it is very easy to run packages from them. Here is the command to run hello world from Hummingbird.

nix run -f channels.nix hummingbird.hello -c hello
Hello, world!

This has run the hello executable from the hummingbird release. Since you are most likely not running Hummingbird, it may take a while to the first time. However, once Nix has downloaded the needed files future execution will be instantaneous. The package is completely self-contained! To start, we will do examples in Impala (18.03) so that things go a little faster.

There are lots of packages in Nixpkgs so we don’t have to restrict ourselves to just hello. Let’s try out cowsay first.

nix run -f channels.nix impala.cowsay -c cowsay hello
 _______
< hello >
 -------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

There are many, many more of these commands. I’ve included a few below for you to try out on your own.

# Look up the weather
nix run -f channels.nix impala.curl -c curl wttr.in/seville

# Download music
nix run -f channels.nix impala.youtube-dl -c \
    youtube-dl -t --extract-audio \
    --audio-format mp3 \
    https://www.youtube.com/watch?v=dQw4w9WgXcQ

# Go see a Star War
nix run -f channels.nix impala.telnet -c telnet towel.blinkenlights.nl 666
nix run -f channels.nix impala.sox -c bash -c \
    'for n in E2 A2 D3 G3 B3 E4;
     do play -n synth 4 pluck $n repeat 2;
     done'

# Play Nethack
nix run -f channels.nix impala.nethack -c nethack

# Get your fortune
nix run -f channels.nix impala.fortune -c fortune

4 The macOS+Nix odyssey

The fact that Nix works so well on macOS is a miracle in its own right. Apple has a proprietary ABI but Nix is intended to be used with free software. To get around this, many hacks are necessary including taking Apple’s standard C library4. Anyway, I was interested in how well the binaries produced by Nixpkgs hold up on my MacBook. For reference, here are the versions of macOS available when each release happened. Those familiar with macOS internals will remember some significant differences between these versions.

NixOS release macOS release
Aardvark (13.10) Mountain Lion (10.8)
Baboon (14.04) Mavericks (10.9)
Caterpillar (14.12) Yosemite (10.10)
Dingo (15.09) Yosemite (10.10)
Emu (16.03) El Capitan (10.11)
Flounder (16.09) El Capitan (10.11)
Gorilla (17.03) Sierra (10.12)
Hummingbird (17.09) High Sierra (10.13)
Impala (18.03) High Sierra (10.13)

So, my MacBook is running the latest macOS 10.13. Naturally we can test that Impala & Hummingbird will work correctly. hello is a good tester, of course, not comprehensive.

nix run -f channels.nix impala.hello -c hello
Hello, world!

nix run -f channels.nix hummingbird.hello -c hello
Hello, world!

But now let’s test Gorilla. It was released when macOS Sierra was still around but the ABI should be compatible.

nix run -f channels.nix gorilla.hello -c hello
dyld: Library not loaded: /usr/lib/system/libsystem_coretls.dylib
 Referenced from: /nix/store/v7i520r9c2p8z6vk26n53hfrxgqn8cl9-Libsystem-osx-10.11.6/lib/libSystem.B.dylib
 Reason: image not found
sh: line 1: 23628 Abort trap: 6           nix run -f channels.nix gorilla.hello -c hello

Oh no!

We can see that libSystem 10.11 has been downloaded for us5. However, libSystem is referring to an image that isn’t on our machine. libsystem_coretls.dylib must have existed in 10.11 macOS but been removed since then6.

At this point, it may look like Nixpkgs will be broken going backwards. But, I want to try Flounder just to see what happens.

nix run -f channels.nix flounder.hello -c hello
Hello, world!

Amazingly, it worked! I am still not sure what the differences are, but it seems that the older executable is still available. Let’s try out Emu to see what happens there.

nix run -f channels.nix emu.hello -c hello
builder for '/nix/store/s41jnb4kmxxbwj40c5l88k9ma0mwfy0b-hello-2.10.drv' failed due to signal 4 (Illegal instruction: 4)
error: build of '/nix/store/s41jnb4kmxxbwj40c5l88k9ma0mwfy0b-hello-2.10.drv' failed

Wow! Again we hit an issue. This is the infamouse Illegal instruction: 4 bug that is frequently hit in Nixpkgs7. It occurs when an executable uses instructions that have been blocked by the XNU kernel. This is usually because they are considered insecure so a patch is needed to fix it. We no longer support Emu, so this is probably the end of the line. Let’s try Dingo out just to be sure though.

nix run -f channels.nix dingo.hello -c hello
builder for '/nix/store/1cyagihl211vsis9bz09cqaz3h2yyc23-libxml2-2.9.3.drv' failed with exit code 77; last 10 log lines:
 checking for awk... awk
 checking whether make sets $(MAKE)... yes
 checking whether make supports nested variables... yes
 checking whether make supports nested variables... (cached) yes
 checking for gcc... gcc
 checking whether the C compiler works... no
 configure: error: in `/private/tmp/nix-build-libxml2-2.9.3.drv-0/libxml2-2.9.3':
 configure: error: C compiler cannot create executables
 See `config.log' for more details

cannot build derivation '/nix/store/jd4y5aps1z61jqbhsz1gy408zwwa49w4-clang-3.6.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/n4q29z97dc1p9mqrn2ydhlfmsqwbgx0j-libarchive-3.1.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/vh2bh7gaw2m0rgxscf3mhm1d3rz3xwfg-clang-wrapper-3.6.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/zg90kfmf99h03z0fl03gw3gh105mb02c-cmake-3.3.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/45ndaky3079nd78042384f8hbidq7f7q-libc++abi-3.6.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/mmyz6rrddfahwl23i9d9vjh7wa8irp5k-stdenv-darwin-boot-3.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/lqjabx84kndk75y8m0lq7zh5190k6zzz-hello-2.10.drv': 1 dependencies couldn't be built
error: build of '/nix/store/lqjabx84kndk75y8m0lq7zh5190k6zzz-hello-2.10.drv' failed

This is a curious error because it is very different from the previous one. Back here we were still using Clang 3.3 & it looks like bootstrapping is failing on our newer machines. I was not using Nix at this time (late 2015), so I will have to defer to someone who remembers that time better. Let’s keep going.

nix run -f channels.nix caterpillar.hello -c hello
error: attribute 'hello' in selection path 'caterpillar.hello' not found

nix run -f channels.nix baboon.hello -c hello
error: attribute 'hello' in selection path 'baboon.hello' not found

nix run -f channels.nix aardvark.hello -c hello
error: attribute 'hello' in selection path 'aardvark.hello' not found

I’ve grouped them together because they have the same output. It appears that hello was not available back then! I’m not sure what is going on. Again, I will defer to someone else to explain why this happens. But, I know for a fact that GNU Hello is one of the first packages to be packaged in the Nix language8.

5 Conclusion

I wanted to also look at what happens on Linux when you go back through channels. I don’t have time currently so I am just including what I have. Anyway, if you are able to report back what happens on Linux when running these old channels, it would certainly be interesting.

My main goal was to just share some useful things in Nix that I don’t think many people outside of the core Nix community know about. Documentation has gotten better recently but lots of times people like to just read blog posts like this. Hopefully you got a feel for what can be done in Nix.

Footnotes:

1

The difference between NixOS & Nixpkgs can sometimes cause confusion especially because they are hosted in the same repository. We usually refer to NixOS for the Linux-specific distro while Nixpkgs refers to the cross-platform set of packages. Here I am referring to them collectively.

2

Note that the channel changing script is not necessary. You can always refer to the Nixpkgs version directly with the -f argument. The script is just an easy way to introduce people to the concept.

3

The in-development version of NixOS/Nixpkgs will be a J for Jackrabbit.

4

Apple’s C standard library is called libSystem. Note that unlike Glibc & Musl it contains much, much more than what is needed to compile simple C programs.

5

Note that the same libSystem is used for all of Nixpkgs to peliminate having to do SDK detection. Eventually we will update this to 10.12 or 10.13 but we prefer to stay a couple releases behind.

6

This is not a complete explanation, but the best I can do for those not aware of the internals of Nixpkgs.

7

See GitHub issue #17372.

8

See release 0.5.

May 13, 2018 12:00 AM

March 31, 2018

Matthew Bauer

2018 Summer

Quick post to annouce summer plans. I have accepted a summer internship position at Obsidian Systems in the greatest city in the world. Very excited to be working with a team of really smart people.

March 31, 2018 12:00 AM

March 29, 2018

Matthew Bauer

66% of Nixpkgs are up-to-date

Just wanted to congratulate everyone for all of the work in keeping Nixpkgs up-to-date!

Repology has just updated our stats and we have shown significant improvement. 66% of nixpkgs-unstable packages are up-to-date, with a total of 6112 up-to-date packages1. Nixpkgs is slowly growing to match package sets that come in many of the big Linux distros. We now have more up-to-date packages than rolling release distros Arch (5924) and Manjaro (5708)! We certainly have a ways to go to compete with heavyweights like FreeBSD Ports (15410), DragonFly BSD Ports (14268), and Debian (14915) but these are ancient projects that have had a huge headstart. We are definitely making progress and it’s visible in the Repology graphs2.

For comparison, in February, nixpkgs-unstable had 60% of packages up-to-date, with a total of 5439 up-to-date packages3. This kind of progress over 1 month is tremendously helpful. Remember that most of these updates will be included in 18.09, not the soon to be release 18.03. This protects users from potentially breaking changes as well as giving maintainers plenty of time to verify that the updates have not had any negative effects.

Ryan Mulligan’s nix-update has made updating packages much easier and I think it will be a game changer for Nixpkgs in the long run. There are still some kinks in it but I think it is slowly getting better. It has certainly been a headache to handle the huge increase in pull requests, but we are slowly clearing through that glut of outdated software. If an update to nixpkgs-unstable breaks something for you, please be sure to open an issue.

March 29, 2018 12:00 AM

March 19, 2018

Munich NixOS Meetup

NixOS für Einsteiger

photoMunich NixOS Meetup

- Profpatsch: Nix(OS): Package-management done right (German)

Augsburg - Germany

Monday, March 19 at 7:00 PM

2

https://www.meetup.com/Munich-NixOS-Meetup/events/248479678/

March 19, 2018 11:37 AM

March 18, 2018

Joachim Schiele

nix-language-atlas-emscripten

motivation

in the nix-language-atlas series on lastlog.de/blog i want to discuss how well programming languages, for which i’m familiar with, integrate with nix. today, let’s revisit emscripten, as there also had been improvements since i wrote about it last time.

projects we have done:

what’s new

  • emscripten toolchain:
    • refactored to force a common revision (for example 1.37.16) on emscripten, emscripten-fastcomp and emscripten-fastcomp-clang called emscriptenVersion
    • added a unit test in emscripten to verify a small part of the toolchain
    • initial emscripten documentation in nixpkgs (not on https://nixos.org/nixpkgs/manual yet)
  • added 2 more unit tests & repaired all builds

for details see the PR https://github.com/NixOS/nixpkgs/pull/37291.

want to give this a shot from nixos?

git clone https://github.com/nixos/nixpkgs/
cd nixpkgs
git checkout f41a3e7d7d327ea66459d17bfbe4a751b2496cb1
nix-env -f default.nix -I nixpkgs=. -iA emscriptenPackages
installing ‘emscripten-json-c-0.13’
installing ‘emscripten-libxml2-2.9.7’
installing ‘emscripten-xmlmirror’
installing ‘emscripten-zlib-1.2.11’
...

dynamic/static libraries

in the above toolchain we are using libraries in .so format, not the .a format and in the end we link them together using emcc. this has some advantages:

  • building .so files is best practice on linux
  • thus easy to do
  • license wise smart, some packages can’t be statically linked legally IIRC

coverage

nix emscripten toolchain is now supported from:

especially the microsoft WSL release with the creators update is a very interesting audience as it makes it so easy to use nix on windows. no mingw, no cygwin! took me 10 minutes to install nix on windows!

for testing i’ve been compiling this very toolchain on my windows 10 computer. IO is slow but it works and it is easy to deploy.

emsdk

i’ve sumbled on this questions in https://github.com/juj/emsdk:

How do I change the currently active SDK version?

How do I build multiple projects with different SDK versions in parallel?

How do I use Emscripten SDK with a custom version of python, java, node.js or some other tool?

this are very interesting questions and all get easy once one uses this nixpkgs based toolchain as pointed out. i’ve been using the emsdk in the past but now that we have the ‘bits’ automated in nixpkgs i’m happy to not have to work statefully anymore!

ideas

during my last two days of work on the toolchain update, i had these ideas and motivations for the future:

  • more documentation:
    • nix-shell -A usage
    • cross platform nix usage
    • section how to (fast) write good and lasting unit tests
    • how to use older nixpkgs vs. newer ones (slightly modify a very old project for instance which has a bug)
  • packaging some common targts, what would these be?
  • package the caching of emscripten in ~/.emscripten properly, so that build artefacts can be reused over builds (time save) and remove the HOME=$TMPDIR requirement (ugly)

      DEBUG:root:adding object /home/joachim/.emscripten_cache/asmjs/dlmalloc.bc to link
      DEBUG:root:adding object /home/joachim/.emscripten_cache/asmjs/libc.bc to link
  • update nix-instantiate, which we use in the ‘tour of nix’, to a more recent version and also package it into the toolchain as an example
  • hydra builds
  • be more transparent on the license, ideally we could generate a list of licenses in the final folder

@nixos community: i’d love to get some feedback on this, so send me an email to js@lastlog.de if you have some interesting input.

summary

thanks to kripken (emscripten author) for his help! i’d love to put more effort into this, i think it’s really worth it so if you have any funding for general development or want to have something special realized, let me know!

another interesting project i’ve learned about lately would be https://github.com/NixOS/nixpkgs/pull/37291 from https://github.com/Ericson2314.

by qknight at March 18, 2018 05:35 PM

February 25, 2018

Sander van der Burg

A more realistic public Disnix example

It has almost been ten years ago when I started developing Disnix -- February 2008 marked the start of my master's thesis internship at Philips Research that resulted in the first prototype version.

Originally, Disnix was specifically developed for one use case only -- a medical service-oriented system called the "Service Development Support System" (SDS2) that can be used for asset tracking and utilisation analysis for medical devices in a hospital environment. More information about this case study can be found in my master's thesis, some of my research papers and my PhD thesis (all of them can be found on my publications page).

Many developments have happened since the realization of the first prototype -- its feature set has been extended considerably, its architecture has been overhauled several times and the code has evolved significantly. Most notably, I have been maintaining a production system for over three years with it.

In all these years, there is always one recurring question that I regularly receive from various kinds of people:

Why should I use Disnix and why would it be useful?

The answer is that Disnix becomes useful when you have a system that can be decomposed into distributable services, such as web services, RESTful services, web applications or processes.

In addition to the fact that Disnix automates its deployment and offers a number of powerful quality properties (e.g. non-destructive upgrades for the static parts of a system), it also helps componentized systems in reaching their full potential -- for example, when services can be built, deployed, and managed individually you can scale a system up and down (e.g. by distributing services to dedicated machines or consolidating all services on a single machine) and you can anticipate more flexibly to events (e.g. by redeploying services when we encounter a crashing machine).

Although the answer may sound simple, service-oriented systems are complicated -- besides facing all kinds of deployment complexities, properly dividing a system into distributable components is also quite challenging. For all the systems I have seen in the last decade, the requirements and their modularization strategies were all quite different from each other. I have also seen a number of systems for which decomposing into services did not work and unnecessary complexities were introduced.

Moreover, it is hard to find representative public examples that people can use as a reference. I was fortunate that I had access to an industrial case study during my research. Nonetheless, I was suffering from many difficulties because of the lack of any meaningful public case studies. As a countermeasure, I developed a collection of example cases in addition to SDS2, but because of their over-simplicity, proving my point often remained hard.

Roughly half a year ago, I have released most parts of my ancient web framework that I used to actively develop before I started doing research in software deployment and I created a couple of example applications for it.


Although my web framework development predates my deployment research, I was already using it to implement information systems that followed some modularity principles that are beneficial when using Disnix as a deployment system.

Recently, I have extended my web framework's example applications repository (providing a homework assistant, CMS, photo gallery and literature survey assistant) to become another public Disnix example case following the same modularity principles I used for the information systems I used to implement at that time.

Creating a componentized web information system


As mentioned earlier in this blog post, I have already implemented a (fairly simple) componentized web information system before I started working on Disnix using my ancient custom made web framework. The "componentization process" (a term that I had neither learned about yet nor something I was consciously implementing at that time) was partially driven by evolution and partially by non-functional requirements.

Originally, the system started out as just one single web application for one specific purpose and consisted of only two components -- a MySQL database responsible for storing the data and web front-end implemented in PHP, which is quite a common separation pattern for PHP applications.

Later, I was asked to implement another PHP application with similar functionality. Initially, I wrote the application from scratch without any reuse in mind, but at some point I made two important decisions:

  • I decided to keep the databases of each applications separate as opposed to integrating all the tables into one single database. My main motivating factor was that I wanted to prevent another developer's wrong decisions from messing up the other application. Moreover, I realized that for the data that was specific to the application domain that other systems did not have to know about it.
  • In addition to domain specific data, I noticed that both databases also stored the same kind of data, namely: user accounts -- both systems had a user account system to allow users to change the data. This also did not motivate me to integrate both databases into one database. Instead, I created a separate user database and authentication system (as a library API) that was shared among both applications.

After completing the two web applications, I had to implement more functionality. I decided to keep all of these new features for these new problem domains in separate applications with separate databases. The only thing they had in common was a shared user authentication system.

At some point I ended up having many sub applications. As a result, I needed a portal application that redirected users to these sub applications. Essentially, what I implemented became a system of systems.

Deployment with Disnix


The "architectural decisions" that I described earlier resulted in a system composed of several kinds of components:

  • Domain-specific web applications exposing functionality that logically belongs together.
  • Domain-specific databases storing tables that are strongly correlated.
  • A shared user database.
  • A portal application redirecting users to the domain-specific web applications.

The above listed components can be distributed over multiple machines in a network, because they connect to each other through network links (e.g. connecting to a MySQL database can be done with a TCP connection and connecting to a domain specific web application can be done through HTTP). As a result, they can also be modeled as services that can be deployed with Disnix.

To replicate the same patterns for demo purposes, I integrated my framework's example applications into a similar system of sub systems. We can deploy the corresponding example system to one single target machine with Disnix, by running:


$ disnixos-env -s services.nix \
-n network-single.nix \
-d distribution-single.nix --use-nixops

The entire system gets deployed to a single machine because of the distribution model (distribution.nix) that maps all services to one target machine:


{infrastructure}:

{
usersdb = [ infrastructure.test1 ];
cmsdb = [ infrastructure.test1 ];
cmsgallerydb = [ infrastructure.test1 ];
homeworkdb = [ infrastructure.test1 ];
literaturedb = [ infrastructure.test1 ];
portaldb = [ infrastructure.test1 ];

cms = [ infrastructure.test1 ];
cmsgallery = [ infrastructure.test1 ];
homework = [ infrastructure.test1 ];
literature = [ infrastructure.test1 ];
users = [ infrastructure.test1 ];
portal = [ infrastructure.test1 ];
}

The resulting deployment architecture looks as follows:


The above visualization of the deployment architecture shows the following aspects:

  • The surrounding light grey colored box denotes a target machine. In this particular example, we only have one single target machine where services are deployed to.
  • The dark grey colored boxes correspond to container environments. For our example system, we have two of them: mysql-database corresponding to a MySQL DBMS server and apache-webapplication corresponding to an Apache HTTP server.
  • The ovals denote services corresponding to MySQL databases and web applications.
  • The arrows denote inter-dependency links that correspond to network connections. As explained in my previous blog post, solid arrows are dependencies with a strict ordering requirement while dashed arrows are dependencies without an ordering requirement.

Some people may argue that it is not really beneficial to deploy such a system with Disnix -- with NixOps you can define a machine configuration having a MySQL DBMS server and an Apache HTTP server with the corresponding databases and web application components. With Disnix, you must first ensure that the machines, the MySQL and Apache HTTP servers are configured by other means first (that could for example be done with NixOps), and then you have to deploy the system's components with Disnix.

In a single machine deployment scenario, it may indeed not be that beneficial. However, what you get in addition to automated deployment is also more flexibility. Since Disnix manages the services directly, as opposed to entire machine configurations as a whole, you can anticipate better in case of events by redeploying the system.

For example, when the amount of visitors keeps growing, you may run into the problem that a single server can no longer handle all the traffic. In such cases, you can easily add another machine to the network and adjust the distribution model to move (for example) the databases to another machine:


{infrastructure}:

{
usersdb = [ infrastructure.test2 ];
cmsdb = [ infrastructure.test2 ];
cmsgallerydb = [ infrastructure.test2 ];
homeworkdb = [ infrastructure.test2 ];
literaturedb = [ infrastructure.test2 ];
portaldb = [ infrastructure.test2 ];

cms = [ infrastructure.test1 ];
cmsgallery = [ infrastructure.test1 ];
homework = [ infrastructure.test1 ];
literature = [ infrastructure.test1 ];
users = [ infrastructure.test1 ];
portal = [ infrastructure.test1 ];
}

By redeploying the system, we can take advantage of the additional system resources that the new machine provides:


$ disnixos-env -s services.nix \
-n network-separate.nix \
-d distribution-separate.nix --use-nixops

resulting in the following deployment architecture:


Likewise, there are countless of other deployment strategies possible to meet all kinds of non-functional requirements. For example, we can also distribute bundles of domain specific application and database pairs over two machines:


$ disnixos-env -s services.nix \
-n network-bundles.nix \
-d distribution-bundles.nix --use-nixops

resulting in the following deployment architecture:


This approach is even more scalable than simply offloading the databases to another server.

In addition to scalability, there are countless of other reasons to pick a certain distribution strategy. You could also, for example, distribute redundant instances of databases and applications as a failover to improve availability or improve security by deploying the databases with privacy sensitive data to a machine with restrictive network access.

State management


When updating the deployment of systems with Disnix (such as moving a database from one machine to another), there may be a recurring limitation that you could run frequently into -- like Nix, Disnix only manages the static parts of the system, but not any state. This means that a service's deployment can be reproduced elsewhere, but data, such as the content of a database is not migrated.

For example, the sub system of example applications stores two kinds of data -- records in the MySQL database and files, such as images uploaded in the photo gallery or PDF files uploaded to the literature application. When moving these applications around the data is not migrated.

As a possible solution, Disnix also provides simple state management facilities. When enabled, Disnix will take snapshots of the databases and filesets on the source machines, transfers the snapshots to the target machines, and finally restores the snapshots when moving a service one machine to another in the distribution model.

State management can be enabled globally by passing the --deploy-state parameter to (disnix-env or annotating the services with deployState = true; in the services model):


$ disnixos-env -s services.nix \
-n network-bundles.nix \
-d distribution-bundles.nix --use-nixops --deploy-state

We can also directly use the state management system, e.g. for backup purposes. When running the following command:


$ disnix-snapshot

Disnix takes snapshots of all databases and web application state (e.g. the images in the photo gallery and uploaded PDF files) and transfers them to the coordinator machine. With the dysnomia-snapshots tool we can inspect the snapshot store:


$ dysnomia-snapshots --query-all
apache-webapplication/cms/1f9ed847885d2b3e3c67c51231122d958751eb5e2443c281e02e1d7108a505a3
apache-webapplication/cmsgallery/28d17a6941cb195a92e748aae737ccf524747477c6943436b734891d0f36fd53
apache-webapplication/literature/ed5ec4f8b9b4fcdb8b740ad1fa7ecb40b10dece03548f1d6e09a6a82c804131b
apache-webapplication/portal/5bbea499f8f8a4f708bb873ad683dbf088afa4c553f90ab287a9249a7ef02651
mysql-database/cmsdb/aa75992f780991c39a0969dcac5f69b04685c4fa764937476b816e938d6972ba
mysql-database/cmsgallerydb/31ebdaba658ca376123ff6a91a3e275731b383346a07840b1acaa1e44d921b65
mysql-database/homeworkdb/f0fda91545af0cb300afd84592d4914dcd48257053401e232438e34d83af828d
mysql-database/literaturedb/cb881c2200a5f1562f0b66f1394d0902bbb8e2361068fe096faac3bc31f76b5d
mysql-database/portaldb/5d8a5cb952f40ce76f93eb939d0b37eab33736d7b1e1426038322f8a572034ee
mysql-database/usersdb/64d11fc7f8969da5da318276a666f2e00e0a020ba619a1d82ed9b84a7f1c2ca6

and with some shell scripting, the actual contents of the snapshot store:


$ find $(dysnomia-snapshots --resolve $(dysnomia-snapshots --query-all)) -type f
/home/sander/state/snapshots/apache-webapplication/cms/1f9ed847885d2b3e3c67c51231122d958751eb5e2443c281e02e1d7108a505a3/state.tar.xz
/home/sander/state/snapshots/apache-webapplication/cmsgallery/28d17a6941cb195a92e748aae737ccf524747477c6943436b734891d0f36fd53/state.tar.xz
/home/sander/state/snapshots/apache-webapplication/literature/ed5ec4f8b9b4fcdb8b740ad1fa7ecb40b10dece03548f1d6e09a6a82c804131b/state.tar.xz
/home/sander/state/snapshots/apache-webapplication/portal/5bbea499f8f8a4f708bb873ad683dbf088afa4c553f90ab287a9249a7ef02651/state.tar.xz
/home/sander/state/snapshots/mysql-database/cmsdb/aa75992f780991c39a0969dcac5f69b04685c4fa764937476b816e938d6972ba/dump.sql.xz
/home/sander/state/snapshots/mysql-database/cmsgallerydb/31ebdaba658ca376123ff6a91a3e275731b383346a07840b1acaa1e44d921b65/dump.sql.xz
/home/sander/state/snapshots/mysql-database/homeworkdb/f0fda91545af0cb300afd84592d4914dcd48257053401e232438e34d83af828d/dump.sql.xz
/home/sander/state/snapshots/mysql-database/literaturedb/cb881c2200a5f1562f0b66f1394d0902bbb8e2361068fe096faac3bc31f76b5d/dump.sql.xz
/home/sander/state/snapshots/mysql-database/portaldb/5d8a5cb952f40ce76f93eb939d0b37eab33736d7b1e1426038322f8a572034ee/dump.sql.xz
/home/sander/state/snapshots/mysql-database/usersdb/64d11fc7f8969da5da318276a666f2e00e0a020ba619a1d82ed9b84a7f1c2ca6/dump.sql.xz

The above output shows that for each MySQL database, we store a compressed SQL dump of the database and for each stateful web application, a compressed tarball of state files.

Conclusion


In this blog post, I have described a more realistic public Disnix example that is inspired by my web framework developments a long time ago. Aside from automating a system's deployment, the purpose of this blog post is to describe how a system that can be decomposed into distributable services that can be deployed with Disnix. Implementing such a system is all but trivial and driven by various kinds of design decisions.

Availability


The example web application system can be obtained from my GitHub page. The Disnix deployment expressions can be found in the deployment/ sub folder.

In addition, I have created a Dysnomia module named: fileset that can capture the state files of web applications in a compressed tarball.

After the recent developments the Disnix toolset has reached a new stable point. As a result, I have decided to release Disnix 0.8. Consult the Disnix homepage for more information!

by Sander van der Burg (noreply@blogger.com) at February 25, 2018 09:59 PM

February 21, 2018

Joachim Schiele

nix-language-atlas-javascript

motivation

in the nix-language-atlas series on lastlog.de/blog i want to discuss how well programming languages, for which i’m familiar with, integrate with nix. let’s revisit javascript, as there had been major improvements since i wrote about it last time. we won’t look into the emscripten toolchain.

DSL-PM in general

are ‘domain specific language package manager(s)’ (DSL-PM) as npm or yarn a good thing?

integrate DSL-PM(s) into nix/nixpkgs

DSL-PM properties nix requires:

  • reproducible dependency calculations/downloads
  • reproducible configuration, build & installation into the store of each dependency
  • reproducible configuration, build & installation into the store of main target

DSL-PM evolution (javascript)

note: read this if you are interested in npm/yarn differences. it seems that yarn forced many of its concepts into npm.

yarn2nix workflow

let’s see a simple example how to integrate a yarn based project into your nix codebase:

  1. we clone an example:

    git clone https://github.com/yarnpkg/example-yarn-package
    cd example-yarn-package
    mkdir bin
    touch bin/myapp
    chmod u+x bin/myapp
  2. let’s add a default.nix

    { pkgs ? import <nixpkgs> {} }:
    let
      yarn2nixSrc = pkgs.fetchFromGitHub {
        owner = "moretea";
        repo = "yarn2nix";
        rev = "0472167f2fa329ee4673cedec79a659d23b02e06";
        sha256 = "10gmyrz07y4baq1a1rkip6h2k4fy9is6sjv487fndbc931lwmdaf";
      };
      yarn2nixRepo = pkgs.callPackage yarn2nixSrc {};
      inherit (yarn2nixRepo) mkYarnPackage;
    in
      mkYarnPackage {
        src = ./.;
      }

    and add contents to bin/myapp

    #!/usr/bin/env node
    
    'use strict';
    
    // For package depenency demonstration purposes only
    var multiply = require('lodash/multiply');
    
    console.log("h" + multiply(2,0.5) + ", it works!")

    add a “bin” to package.json, see this patch:

  3. now build the source

    nix-build -Q
    building path(s) ‘/nix/store/gkdb8hsykw3idp4xpv6by618wad5s052-offline’
    building path(s) ‘/nix/store/b6kiqlcaacxrlq4nnjgk69mwjnrlvkpv-yarn2nix-modules-0.1.0’
    building
    yarn config v1.3.2
    success Set "yarn-offline-mirror" to "/nix/store/gkdb8hsykw3idp4xpv6by618wad5s052-offline".
    Done in 0.10s.
    yarn install v1.3.2
    [1/4] Resolving packages...
    [2/4] Fetching packages...
    [3/4] Linking dependencies...
    [4/4] Building fresh packages...
    warning Ignored scripts due to flag.
    Done in 4.99s.
    ...
    <skipped many lines>>
    ...
    building path(s) ‘/nix/store/7yg1mah963w35dhxjcylw5q0r62bkk83-yarn.nix’
    these derivations will be built:
      /nix/store/07xy1c1yp4ada1sjiz7m416ic2i1b908-webidl-conversions-3.0.1.tgz.drv
      /nix/store/0c2xgds6pzsnsp1hmxdgaqh1n28d68k5-lodash.isarray-3.0.4.tgz.drv
      /nix/store/0dj6mwhpmzvnxzz2122isdbw85mjg0wv-regex-cache-0.4.3.tgz.drv
      /nix/store/0in2pcwsc3ln6glqkab5fsx98m28vq8g-lodash._baseassign-3.2.0.tgz.drv
      /nix/store/1rhagnq3gz68x5lbhca6z4rsx9jnabgh-is-dotfile-1.0.2.tgz.drv
      /nix/store/1vq9rlkn3qdy2acndj2q47ryswiv5yir-lodash.assign-4.2.0.tgz.drv
      /nix/store/2a4kqk4p54nvm31cyy2nnl2vzr4klyb8-babel-generator-6.14.0.tgz.drv
      /nix/store/2mf2dr0gx3bqxiqbpqdhhp8bhz95d34h-arr-diff-2.0.0.tgz.drv
      /nix/store/2pbiyjyqg57dnn0apj24lrvqp1cbix9f-expand-brackets-0.1.5.tgz.drv
      /nix/store/31h1f7rair2xq0di3b3agr7g2xfgncyz-jest-matcher-utils-15.1.0.tgz.drv
      /nix/store/3l6amqm7wdx1c45mypbgh8xgsl92zw76-exec-sh-0.2.0.tgz.drv
      /nix/store/3w6wxplb811vzfky5sgvbifm6j5p6fac-babel-core-6.14.0.tgz.drv
      /nix/store/48d8jwrpgg2pxm8mkyqvavrv4k5i2nab-lodash.keys-3.1.2.tgz.drv
      /nix/store/4l0s8plnxw2nzszbjjdicrr5np9cc2ni-jest-resolve-15.0.1.tgz.drv
      /nix/store/4mn98xigyzybcnsb71731pw8biirzzax-micromatch-2.3.11.tgz.drv
    ...
    <skipped many lines>>
    ...
    building path(s) ‘/nix/store/chsx89ifr7dk7mc00d6789indh0rn41x-to-fast-properties-1.0.2.tgz’
    building path(s) ‘/nix/store/sqpsd9y77kkq1vvfpfcfzk9q347dwg14-walker-1.0.7.tgz’
    building path(s) ‘/nix/store/j7fp6iz0wj6dcp8ibsqyph2vy4p007wk-watch-0.10.0.tgz’
    building path(s) ‘/nix/store/nv5p8ijp15mc2kbpp26mvd8pn7vqlwyy-webidl-conversions-3.0.1.tgz’
    building path(s) ‘/nix/store/a3amdrmhzrvd3v1lsy1ih5rfpdamyj46-whatwg-url-3.0.0.tgz’
    building path(s) ‘/nix/store/vhwym48w1pvgzli9i57fnzqw0pjp3a83-window-size-0.2.0.tgz’
    building path(s) ‘/nix/store/2dg59ikzfrdwbcpkwg2xwjxs9rhpilb8-worker-farm-1.3.1.tgz’
    building path(s) ‘/nix/store/497p7kg9iyb09ah0vvqx88dpq3bn843p-yargs-5.0.0.tgz’
    building path(s) ‘/nix/store/w3lmwwq4cac653nhajnvyl1k2vhivaws-yargs-parser-3.2.0.tgz’
    building path(s) ‘/nix/store/ywn6ysmnvrkyzjwpdiwr4iivk9z9kx2y-offline’
    building path(s) ‘/nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0’
    building path(s) ‘/nix/store/r9s6cmq5ik47c0s1hd48xz3r6pixmm1r-example-yarn-package-1.0.0’
    /nix/store/r9s6cmq5ik47c0s1hd48xz3r6pixmm1r-example-yarn-package-1.0.0

    note: the last output line of nix-build is what we are interested in!

  4. checking the result

    ls -lathr result/node_modules/
    
    ...
    <skipped many lines>>
    ...
    lrwxrwxrwx 1 root root  102  1. Jan 1970  align-text -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/align-text
    lrwxrwxrwx 1 root root  105  1. Jan 1970  acorn-globals -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/acorn-globals
    lrwxrwxrwx 1 root root   97  1. Jan 1970  acorn -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/acorn
    lrwxrwxrwx 1 root root   98  1. Jan 1970  abbrev -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/abbrev
    lrwxrwxrwx 1 root root   96  1. Jan 1970  abab -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/abab
  5. execute the binary

    /nix/store/r9s6cmq5ik47c0s1hd48xz3r6pixmm1r-example-yarn-package-1.0.0/bin/myapp
    h1, it works!

using nix-build we declaratively built the node package!

yarn2nix internals

yarn (imperative)

  1. yarn -> read yarn.lock
  2. make / build all the downloads
  3. come up with node_packages folder
  4. yarn install

    note: in general we skip step (4.)

yarn (declarative)

  1. yarn2nix -> read yarn.lock
  2. translate each dependency into a mkDerivation

    each is a mkDerivation residing in /nix/store/…

  3. evaluate each mkDerivation, gain store path
  4. call yarn with the list of all mkDerivation(s), yarn comes up with node_modules

    a single mkDerivation also residing in /nix/store/… but encapsulating other /nix/store entries

  5. final mkDerivation uses this node_modules and creates the store path we are interested in

conclusion

  • take notice, and i can’t stress this enough, we map yarn.lock entries into single pkgs.stdenv.mkDerivation (by calling mkYarnPackage) but later pass all these into yarn which creates the node_modules contents from this. this is exactly how a DSL-PM must be designed to be easily integrated
  • yarn.lock obsoletes the requirement of manually creating a dependency file per project:
    • shown in the example above where i only added a default.nix
    • i’d wish dep’s Gopkg.lock used with go would also contain a hash already but they are using GIT and we have to call nix-prefetch-git to generate a sha256 hash manually
  • yarn2nix might not be as advanced as node2nix but we at nixcloud.io use it for all projects. yarn/yarn2nix is really fast, in dependency calculation and deployment, compared what we had before
  • i’d wish that npm’s sha1 would be replaced by something more recent but in comparison to dep they at least have a hash of some sort
  • yarn2nix is not a part of nixpkgs yet, see https://github.com/moretea/yarn2nix/issues/5

by qknight at February 21, 2018 12:35 PM

February 11, 2018

Sander van der Burg

Deploying systems with circular dependencies using Disnix


Some time ago, during my PhD thesis defence, one of my committee members asked me how I would deploy systems with Disnix in which services have circular dependencies.

It was an interesting question because Disnix defines dependencies between services (that typically involve network connections) as inter-dependencies that have two properties:

  • They allow services to find services they depend on by providing their connection properties
  • They ensure that any inter-dependency is activated before the service itself, so that no failures will occur because of missing dependencies -- in Disnix, a service is either available or unavailable, but never in a broken state due to missing inter-dependencies at runtime.

In a system with circular dependencies, the ordering property is problematic -- it is impossible to activate one dependency before another without having broken connections between them.

During the defence, I had to admit that I have never deployed such systems with Disnix before, but that there were a couple of possible solutions to cope with such constraints. For example, you can propagate properties of the distribution model directly to a service, as opposed to declaring circular inter-dependencies. Then the ordering requirement is not enforced.

I also explained that systems should not have any hard cyclic requirements on other services, but instead compose their (potential bidirectional) communication channels at runtime. Furthermore, I explained that circular dependencies are bad from a reuse perspective -- when two services mutually depend on each other, then they should ideally be one service.

Although the answer sufficed (e.g. it provided the answer that it was possible), the solution basically relies on unconventional usage of the deployment tool. Recently, as a personal exercise, I have decided to dig up this question again and explore the possibilities of deploying systems with circular dependencies.

Chord: a peer-to-peer distributed hash table


When thinking of an example system that has a circular dependency structure, the first thing that came up in my mind is Chord: a peer-to-peer distributed hash table (a copy of the research paper written by Stoica et al can be found here). Interesting fact is that I had to implement it many years ago in the lab course of the distributed algorithms course taught by another member of my PhD thesis committee.

A Chord network has circular runtime dependencies because it has a a ring structure -- in a network that has more than one node, each node has a successor and predecessor link, in which no node has the same predecessor or successor and the last successor link refers to the first node:


The Chord nodes (shown in the figure above) constitute a distributed peer-to-peer hash table. In addition to the fact that it can store key and value pairs (all kinds of objects), it also distributes the data over the nodes in the network.

Moreover, its operations are decentralized -- for example, when it is desired to search for an object or to store new objects in the hash table, it is possible to consult any node in the network. The system will redirect the caller to the appropriate node that should host the data.

Various kinds of implementations exist of the Chord protocol. The official reference implementation is a filesystem abstraction layer built on top of it. I experimented with the Java-based OpenChord implementation that is capable of storing arbitrary serializable Java objects.

More details about the implementation details of Chord operations can be found in the research paper.

Deploying a Chord network


One of the challenges I faced during the lab course is that I had deploy a test Chord network with a small collection of nodes. At that time, I had no proper deployment automation. I ended up writing a bash shell script that spawned a collection of processes in parallel.

Because deployment was complicated, I never tried more complex scenarios than running a small collection of processes on a single machine. Because it was not required for the lab course to do more than just that I, for example, never tried any real network communication deployments in which I had to distribute Chord nodes over multiple computer systems. The latter would have introduced even more complexity to the deployment process.

Deploying a Chord network basically works as follows:

  • First, we must deploy an initial node that has no connection to a predecessor or successor node.
  • Then for each additional node, we call the join operation to attach it to the network. As explained earlier, a Chord hash-table is decentralized and as a result, we can consult any node we want in the network for the join process. The join and stabilization procedures decide which predecessor and successor a new node actually gets.

There are various strategies to join additional nodes to the network, but I what I ended up doing is using the initial node as a bootstrap node -- all successive nodes, simply join to the bootstrap node and the network stabilizes to become a ring.

(As a sidenote: you could argue whether this is a good process, since the introduction of a central bootstrap node during the deployment process violates the peer-to-peer contraint, but that is a different story. Obviously, you could also think of other bootstrap strategies but that is beyond the scope of this blog post).

Automating a Chord network deployment with Disnix


To experiment with a Chord network, I have decided to create a simple server process (using the OpenChord API) whose only responsibility is to store data. It can optionally join another node in the network and it has a command-line interface allowing me to conveniently specify the connection parameters.

The deployment strategy using the initial node as a bootstrap node can be easily automated with Disnix. In the Disnix services model, we can define the bootstrap node as follows:


ChordBootstrapNode = rec {
name = "ChordBootstrapNode";
pkg = customPkgs.ChordBootstrapNode { inherit port; };
port = 8001;
portAssign = "private";
type = "process";
};

The above service configuration corresponds to a process that binds the service to a provided TCP port.

Each successive node can be defined as a service that has an inter-dependency on the bootstrap node:


ChordNode1 = rec {
name = "ChordNode1";
pkg = customPkgs.ChordNode { inherit port; };
port = 8002;
portAssign = "private";
type = "process";
dependsOn = {
inherit ChordBootstrapNode;
};
};

As can be seen in the above Nix expression, the dependsOn attribute specifies that the node has an inter-dependency on the bootstrap node. The inter-dependency declaration provides the connection settings of the bootstrap node to the command-line utility that spawns the service and ensures that the bootstrap node is deployed first.

By providing an infrastructure model containing a number of machines and writing a distribution model that maps the node to the machine, such as:


{infrastructure}:

{
ChordBootstrapNode = [ infrastructure.test1 ];
ChordNode1 = [ infrastructure.test1 ];
ChordNode2 = [ infrastructure.test2 ];
ChordNode3 = [ infrastructure.test2 ];
}

we can deploy a Chord network consisting of 4 nodes distributed over two machines by running:


$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix

This is the resulting deployment architecture of the Chord network that gets deployed:
>

In the above picture, the light grey colored boxes denote machines, the dark grey colored boxes container environments, the ovals services and the arrows inter-dependency relationships.

By running the OpenChord console, we can join any of our nodes in the network, such as the third node deployed to machine test2:


$ /nix/var/nix/profiles/disnix/default/bin/openchord-console
> joinN -port 9000 -bootstrap test2:8001
Trying to join chord network with boostrap URL ocsocket://test2:8001/
URL of created chord node ocsocket://192.168.56.102:9000/.

we can check the references that the console node has:


> refsN
Node: C1 F0 42 95 , ocsocket://192.168.56.102:9000/
Finger table:
59 E4 86 AC , ocsocket://test2:8001/ (0-159)
Successor List:
59 E4 86 AC , ocsocket://test2:8001/
64 F1 96 B9 , ocsocket://test1:8001/
Predecessor: 9C 51 42 1F , ocsocket://test2:8002/

As may be observed in the output above, our predecessor is the node 3 deployed to machine test2 and our successors are node 3 deployed to machine test2 and node 1 deployed to machine test1.

We can also insert and retrieve the data we want:


> insertN -key test -value test
> entriesN
Entries:
key = A9 4A 8F E5 , value = [( key = A9 4A 8F E5 , value = test)]

Defining services with circular dependencies in Disnix


As shown in the previous section, the ring structure of a Chord hash table is constructed at runtime. As a result, Disnix does not need to manage any circular dependencies. Instead, it only has to know the dependencies of the bootstrap phase which are not cyclic at all.

I was also curious whether I could modify Disnix to properly define circular-dependencies, without any workarounds such as directly propagating properties from the distribution model. As explained in the introduction, inter-dependencies have two properties in which the second property is problematic: the ordering constraint.

To cope with the problematic ordering property, I have introduced a new property in the services model called: connectsTo allowing users to specify inter-dependencies for which the ordering does not matter. The connectsTo property makes it possible for services to define mutual dependencies on each other.

As an example case, I have extended the Disnix composition examples (a set of trivial examples implementing "Hello world" testcases) with a cyclic example case. In this new sub example, I have created a web application that both contains a server returning the "Hello world!" string and a client displaying the string. The result would be the following screen:


(Does it look cool? :p)

A web application instance is capable of connecting to another web service to obtain the "Hello world!" message to display. We can compose two web application instances that refer to each other to accomplish this.

The corresponding services model looks as follows:


{distribution, invDistribution, system, pkgs}:

let customPkgs = import ../top-level/all-packages.nix {
inherit system pkgs;
};
in
rec {
HelloWorldCycle1 = {
name = "HelloWorldCycle1";
pkg = customPkgs.HelloWorldCycle;
connectsTo = {
# Depends on the other cyclic service
HelloWorldCycle = HelloWorldCycle2;
};
type = "tomcat-webapplication";
};

HelloWorldCycle2 = {
name = "HelloWorldCycle2";
pkg = customPkgs.HelloWorldCycle;
connectsTo = {
# Depends on the other cyclic service
HelloWorldCycle = HelloWorldCycle1;
};
type = "tomcat-webapplication";
};
}

As may be observed in the above code fragment, the first service has a dependency on the second, while the second also has a dependency on the first. They are allowed to refer to each other because the connectsTo property disregards ordering.

By mapping the services to a network of machines that have Apache Tomcat hosted:


{infrastructure}:

{
HelloWorldCycle1 = [ infrastructure.test1 ];
HelloWorldCycle2 = [ infrastructure.test2 ];
}

and deploying the system:


$ disnix-env -s services-cyclic.nix \
-i infrastructure.nix \
-d distribution-cyclic.nix

We end-up with a deployment architecture of two services having cyclic dependencies:


To produce the above visualization, I have extended the disnix-visualize tool with support for the connectsTo property that displays inter-dependencies as dashed arrows (as opposed to solid arrows that denote ordinary inter-dependencies).

In addition to the option to specify circular dependencies, the connectsTo property has another interesting use case -- when services have inter-dependencies that may be broken, we can optimize the duration of an upgrade processes.

Normally, when a service gets upgraded, all its inter-dependent services will be reactivated. This is an implication of Disnix's strictness -- a service is either available or unavailable, but never broken because of missing inter-dependencies.

However, all the extra reactivations in the upgrade phase can be quite expensive as a result. If a link is non-critical and it is permitted to be down for a short while, then redeployments can be made faster.

Conclusion


In this blog post, I have described two deployment experiments with Disnix involving systems that have circular dependencies -- a Chord-based distributed hash table (that constructs a ring structure at runtime) and a trivial toy example system in which two services have mutual dependencies on each other.

Availability


The newly introduced connectsTo property is part of the development version of Disnix and will become available in the next release.

The composition example and newly created Chord example can be found on my GitHub page.

by Sander van der Burg (noreply@blogger.com) at February 11, 2018 11:31 PM

Matthew Bauer

nix-buffer: nix-shell in Emacs

Shea Levy has a wonderful package out for users of Nix and Emacs. It’s called nix-buffer and it greatly improves working on Nix stuff in Emacs. To try it just get it from MELPA by running M-x package-install<RET>nix-buffer.

To use it in a project you need to create a new file in your project directory called dir-locals.nix. It works a lot like .dir-locals.el but it allows you to evaluate Nix expressions.

To setup, just create dir-locals.nix that looks like this:

let pkgs = import <nixpkgs> {};
in with pkgs; nixBufferBuilders.withPackages [ … ]

You can put anything from Nixpkgs inside of withPackages. That path is added to your exec-path in Emacs. Using this you can wrap your project in a container and avoid conflicts between project configurations!

There’s lot of future extensions for this. Ideally we could skip and the dir-locals.nix configuration and automatically detect what dependencies you need based on your default.nix file.

February 11, 2018 12:00 AM