{stdenv, foo, bar}:
{name, buildInputs ? [], ...}@args:

let
  extraArgs = removeAttrs args [ "name" "buildInputs" ];
in
stdenv.someBuildFunction ({
  name = "mypackage-"+name;
  buildInputs = [ foo bar ] ++ buildInputs;
} // extraArgs)

• The outer function header (first line) specifies all common build-time dependencies required to build a project. For example, if we want to build a function abstraction for Python projects, then python is such a common build-time dependency.
• The inner function header specifies all relevant build parameters and accepts an arbitrary number of arguments. Some arguments have a specific purpose for the kind of software project that we want to build (e.g. name and buildInputs) while other arguments can be passed verbatim to the build function abstraction that we use as a basis.
• In the body, we invoke a function abstraction (quite frequently stdenv.mkDerivation {}) that builds the project. We use the build parameters that have a specific meaning to configure specialized build properties and we pass all remaining build parameters that are not conflicting verbatim to the build function that we use a basis.
  
  A subset of these arguments have no specific meaning and are simply exposed as environment variables in the builder environment.
  
  Because some parameters are already being used for a specific purpose and others may be incompatible with the build function that we invoke in the body, we compose a variable named: extraArgs in which we remove the conflicting arguments.

• A build procedure is extendable/tweakable -- we can adjust the build procedure by adding or changing the build phases, and tweak them by providing build hooks (that execute arbitrary command-line instructions before or after the execution of a phase). This is particularly useful to build additional abstractions around it for more specialized deployment procedures.
• Because an arbitrary number of arguments can be propagated (that can be exposed as environment variables in the build environment), we have more configuration flexibility.

Deploying SDKs

with import <nixpkgs> {};

let
  androidComposition = androidenv.composeAndroidPackages {
    toolsVersion = "25.2.5";
    platformToolsVersion = "27.0.1";
    buildToolsVersions = [ "27.0.3" ];
    includeEmulator = true;
    emulatorVersion = "27.2.0";
  };
in
androidComposition.androidsdk

$ nix-build
/nix/store/zvailnl4f1261cn87s9n29lhj9i7y7iy-androidsdk

Writing build function abstractions for SDKs

{composeMySDK, stdenv}:
{foo, bar, ...}@args:

let
  mySDKFormalArgs = builtins.functionArgs composeMySDK;
  mySDKArgs = builtins.intersectAttrs mySDKFormalArgs args;
  mySDK = composeMySDK mySDKArgs;
  extraArgs = removeAttrs args ([ "foo" "bar" ]
    ++ builtins.attrNames mySDKFormalArgs);
in
stdenv.mkDerivation ({
  buildInputs = [ mySDK ];
  buildPhase = ''
    ${mySDK}/bin/build
  '';
} // extraArgs)

• First, we dynamically extract the formal arguments of the function that composes the SDK (mySDKFormalArgs).
• Then, we compute the intersection of the formal arguments of the composition function and the actual arguments from the build function arguments set (args). The resulting attribute set (mySDKArgs) are the actual arguments we need to propagate to the SDK composition function.
• The next step is to deploy the SDK with all its plugins by propagating the SDK arguments set as function parameters to the SDK composition function (mySDK).
• Finally, we remove the arguments that we have passed to the SDK composition function from the extra arguments set (extraArgs), because these parameters have no specific meaning for the build procedure.

{androidenv}:

androidenv.buildApp {
  # Build parameters
  name = "MyFirstApp";
  src = ../../src/myfirstapp
  antFlags = "-Dtarget=android-16";

  # SDK composition parameters
  platformVersions = [ 16 ];
  toolsVersion = "25.2.5";
  platformToolsVersion = "27.0.1";
  buildToolsVersions = [ "27.0.3" ];
}

• The above function invocation propagates three build parameters: name referring to the name of the Nix package, src referring to a filesystem location that contains the source code of an Android project, and antFlags that contains command-line arguments that are passed to the Apache Ant build tool.
• It propagates four SDK composition parameters: platformVersions referring to the platform SDKs that must be installed, toolsVersion to the version of the tools package, platformToolsVersion to the platform-tools package and buildToolsVersion to the build-tool packages.

Availability

Munich NixOS Meetup

There are no talks or specific topics planned for now.

If you are interested in giving a talk, feel free to ask us, otherwise we will have a casual discussion session, help each other on issues and hack away at bugs we find or features we'd like!

Anyone is welcome, no matter if you haven't started using Nix(OS), yet or are using it in production!

Drinks and food will be provided.

München - Germany

Thursday, September 20 at 7:00 PM

17

https://www.meetup.com/Munich-NixOS-Meetup/events/254554928/

Cachix - Nix binary cache as a service was down:

• On Aug 22nd from 16:55 until 18:55 UTC (120 minutes)
• On Aug 23rd from 20:01 until 20:09 UTC (8 minutes)

On the 22nd there was no action from my side; the service recovered itself. I did have monitoring configured and I received email alerts, but I have not noticed them.

I have spent most of the 23rd gathering data and evidence on what went wrong. Just before monitoring stopped receiving data at 16:58 UTC, white-box system monitoring revealed:

• Outgoing bandwidth skyrocketed to 23MB/s
• Resident memory went through the roof to ~90%

On 23rd I have immediately seen the service was down and I've rebooted the machine.

I have spent a significant amount of time trying to determine if a specific request caused this, but it seems likely that it was just an overload, although I have not proved this theory.

Countermeasures taken

a) Server-side is implemented in GHC Haskell, so I have enabled -O2. Although GHC wiki on Performance says it is indistinguishable from -O1, in the last week I've seen an approximately 10% reduction of resident memory and most importantly, fewer memory spikes. Again, no hard evidence, time will tell.

b) Most importantly, production now runs with GHCRTS='-M2G' flag, limiting overall heap to 2G of memory, so we are not depending on the Linux OOM killer to handle out of memory situations. It is not entirely clear to me why the machine was unresponsive for two hours since OOM should have kicked in, but during that period there was not a single monitoring datapoint sent.

c) I have configured EKG to send GC stats to datadog so if it happens again, that should provide better insight into what is going on with memory consumption.

Countermeasures to be taken

1) Use a service like Pagerduty to be alerted immediately on the phone

2) Upgrade Datadog agent to version 6, which allows more precise per process monitoring

So far I am quite happy how Haskell works in production. I have taken Well-Typed training on GHC performance and if this turns out to be a space leak, I am confident that I will find it.

The only thing that saddens me, coming from Python, is that GHC has poor profiling options for long-running programs. Compiling GHC with profiling options significantly slows the performance. There is unmerged work making the GHC eventlog useful for such cases, but the state of this work is unclear.

Looking forward

So there it is, the first operational issue with Cachix. Despite this issue, I am happy to have made the choices that both allow me to respond quickly to the needs of Nix community, yet still allow me to further improve and stabilize the code with confidence as the product matures.

Speaking of maturing the product, I will share another announcement soon!

• Making sure that all dependencies of an app are present, such as a database for storage.
• Executing all relevant deployment activities to make an app available for use.
• Upgrading is risky and difficult -- it may break the application and introduce downtime.

Using tools from the Nix project

Automating Mendix application deployments with Nix

• We must obtain the Mendix runtime that interprets the Mendix models. Packaging the Mendix runtime in Nix is fairly straight forward -- simply unzipping the distribution, and moving the package contents into the Nix store, and adding a wrapper script launches the runtime suffices.
• We must produce a Mendix Deployment Archive (MDA file) that creates a Zip container with all artifacts required to run a Mendix app by the runtime. An MDA file can be produced from a Mendix project by invoking the MxBuild tool. Since MxBuild is required for this, I had to package it as well. Packaging mxbuild is a bit trickier, as it requires mono and Node.js.

Building an MDA file with Nix

{stdenv, mxbuild, jdk, nodejs}:
{name, mendixVersion, looseVersionCheck ? false, buildInputs ? [], ...}@args:

let
  mxbuildPkg = mxbuild."${mendixVersion}";
  extraArgs = removeAttrs args [ "buildInputs" ];
in
stdenv.mkDerivation ({
  buildInputs = [ mxbuildPkg nodejs ] ++ buildInputs;
  installPhase = ''
    mkdir -p $out
    mxbuild --target=package \
      --output=$out/${name}.mda \
      --java-home ${jdk} \
      --java-exe-path ${jdk}/bin/java \
      ${stdenv.lib.optionalString looseVersionCheck "--loose-version-check"} \
      "$(echo *.mpr)"
    mkdir -p $out/nix-support
    echo "file binary-dist \"$(echo $out/*.mda)\"" > $out/nix-support/hydra-build-products
  '';
} // extraArgs)

{ pkgs ? import  { inherit system; }                                                                                                                             
, system ? builtins.currentSystem
}:

let
  mendixPkgs = import ./nixpkgs-mendix/top-level/all-packages.nix {
    inherit pkgs system;
  };
in
mendixPkgs.packageMendixApp {
  name = "conferenceschedule";
  src = /home/sander/SharedWindowsFolder/ConferenceSchedule-main;
  mendixVersion = "7.13.1";
}

$ nix-build conferenceschedule.nix 
/nix/store/nbaa7fnzi0xw9nkf27mixyr9awnbj16i-conferenceschedule
$ ls /nix/store/nbaa7fnzi0xw9nkf27mixyr9awnbj16i-conferenceschedule
conferenceschedule.mda  nix-support

Running a Mendix application

• We must unzip the MDA file into a directory with write permissions.
• We must create writable state sub directories, e.g. data/tmp, data/files.
• After starting the runtime, we must configure the admin interface, to send instructions to the runtime to initialize the database and start the app:
  

  
  $ export M2EE_ADMIN_PORT=9000
  $ export M2EE_ADMIN_PASS=secret
  

• Finally, we must communicate over the admin interface to configure, initialize the database and start the app:
  

  
  curlCmd="curl -X POST http://localhost:$M2EE_ADMIN_PORT \
  -H 'Content-Type: application/json' \
  -H 'X-M2EE-Authentication: $(echo -n "$M2EE_ADMIN_PASS" | base64)' \
  -H 'Connection: close'"
  $curlCmd -d '{ "action": "update_appcontainer_configuration", "params": { "runtime_port": 8080 } }'
  $curlCmd -d '{ "action": "update_configuration", "params": { "DatabaseType": "HSQLDB", "DatabaseName": "myappdb", "DTAPMode": "D" } }'
  $curlCmd -d '{ "action": "execute_ddl_commands" }'
  $curlCmd -d '{ "action": "start" }'
  

NixOS integration

{pkgs, ...}:

{
  ...

  systemd.services.mendixappcontainer =
    let
      runScripts = ...
      appContainerConfigJSON = ...
      configJSON = ...
    in {
      enable = true;
      description = "My Mendix App";
      wantedBy = [ "multi-user.target" ];
      environment = {
        M2EE_ADMIN_PASS = "secret";
        M2EE_ADMIN_PORT = "9000";
        MENDIX_STATE_DIR = "/home/mendix";
      };
      serviceConfig = {
        ExecStartPre = "${runScripts}/bin/undeploy-app";
        ExecStart = "${runScripts}/bin/start-appcontainer";
        ExecStartPost = "${runScripts}/bin/configure-appcontainer ${appContainerConfigJSON} ${configJSON}";
      };
    };

• The undeploy-app script removes all non-state artefacts from the working directory.
• The start-appcontainer script starts the Mendix runtime.
• The configure-appcontainer script configures the runtime, such as the embedded Jetty server and the database, and starts the application.

{pkgs, ...}:

{
  require = [ ../nixpkgs-mendix/nixos/modules/mendixappcontainer.nix ];

  services = {
    openssh.enable = true;

     mendixAppContainer = {
       enable = true;
       adminPassword = "secret";
       databaseType = "HSQLDB";
       databaseName = "myappdb";
       DTAPMode = "D";
       app = import ../../conferenceschedule.nix {
         inherit pkgs;
         inherit (pkgs.stdenv) system;
      };
    };
  };

  networking.firewall.allowedTCPPorts = [ 8080 ];
}

• The password for communicating over the admin interface.
• The type of database we want to use (in this particular case an in memory HSQLDB instance) and the name of the database.
• Whether we want to use the application in development (D), test (T), acceptance (A) or production (P) mode.
• A reference to the MDA that we want to deploy (deployed by a Nix expression that invokes the Mendix build function abstraction shown earlier).

$ nixos-rebuild switch

Conclusion

• A Nix function that can be used to compile an MDA file from a Mendix project.
• Generated scripts that configure and launch the runtime and the application.
• A NixOS module that can be used to deploy a running Mendix app as part of a NixOS machine configuration.

Future work

Availability

Acknowledgements

EPYC vs m1.xlarge.x86

Nix is a powerful package manager for Linux and other Unix systems that makes package management reliable and reproducible. It provides atomic upgrades and rollbacks, side-by-side installation of multiple versions of a package, multi-user package management and easy setup of build environments.

The Nix community has collected and curated build instructions (expressions) for many thousands of packages in the Nix package collection, Nixpkgs. The collection is a large GitHub repository, which receives over a thousand pull requests each month. Some of these changes can sometimes cause all of the packages to rebuild.

To test changes to Nixpkgs and release updates for Nix and NixOS, we necessarily created our own build infrastructure. This allows us to give better quality guarantees to our users.

The NixOS infrastructure team runs several types of servers: VMs on AWS, bare metal, macOS systems, among others. We build thousands of packages a day, sometimes reaching many tens of thousands per day.

Some of our builds depend on unique features like KVM which are only available by using bare metal servers, and all of them benefit from numerous, powerful cores.

For over a year now, Packet.net (where you can natively deploy NixOS by the way!) has been generously providing bare metal hardware build resources for the NixOS build farm, and together we were curious how the new EPYC from AMD would compare to the hardware we were already using.

For this benchmark we are comparing Packet’s m1.xlarge.x86 against Packet’s first EPYC machine, c2.medium.x86. Hydra already runs a m1.xlarge.x86 build machine, so the comparison will be helpful in deciding if we should replace it with EPYC hardware.

AMD EPYC has the chance to reduce our hardware footprint, reduce our need for our AWS scale-out, and improve our turnaround time for time-sensitive security patches.

System Comparison:

Benchmark Methods

All of these tests were run on a fresh Packet.net server running NixOS 18.03 and building Nixpkgs revision 49a6964a425.

For each test, I ran each build five times on each machine with --check which forces a local rebuild:

checkM() {
  nix-build . -A "$1"
  for i in $(seq 1 5); do
    rm result
    nix-collect-garbage
    /run/current-system/sw/bin/time -ao "./time-$1" \
      nix-build . -A "$1" --check
  done
}

Benchmark Results

Kernel Builds

NixOS builds a generously featureful kernel by default, and the build can take some time. However, the compilation is easy to spread across multiple cores. In what we will further see is a theme, the EPYC beat the Intel CPU with by about five minutes, or about 35% speed-up.

nix-build '<nixpkgs>' -A linuxPackages.kernel

NixOS Tests

The NixOS release process is novel in that package updates happen automatically as an entire cohesive set. We of course test that specific software compiles properly, but we are also able to perform fully automatic integration tests. We test that the operating system can boot, desktop environments work, and even server tests like validating that our MySQL server replication is still working. These tests happen automatically on every release and is a unique benefit to the Nix build system being applied to an operating system.

These tests use QEMU and KVM, and spawn one or more virtual machines running NixOS.

Plasma5 and KDE

The Plasma5 test looks like the following:

1. First launches a NixOS VM configured to run the Plasma5 desktop manager.

2. Uses Optical Character Recognition to wait until it sees the system is ready for Alice to log in and then types in her password:

   $machine->waitForText(qr/Alice Foobar/);
   $machine->sendChars("foobar\n");
   

3. Waits for the Desktop to be showing, which reliably indicates she has finished logging in:

   $machine->waitForWindow("^Desktop ");
   

4. Launches Dolphin, Konsole, and Settings and waits for each window to appear before continuing:

   $machine->execute("su - alice -c 'dolphin &'");
   $machine->waitForWindow(" Dolphin");
   $machine->execute("su - alice -c 'konsole &'");
   $machine->waitForWindow("Konsole");
   $machine->execute("su - alice -c 'systemsettings5 &'");
   $machine->waitForWindow("Settings");
   

5. If all of these work correctly, the VM shuts down and the tests pass.

Better than the kernel tests, we’re pushing a 40% improvement.

nix-build '<nixpkgs/nixos/tests/plasma5.nix>'

MySQL Replication

The MySQL replication test launches a MySQL master and two slaves, and runs some basic replication tests. NixOS allows you to define replication as part of the regular configuration management, so I will start by showing the machine configuration of a slave:

    {
      services.mysql.replication.role = "slave";
      services.mysql.replication.serverId = 2;
      services.mysql.replication.masterHost = "master";
      services.mysql.replication.masterUser = "replicate";
      services.mysql.replication.masterPassword = "secret";
    }

1. This test starts by starting the $master and waiting for MySQL to be healthy:

   $master->start;
   $master->waitForUnit("mysql");
   $master->waitForOpenPort(3306);
   

2. Continues to start $slave1 and $slave2 and wait for them to be up:

   $slave1->start;
   $slave2->start;
   $slave1->waitForUnit("mysql");
   $slave2->waitForUnit("mysql");
   $slave1->waitForOpenPort(3306);
   $slave2->waitForOpenPort(3306);
   

3. It then validates some of the scratch data loaded in to the $master has replicated properly to $slave2:

   $slave2->succeed("
        echo 'use testdb; select * from tests' \
            | mysql -u root -N | grep 4
   ");
   

4. Then shuts down $slave2:

   $slave2->succeed("systemctl stop mysql");
   

5. Writes some data to the $master:

   $master->succeed("
        echo 'insert into testdb.tests values (123, 456);' \
            | mysql -u root -N
   ");
   

6. Starts $slave2, and verifies the queries properly replicated from the $master to the slave:

   $slave2->succeed("systemctl start mysql");
   $slave2->waitForUnit("mysql");
   $slave2->waitForOpenPort(3306);
   $slave2->succeed("
        echo 'select * from testdb.tests where Id = 123;' \
            | mysql -u root -N | grep 456
   ");
   

Due to the multiple VM nature, and increased coordination between the nodes, we saw a 30% increase.

nix-build '<nixpkgs/nixos/tests/mysql-replication.nix>'

BitTorrent

The BitTorrent test follows the same pattern of starting and stopping a NixOS VM, but this test takes it a step further and tests with four VMs which talk to each other. I could do a whole post just on this test, but in short:

• a machine serving as the tracker, named $tracker.
• a client machine, $client1
• another client machine, $client2
• a router which facilitates some of the incredible things this test is actually doing.

I’m going to gloss over the details here, but:

1. The $tracker starts seeding a file:

   $tracker->succeed("opentracker -p 6969 &");
   $tracker->waitForOpenPort(6969);
   my $pid = $tracker->succeed("transmission-cli \
        /tmp/test.torrent -M -w /tmp/data");
   

2. $client1 fetches the file from the tracker:

   $client1->succeed("transmission-cli \
        http://tracker/test.torrent -w /tmp &");
   

3. Kills the seeding process on tracker so now only $client1 is able to serve the file:

   $tracker->succeed("kill -9 $pid");
   

4. $client2 fetches the file from $client1:

   $client2->succeed("transmission-cli \
        http://tracker/test.torrent -M -w /tmp &");
   

If both $client1 and $client2 receive the file intact, the test passes.

This test sees a much lower performance improvement, largely due to the networked coordination across four VMs.

nix-build '<nixpkgs/nixos/tests/bittorrent.nix>'

The remarkable part I left out is Client1 uses UPnP to open a port of the firewall of the router which Client2 uses to read from Client1.

Standard Environment

Our standard build environment, stdenv, is at the deepest part of this tree and almost nothing can be built until after it is completed. stdenv is like build-essential on Ubuntu.

This is an important part of the performance story for us: Nix builds are represented by a tree, where Nix will schedule as many parallel builds as possible as long as its parents are done building. Single core performance is the primary factor impacting how long these builds take. Shaving off even a few minutes means our entire build cluster is able to get to share the work sooner.

For this reason, the stdenv test is the one exception to the methodology. I wanted to test a full build from bootstrap to a working standard build environment. To force this, I changed the very root build causing everything “beneath” it to require a rebuild by applying the following patch:

diff --git a/pkgs/stdenv/linux/default.nix b/pkgs/stdenv/linux/default.nix
index 63b4c8ecc24..1cd27f216f9 100644
--- a/pkgs/stdenv/linux/default.nix
+++ b/pkgs/stdenv/linux/default.nix
@@ -37,6 +37,7 @@ let

   commonPreHook =
     ''
+
       export NIX_ENFORCE_PURITY="''${NIX_ENFORCE_PURITY-1}"
       export NIX_ENFORCE_NO_NATIVE="''${NIX_ENFORCE_NO_NATIVE-1}"
       ${if system == "x86_64-linux" then "NIX_LIB64_IN_SELF_RPATH=1" else ""}

The impact on build time here is stunning and makes an enormous difference: almost a full 20 minutes shaved off the bootstrapping time.

nix-build '<nixpkgs>' -A stdenv

Conclusion

This EPYC machine has made a remarkable improvement in our build times and is helping the NixOS community push timely security updates and software updates to users and businesses alike. We look forward to expanding our footprint to keep up with the incredible growth of the Nix project.

Thank you to Packet.net for providing this hardware free of charge for this test through their EPYC Challenge.

and thank you Gustav, Daiderd, Andi, and zimbatm for their help with this article

with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "hello-2.10";

  src = fetchurl {
    url = mirror://gnu/hello/hello-2.10.tar.gz;
    sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
  };

  meta = {
    description = "A program that produces a familiar, friendly greeting";
    longDescription = ''
      GNU Hello is a program that prints "Hello, world!" when you run it.
      It is fully customizable.
    '';
    homepage = http://www.gnu.org/software/hello/manual/;
    license = "GPLv3+";
  };
}

$ nix-build
/nix/store/188avy0j39h7iiw3y7fazgh7wk43diz1-hello-2.10

• Build scripts can only write to designated output directories and temp directories. They are restricted from writing to any other file system location.
• All environment variables are cleared and some of them are set to default or dummy values, such as search path environment variables (e.g. PATH).
• All build results are made immutable by removing the write permission bits and their timestamps are reset to one second after the epoch.
• Running builds as unprivileged users.
• Optionally, builds run in a chroot environment and use namespaces to restrict access to the host filesystem and the network as much as possible.

Writing "raw" derivations

derivation {
  name = "test";
  builder = ./test.sh;
  system = "x86_64-linux";
  person = "Sander";
}

• The name attribute specifies the name of the package, that should appear in the resulting Nix store path.
• The builder attribute specifies that the test.sh executable should be run inside the pure build environment.
• The system attribute is used to tell Nix that this build should be carried out for x86-64 Linux systems. When Nix is unable to build the package for the requested system architecture, it can also delegate a build to a remote machine that is capable.
• All attributes (including the attributes described earlier) are converted to environment variables (e.g. strings, numbers and URLs are converted to strings and the boolean value: 'true' is converted to '1') and can be used by the builder process for a variety of reasons.

#!/bin/sh -e

echo "Hello $person" > $out

$ nix-build
/nix/store/7j4y5d8rx1vah5v64bpqd5dskhwx5105-test
$ cat result
Hello Sander

$ nix repl
Welcome to Nix version 2.0.4. Type :? for help.

nix-repl> test = import ./default.nix

nix-repl> :t test
a set

nix-repl> builtins.attrNames test
[ "all" "builder" "drvAttrs" "drvPath" "name" "out" "outPath" "outputName" "person" "system" "type" ]

• The type attribute that refers to the string: "derivation".
• The drvAttrs attribute refers to an attribute set containing the original parameters passed to derivation {}.
• drvPath and outPath refer to the Nix store paths of the store derivation file and output of the build. A side effect of requesting these members is that the expression gets evaluated or built.
• The out attribute is a reference to the derivation producing the out result, all is a list of derivations of all outputs produced (Nix derivations can also produce multiple output paths in the Nix store).
• In case there are multiple outputs, the outputName determines the name of the output path that is the default.

Providing basic dependencies

{bash, basePackages, system}:

let
  shell = "${bash}/bin/sh";
in
derivation {
  name = "stdenv";
  inherit shell basePackages system;
  builder = shell;
  args = [ "-e" ./builder.sh ];
}

set -e

# Setup PATH for base packages
for i in $basePackages
do
    basePackagesPath="$basePackagesPath${basePackagesPath:+:}$i/bin"
done

export PATH="$basePackagesPath"

# Create setup script
mkdir $out
cat > $out/setup <<EOF
export SHELL=$shell
export PATH="$basePackagesPath"
EOF

# Allow the user to install stdenv using nix-env and get the packages
# in stdenv.
mkdir $out/nix-support
echo "$basePackages" > $out/nix-support/propagated-user-env-packages

{stdenv}:

derivation {
  name = "hello";
  inherit stdenv;
  builder = ./builder.sh;
  system = "x86_64-linux";
}

#!/bin/sh -e
source $stdenv/setup

mkdir -p $out/bin

cat > $out/bin/hello <<EOF
#!$SHELL -e

echo "Hello"
EOF

chmod +x $out/bin/hello

Writing more simple derivations

{stdenv, system, shell}:
{builder, ...}@args:

let
  extraArgs = removeAttrs args [ "builder" "meta" ];

  buildResult = derivation ({
    inherit system stdenv;
    builder = shell; # Make bash the default builder
    args = [ "-e" builder ]; # Pass builder executable as parameter to bash
    setupSimpleDerivation = ./setup.sh;
  } // extraArgs);
in
buildResult //
# Readd the meta attribute to the resulting attribute set
(if args ? meta then { inherit (args) meta; } else {})

{stdenv}:

stdenv.simpleDerivation {
  name = "hello";
  builder = ./builder.sh;
  meta = {
    description = "This is a simple testcase";
  };
}

{stdenv, fetchurl, gnumake, gnutar, gzip, gcc, binutils}:

stdenv.simpleDerivation {
  name = "hello-2.10";
  src = fetchurl {
    url = mirror://gnu/hello/hello-2.10.tar.gz;
    sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
  };
  inherit stdenv gnumake gnutar gzip gcc binutils;
  builder = ./builder.sh;
}

source $setupSimpleDerivation

export PATH=$PATH:$gnumake/bin:$gnutar/bin:$gzip/bin:$gcc/bin:$binutils/bin

tar xfv $src
cd hello-2.10
./configure --prefix=$out
make
make install

The run command abstraction

{stdenv}:

stdenv.runCommand {
  name = "hello";
  buildCommand = ''
    mkdir -p $out/bin
    cat > $out/bin/hello <<EOF
    #! ${stdenv.shell} -e

    echo "Test"
    EOF
    chmod +x $out/bin/hello
  '';
}

{stdenv, ...}:

stdenv.mkDerivation {
  name = "perl";
  ...
  setupHook = ./setup-hook.sh
}

addPerlLibPath()
{
    addToSearchPath PERL5LIB $1/lib/perl5/site_perl
}

envHooks+=(addPerlLibPath)

{stdenv, fetchurl, gnumake, gnutar, gzip, gcc, binutils}:

stdenv.runCommand {
  name = "hello-2.10";
  src = fetchurl {
    url = mirror://gnu/hello/hello-2.10.tar.gz;
    sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
  };
  buildInputs = [ gnumake gnutar gzip gcc binutils ];
  buildCommand = ''
    tar xfv $srcb
    cd hello-2.10
    ./configure --prefix=$out
    make
    make install
  '';
}

The run phases abstraction

{stdenv}:

stdenv.runPhases {
  name = "hello";
  phases = [ "build" "install" ];
  buildPhase = ''
    cat > hello <<EOF
    #! ${stdenv.shell} -e
    echo "Hello"
    EOF
    chmod +x hello
  '';
  installPhase = ''
    mkdir -p $out/bin
    mv hello $out/bin
  '';
}

{stdenv}:

stdenv.runPhases {
  name = "hello2";
  builder = ./builder.sh;
}

source $setupRunPhases

phases="build install"

buildPhase()
{
    cat > hello <<EOF
#! $SHELL -e
echo "Hello"
EOF
    chmod +x hello
}

installPhase()
{
    mkdir -p $out/bin
    mv hello $out/bin
}

genericBuild

The generic build abstraction

• The unpack phase generically unpacks the provided sources, makes it content writable and opens the source directory. The unpack command is determined by the unpack hook that each potential unpacker provides -- for example, the GNU tar package includes a setup hook that untars the file if it looks like a tarball or compressed tarball:
  
  

  
  _tryUntar()
  {
      case "$1" in
          *.tar|*.tar.gz|*.tar.bz2|*.tar.lzma|*.tar.xz)
              tar xfv "$1"
              ;;
          *)
              return 1
              ;;
      esac
  }
  
  unpackHooks+=(_tryUntar)
  

• The patch phase applies any patch that is provided by the patches parameter uncompressing them when necessary. The uncompress file operation also works with setup hooks -- uncompressor packages (such as gzip and bzip2) provide a setup hook that uncompresses the file if it is of the right filetype.
• The strip phase processes all sub directories containing ELF binaries (e.g. bin/ and lib/) and strips their debugging symbols. This reduces the size of the binaries and removes non-deterministic timestamps.
• The patchShebangs phase processes all scripts with a shebang line and changes it to correspond to a path in the Nix store.
• The compressManPages phase compresses all manual pages with gzip.

{stdenv, fetchurl, gnumake, gnutar, gzip, gcc, binutils}:

stdenv.genericBuild {
  name = "hello-2.10";
  src = fetchurl {
    url = mirror://gnu/hello/hello-2.10.tar.gz;
    sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
  };
  buildInputs = [ gnumake gnutar gzip gcc binutils ];
  buildCommandPhase = ''
    ./configure --prefix=$out
    make
    make install
  '';
}

GNU Make/GNU Autotools abstraction

{stdenv, fetchurl}:

stdenv.mkDerivation {
  name = "hello-2.10";
  src = fetchurl {
    url = mirror://gnu/hello/hello-2.10.tar.gz;
    sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
  };
}

Composing custom function abstractions

{stdenv}:

{ name # the name of the derivation
, text
, executable ? false # run chmod +x ?
, destination ? ""   # relative path appended to $out eg "/bin/foo"
, checkPhase ? ""    # syntax checks, e.g. for scripts
}:

stdenv.runCommand {
  inherit name text executable;
  passAsFile = [ "text" ];

  # Pointless to do this on a remote machine.
  preferLocalBuild = true;
  allowSubstitutes = false;

  buildCommand = ''
    target=$out${destination}
    mkdir -p "$(dirname "$target")"

    if [ -e "$textPath" ]
    then
        mv "$textPath" "$target"
    else
        echo -n "$text" > "$target"
    fi

    [ "$executable" = "1" ] && chmod +x "$target" || true
  '';
}

{stdenv, foo, bar}:
{name, buildInputs ? [], ...}@args:

let
  extraArgs = removeAttrs args [ "name" "buildInputs" ];
in
stdenv.someBuildFunction ({
  name = "mypackage-"+name;
  buildInputs = [ foo bar ] ++ buildInputs;
} // extraArgs)

• A build function is a nested function in which the first line is a function header that captures the common build-time dependencies required to build a package. For example, when we want to build Perl packages, then perl is such a common dependency.
• The second line is the inner function header that captures the parameters that should be passed to the build function. The notation allows an arbitrary number of parameters. The parameters in the { } block (name, buildInputs) are considered to have a specific use in the body of the function. The remainder of parameters are non-essential -- they are used as environment variables in the builder environment or they can be propagated to other functions.
• We compose an extraArgs variable that contains all non-essential arguments that we can propagate to the build function. Basically, all function arguments that are used in the body need to be removed and function arguments that are attribute sets, because they cannot be converted to strings.
• In the body of the function, we set up important aspects of the build environment, such as the mandatory build parameters, and we propagate the remaining function arguments to the builder abstraction function.

Discussion

0. "Raw" derivations
1. The setup script ($stdenv/setup)
2. Simple derivation (stdenv.simpleDerivation)
3. The run command abstraction (stdenv.runCommand)
4. The run phases abstraction (stdenv.runPhases)
5. The generic build abstraction (stdenv.genericBuild)
6. The GNU Make/GNU Autotools abstraction (stdenv.mkDerivation)

• We could also split the implementation of stdenv.mkDerivation and the corresponding setup.sh script into layered sub functions. Currently, the setup.sh script is huge (e.g. over 1200 LOC) and has many responsibilities (perhaps too many). By splitting the build abstraction functions and their corresponding setup scripts, we can separate concerns better and reduce the size of the script so that it becomes more readable and better maintainable.
• In the Nixpkgs implementation, the phases that the generic builder executes are built for GNU Make/GNU Autotools specifically. Furthermore, the invocation of pre and post hooks and do and dont flags are all hand coded for every phase (there is no generic mechanism that deals with them). As a result, when you define a new custom phase, you need to reimplement the same aspects over and over again. In my implementation, you only have to define phases -- the generic builder automatically executes the coresponding pre and post hooks and evaluates the do and dont flags.
• In the Nixpkgs implementation there is no uncompressHook -- as a result, the decompression of patch files is completely handcoded for every uncompressor, e.g. gzip, bzip2, xz etc. In my implementation, we can delegate this responsibility to any potential uncompressor package.
• In my implementation, I turned some of the phases of the generic builder into command-line tools that can be invoked outside the build environment (e.g. patch-shebangs, compress-man). This makes it easier to experiment with these tools and to make adjustments.

Availability

I’m delighted to be able to announce that users all around the world will now have a great experience when fetching from the NixOS cache.

I heard several times from users in Hong Kong and Singapore that the cache was “slow”, but I didn’t know it was this slow! After working closely with a team of Nix users in Bangalore, I experienced first-hand just how eye-wateringly slow it could be.

The NixOS cache is now being served from all of AWS CloudFront edge locations, significantly reducing latency for users in Asia, Africa, South America, and Oceania.

By expanding the cache’s distribution settings to include all of the edge locations, performance has been substantially improved build time:

Experiments in Tokyo and Hong Kong produced similar results.

NixOS’s cache is stored in AWS S3, and distributed using AWS CloudFront. This combination gives us the excellent durability guarantees of S3 combined with the large geographical distribution of CloudFront.

Until today, the NixOS cache was only served through edge nodes in the United States, Canada, and Europe.

A big thank-you to Amine Chikhaoui, and Eelco Dolstra for their help in researching this change and turning on such a massive improvement.

In the last 6 years working with Nix and mostly in last two years full-time, I've noticed a few patterns.

These are mostly direct or indirect result of not having a "good enough" infrastructure to support how much Nix has grown (1600+ contributors, 1500 pull requests per month).

Without further ado, I am announcing https://cachix.org - Binary Cache as a Service that is ready to be used after two months of work.

What problem(s) does cachix solve?

The main motivation is to save you time and compute resources waiting for your packages to build. By using a shared cache of already built packages, you'll only have to build your project once.

This should also speed up CI builds, as Nix can take use of granular caching of each package, rather than caching the whole build.

Another one (which I personally consider even more important) is decentralization of work produced by Nix developers. Up until today, most devs pushed their software updates into the nixpkgs repository, which has the global binary cache at https://cache.nixos.org.

But as the community grew, fitting different ideologies into one global namespace became impossible. I consider nixpkgs community to be mature but sometimes clash of ideologies with rational backing occurs. Some want packages to be featureful by default, some prefer them to be minimalist. Some might prefer lots of configuration knobs available (for example cross-compilation support or musl/glib swapping), some might prefer the build system to do just one thing, as it's easier to maintain.

These are not right or wrong opinions, but rather a specific view of use cases that software might or might not cover.

There are also many projects that don't fit into nixpkgs because their releases are too frequent, they are not available under permissive license, are simpler to manage over complete control or maintainers simply disagree with requirements that nixpkgs developers impose on contributors.

And that's fine. What we've learned in the past is not to fight these ideas, but allow them to co-exist in different domains.

If you're interested:

• roadmap
• faq
• GitHub repo for the client
• HN

Domen (domen@enlambda.com)

Quick post to annouce summer plans. I have accepted a summer internship position at Obsidian Systems in the greatest city in the world. Very excited to be working with a team of really smart people.

Just wanted to congratulate everyone for all of the work in keeping Nixpkgs up-to-date!

Repology has just updated our stats and we have shown significant improvement. 66% of nixpkgs-unstable packages are up-to-date, with a total of 6112 up-to-date packages¹. Nixpkgs is slowly growing to match package sets that come in many of the big Linux distros. We now have more up-to-date packages than rolling release distros Arch (5924) and Manjaro (5708)! We certainly have a ways to go to compete with heavyweights like FreeBSD Ports (15410), DragonFly BSD Ports (14268), and Debian (14915) but these are ancient projects that have had a huge headstart. We are definitely making progress and it’s visible in the Repology graphs².

For comparison, in February, nixpkgs-unstable had 60% of packages up-to-date, with a total of 5439 up-to-date packages³. This kind of progress over 1 month is tremendously helpful. Remember that most of these updates will be included in 18.09, not the soon to be release 18.03. This protects users from potentially breaking changes as well as giving maintainers plenty of time to verify that the updates have not had any negative effects.

Ryan Mulligan’s nix-update has made updating packages much easier and I think it will be a game changer for Nixpkgs in the long run. There are still some kinks in it but I think it is slowly getting better. It has certainly been a headache to handle the huge increase in pull requests, but we are slowly clearing through that glut of outdated software. If an update to nixpkgs-unstable breaks something for you, please be sure to open an issue.

Munich NixOS Meetup

- Profpatsch: Nix(OS): Package-management done right (German)

Augsburg - Germany

Monday, March 19 at 7:00 PM

2

https://www.meetup.com/Munich-NixOS-Meetup/events/248479678/

motivation

in the nix-language-atlas series on lastlog.de/blog i want to discuss how well programming languages, for which i’m familiar with, integrate with nix. today, let’s revisit emscripten, as there also had been improvements since i wrote about it last time.

projects we have done:

• https://lastlog.de/blog/posts/xmlmirror.html
• https://lastlog.de/blog/posts/tour_of_nix.html

what’s new

• emscripten toolchain:
  • refactored to force a common revision (for example 1.37.16) on emscripten, emscripten-fastcomp and emscripten-fastcomp-clang called emscriptenVersion
  • added a unit test in emscripten to verify a small part of the toolchain
  • initial emscripten documentation in nixpkgs (not on https://nixos.org/nixpkgs/manual yet)
• added 2 more unit tests & repaired all builds

for details see the PR https://github.com/NixOS/nixpkgs/pull/37291.

want to give this a shot from nixos?

git clone https://github.com/nixos/nixpkgs/
cd nixpkgs
git checkout f41a3e7d7d327ea66459d17bfbe4a751b2496cb1
nix-env -f default.nix -I nixpkgs=. -iA emscriptenPackages
installing ‘emscripten-json-c-0.13’
installing ‘emscripten-libxml2-2.9.7’
installing ‘emscripten-xmlmirror’
installing ‘emscripten-zlib-1.2.11’
...

dynamic/static libraries

in the above toolchain we are using libraries in .so format, not the .a format and in the end we link them together using emcc. this has some advantages:

• building .so files is best practice on linux
• thus easy to do
• license wise smart, some packages can’t be statically linked legally IIRC

coverage

nix emscripten toolchain is now supported from:

• NixOS (nix)
• linux (nix)
• mac os (NI)X
• windows (WSL+ubuntu+nix)

especially the microsoft WSL release with the creators update is a very interesting audience as it makes it so easy to use nix on windows. no mingw, no cygwin! took me 10 minutes to install nix on windows!

for testing i’ve been compiling this very toolchain on my windows 10 computer. IO is slow but it works and it is easy to deploy.

emsdk

i’ve sumbled on this questions in https://github.com/juj/emsdk:

How do I change the currently active SDK version?

How do I build multiple projects with different SDK versions in parallel?

How do I use Emscripten SDK with a custom version of python, java, node.js or some other tool?

this are very interesting questions and all get easy once one uses this nixpkgs based toolchain as pointed out. i’ve been using the emsdk in the past but now that we have the ‘bits’ automated in nixpkgs i’m happy to not have to work statefully anymore!

ideas

during my last two days of work on the toolchain update, i had these ideas and motivations for the future:

• more documentation:
  • nix-shell -A usage
  • cross platform nix usage
  • section how to (fast) write good and lasting unit tests
  • how to use older nixpkgs vs. newer ones (slightly modify a very old project for instance which has a bug)
• packaging some common targts, what would these be?
  • bzip2?
  • empython?
  • SDL?
  • some games?
  • https://github.com/kripken/emscripten/wiki/Porting-Examples-and-Demos
  • https://github.com/emscripten-ports

• package the caching of emscripten in ~/.emscripten properly, so that build artefacts can be reused over builds (time save) and remove the HOME=$TMPDIR requirement (ugly)

    DEBUG:root:adding object /home/joachim/.emscripten_cache/asmjs/dlmalloc.bc to link
    DEBUG:root:adding object /home/joachim/.emscripten_cache/asmjs/libc.bc to link

• update nix-instantiate, which we use in the ‘tour of nix’, to a more recent version and also package it into the toolchain as an example
• hydra builds

• be more transparent on the license, ideally we could generate a list of licenses in the final folder

@nixos community: i’d love to get some feedback on this, so send me an email to js@lastlog.de if you have some interesting input.

summary

thanks to kripken (emscripten author) for his help! i’d love to put more effort into this, i think it’s really worth it so if you have any funding for general development or want to have something special realized, let me know!

another interesting project i’ve learned about lately would be https://github.com/NixOS/nixpkgs/pull/37291 from https://github.com/Ericson2314.

Why should I use Disnix and why would it be useful?

Creating a componentized web information system

• I decided to keep the databases of each applications separate as opposed to integrating all the tables into one single database. My main motivating factor was that I wanted to prevent another developer's wrong decisions from messing up the other application. Moreover, I realized that for the data that was specific to the application domain that other systems did not have to know about it.
• In addition to domain specific data, I noticed that both databases also stored the same kind of data, namely: user accounts -- both systems had a user account system to allow users to change the data. This also did not motivate me to integrate both databases into one database. Instead, I created a separate user database and authentication system (as a library API) that was shared among both applications.

Deployment with Disnix

• Domain-specific web applications exposing functionality that logically belongs together.
• Domain-specific databases storing tables that are strongly correlated.
• A shared user database.
• A portal application redirecting users to the domain-specific web applications.

$ disnixos-env -s services.nix \
  -n network-single.nix \
  -d distribution-single.nix --use-nixops

{infrastructure}:

{
  usersdb = [ infrastructure.test1 ];
  cmsdb = [ infrastructure.test1 ];
  cmsgallerydb = [ infrastructure.test1 ];
  homeworkdb = [ infrastructure.test1 ];
  literaturedb = [ infrastructure.test1 ];
  portaldb = [ infrastructure.test1 ];

  cms = [ infrastructure.test1 ];
  cmsgallery = [ infrastructure.test1 ];
  homework = [ infrastructure.test1 ];
  literature = [ infrastructure.test1 ];
  users = [ infrastructure.test1 ];
  portal = [ infrastructure.test1 ];
}

• The surrounding light grey colored box denotes a target machine. In this particular example, we only have one single target machine where services are deployed to.
• The dark grey colored boxes correspond to container environments. For our example system, we have two of them: mysql-database corresponding to a MySQL DBMS server and apache-webapplication corresponding to an Apache HTTP server.
• The ovals denote services corresponding to MySQL databases and web applications.
• The arrows denote inter-dependency links that correspond to network connections. As explained in my previous blog post, solid arrows are dependencies with a strict ordering requirement while dashed arrows are dependencies without an ordering requirement.

{infrastructure}:

{
  usersdb = [ infrastructure.test2 ];
  cmsdb = [ infrastructure.test2 ];
  cmsgallerydb = [ infrastructure.test2 ];
  homeworkdb = [ infrastructure.test2 ];
  literaturedb = [ infrastructure.test2 ];
  portaldb = [ infrastructure.test2 ];

  cms = [ infrastructure.test1 ];
  cmsgallery = [ infrastructure.test1 ];
  homework = [ infrastructure.test1 ];
  literature = [ infrastructure.test1 ];
  users = [ infrastructure.test1 ];
  portal = [ infrastructure.test1 ];
}

$ disnixos-env -s services.nix \
  -n network-separate.nix \
  -d distribution-separate.nix --use-nixops

$ disnixos-env -s services.nix \
  -n network-bundles.nix \
  -d distribution-bundles.nix --use-nixops

State management

$ disnixos-env -s services.nix \
  -n network-bundles.nix \
  -d distribution-bundles.nix --use-nixops --deploy-state

$ disnix-snapshot

$ dysnomia-snapshots --query-all
apache-webapplication/cms/1f9ed847885d2b3e3c67c51231122d958751eb5e2443c281e02e1d7108a505a3
apache-webapplication/cmsgallery/28d17a6941cb195a92e748aae737ccf524747477c6943436b734891d0f36fd53
apache-webapplication/literature/ed5ec4f8b9b4fcdb8b740ad1fa7ecb40b10dece03548f1d6e09a6a82c804131b
apache-webapplication/portal/5bbea499f8f8a4f708bb873ad683dbf088afa4c553f90ab287a9249a7ef02651
mysql-database/cmsdb/aa75992f780991c39a0969dcac5f69b04685c4fa764937476b816e938d6972ba
mysql-database/cmsgallerydb/31ebdaba658ca376123ff6a91a3e275731b383346a07840b1acaa1e44d921b65
mysql-database/homeworkdb/f0fda91545af0cb300afd84592d4914dcd48257053401e232438e34d83af828d
mysql-database/literaturedb/cb881c2200a5f1562f0b66f1394d0902bbb8e2361068fe096faac3bc31f76b5d
mysql-database/portaldb/5d8a5cb952f40ce76f93eb939d0b37eab33736d7b1e1426038322f8a572034ee
mysql-database/usersdb/64d11fc7f8969da5da318276a666f2e00e0a020ba619a1d82ed9b84a7f1c2ca6

$ find $(dysnomia-snapshots --resolve $(dysnomia-snapshots --query-all)) -type f
/home/sander/state/snapshots/apache-webapplication/cms/1f9ed847885d2b3e3c67c51231122d958751eb5e2443c281e02e1d7108a505a3/state.tar.xz
/home/sander/state/snapshots/apache-webapplication/cmsgallery/28d17a6941cb195a92e748aae737ccf524747477c6943436b734891d0f36fd53/state.tar.xz
/home/sander/state/snapshots/apache-webapplication/literature/ed5ec4f8b9b4fcdb8b740ad1fa7ecb40b10dece03548f1d6e09a6a82c804131b/state.tar.xz
/home/sander/state/snapshots/apache-webapplication/portal/5bbea499f8f8a4f708bb873ad683dbf088afa4c553f90ab287a9249a7ef02651/state.tar.xz
/home/sander/state/snapshots/mysql-database/cmsdb/aa75992f780991c39a0969dcac5f69b04685c4fa764937476b816e938d6972ba/dump.sql.xz
/home/sander/state/snapshots/mysql-database/cmsgallerydb/31ebdaba658ca376123ff6a91a3e275731b383346a07840b1acaa1e44d921b65/dump.sql.xz
/home/sander/state/snapshots/mysql-database/homeworkdb/f0fda91545af0cb300afd84592d4914dcd48257053401e232438e34d83af828d/dump.sql.xz
/home/sander/state/snapshots/mysql-database/literaturedb/cb881c2200a5f1562f0b66f1394d0902bbb8e2361068fe096faac3bc31f76b5d/dump.sql.xz
/home/sander/state/snapshots/mysql-database/portaldb/5d8a5cb952f40ce76f93eb939d0b37eab33736d7b1e1426038322f8a572034ee/dump.sql.xz
/home/sander/state/snapshots/mysql-database/usersdb/64d11fc7f8969da5da318276a666f2e00e0a020ba619a1d82ed9b84a7f1c2ca6/dump.sql.xz

Conclusion

Availability