Most Python projects are in fact polyglot. Indeed, many popular libraries on PyPi are Python wrappers around C code. This applies particularly to popular scientific computing packages, such as scipy and numpy. Normally, this is the terrain where Nix shines, but its support for Python projects has often been labor-intensive, requiring lots of manual fiddling and fine-tuning. One of the reasons for this is that most Python package management tools do not give enough static information about the project, not offering the determinism needed by Nix.

Thanks to Poetry, this is a problem of the past — its rich lock file offers more than enough information to get Nix running, with minimal manual intervention. In this post, I will show how to use Poetry, together with Poetry2nix, to easily manage Python projects with Nix. I will show how to package a simple Python application both using the existing support for Python in Nixpkgs, and then using Poetry2nix. This will both show why Poetry2nix is more convenient, and serve as a short tutorial covering its features.

Our application

We are going to package a simple application, a Flask server with two endpoints: one returning a static string “Hello World” and another returning a resized image. This application was chosen because:

1. It can fit into a single file for the purposes of this post.
2. Image resizing using Pillow requires the use of native libraries, which is something of a strength of Nix.

The code for it is in the imgapp/__init__.py file:

from flask import send_file
from flask import Flask
from io import BytesIO
from PIL import Image
import requests


app = Flask(__name__)


IMAGE_URL = "https://farm1.staticflickr.com/422/32287743652_9f69a6e9d9_b.jpg"
IMAGE_SIZE = (300, 300)


@app.route('/')
def hello():
    return "Hello World!"


@app.route('/image')
def image():
    r = requests.get(IMAGE_URL)
    if not r.status_code == 200:
        raise ValueError(f"Response code was '{r.status_code}'")

    img_io = BytesIO()

    img = Image.open(BytesIO(r.content))
    img.thumbnail(IMAGE_SIZE)
    img.save(img_io, 'JPEG', quality=70)

    img_io.seek(0)

    return send_file(img_io, mimetype='image/jpeg')


def main():
    app.run()


if __name__ == '__main__':
    main()

The status quo for packaging Python with Nix

There are two standard techniques for integrating Python projects with Nix.

Nix only

The first technique uses only Nix for package management, and is described in the Python section of the Nix manual. While it works and may look very appealing on the surface, it uses Nix for all package management needs, which comes with some drawbacks:

1. We are essentially tied to whatever package version Nixpkgs provides for any given dependency. This can be worked around with overrides, but those can cause version incompatibilities. This happens often in complex Python projects, such as data science ones, which tend to be very sensitive to version changes.
2. We are tied to using packages already in Nixpkgs. While Nixpkgs has many Python packages already packaged up (around 3000 right now) there are many packages missing — PyPi, the Python Package Index has more than 200000 packages. This can of course be worked around with overlays and manual packaging, but this quickly becomes a daunting task.
3. In a team setting, every team member wanting to add packages needs to buy in to Nix and at least have some experience using and understanding Nix.

All these factors lead us to a conclusion: we need to embrace Python tooling so we can efficiently work with the entire Python ecosystem.

Pip and Pypi2Nix

The second standard method tries to overcome the faults above by using a hybrid approach of Python tooling together with Nix code generation. Instead of writing dependencies manually in Nix, they are extracted from the requirements.txt file that users of Pip and Virtualenv are very used to. That is, from a requirements.txt file containing the necessary dependencies:

requests
pillow
flask

we can use pypi2nix to package our application in a more automatic fashion than before:

nix-shell -p pypi2nix --run "pypi2nix -r requirements.txt"

However, Pip is not a dependency manager and therefore the requirements.txt file is not explicit enough — it lacks both exact versions for libraries, and system dependencies. Therefore, the command above will not produce a working Nix expression. In order to make pypi2nix work correctly, one has to manually find all dependencies incurred by the use of Pillow:

nix-shell -p pypi2nix --run "pypi2nix -V 3.8 -E pkgconfig -E freetype -E libjpeg -E openjpeg -E zlib -E libtiff -E libwebp -E tcl -E lcms2 -E xorg.libxcb -r requirements.txt"

This will generate a large Nix expression, that will indeed work as expected. Further use of Pypi2nix is left to the reader, but we can already draw some conclusions about this approach:

1. Code generation results in huge Nix expressions that can be hard to debug and understand. These expressions will typically be checked into a project repository, and can get out of sync with actual dependencies.
2. It’s very high friction, especially around native dependencies.

Having many large Python projects, I wasn’t satisfied with the status quo around Python package management. So I looked into what could be done to make the situation better, and which tools could be more appropriate for our use-case. A potential candidate was Pipenv, however its dependency solver and lock file format were difficult to work with. In particular, Pipenv’s detection of “local” vs “non-local” dependencies did not work properly inside the Nix shell and gave us the wrong dependency graph. Eventually, I found Poetry and it looked very promising.

Poetry and Poetry2nix

The Poetry package manager is a relatively recent addition to the Python ecosystem but it is gaining popularity very quickly. Poetry features a nice CLI with good UX and deterministic builds through lock files.

Poetry uses pip under the hood and, for this reason, inherited some of its shortcomings and lock file design. I managed to land a few patches in Poetry before the 1.0 release to improve the lock file format, and now it is fit for use in Nix builds. The result was Poetry2nix, whose key design goals were:

1. Dead simple API.
2. Work with the entire Python ecosystem using regular Python tooling.
3. Python developers should not have to be Nix experts, and vice versa.
4. Being an expert should allow you to “drop down” into the lower levels of the build and customise it.

Poetry2nix is not a code generation tool — it is implemented in pure Nix. This fixes many of problems outlined in previous paragraphs, since there is a single point of truth for dependencies and their versions.

But what about our native dependencies from before? How does Poetry2nix know about those? Indeed, Poetry2nix comes with an extensive set of overrides built-in for a lot of common packages, including Pillow. Users are encouraged to contribute overrides upstream for popular packages, so everyone can have a better user experience.

Now, let’s see how Poetry2nix works in practice.

Developing with Poetry

Let’s start with only our application file above (imgapp/__init__.py) and a shell.nix:

{ pkgs ? import <nixpkgs> {} }:

pkgs.mkShell {

  buildInputs = [
    pkgs.python3
    pkgs.poetry
  ];

}

Poetry comes with some nice helpers to create a project, so we run:

$ poetry init

And then we’ll add our dependencies:

$ poetry add requests pillow flask

We now have two files in the folder:

• The first one is pyproject.toml which not only specifies our dependencies but also replaces setup.py.
• The second is poetry.lock which contains our entire pinned Python dependency graph.

For Nix to know which scripts to install in the bin/ output directory, we also need to add a scripts section to pyproject.toml:

[tool.poetry]
name = "imgapp"
version = "0.1.0"
description = ""
authors = ["adisbladis <adisbladis@gmail.com>"]

[tool.poetry.dependencies]
python = "^3.7"
requests = "^2.23.0"
pillow = "^7.1.2"
flask = "^1.1.2"

[tool.poetry.dev-dependencies]

[tool.poetry.scripts]
imgapp = 'imgapp:main'

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Packaging with Poetry2nix

Since Poetry2nix is not a code generation tool but implemented entirely in Nix, this step is trivial. Create a default.nix containing:

{ pkgs ? import <nixpkgs> {} }:
pkgs.poetry2nix.mkPoetryApplication {
  projectDir = ./.;
}

We can now invoke nix-build to build our package defined in default.nix. Poetry2nix will automatically infer package names, dependencies, meta attributes and more from the Poetry metadata.

Manipulating overrides

Many overrides for system dependencies are already upstream, but what if some are lacking? These overrides can be manipulated and extended manually:

poetry2nix.mkPoetryApplication {
    projectDir = ./.;
    overrides = poetry2nix.overrides.withDefaults (self: super: {
      foo = foo.overridePythonAttrs(oldAttrs: {});
    });
}

Conclusion

By embracing both modern Python package management tooling and the Nix language, we can achieve best-in-class user experience for Python developers and Nix developers alike.

There are ongoing efforts to make Poetry2nix and other Nix Python tooling work better with data science packages like numpy and scipy. I believe that Nix may soon rival Conda on Linux and MacOS for data science.

Python + Nix has a bright future ahead of it!

• Nix is a source-based package manager responsible for obtaining, installing, configuring and upgrading packages in a reliable and reproducible manner and facilitating the construction of packages from source code and their dependencies.
• Docker's purpose is to fully manage the life-cycle of applications (services and ordinary processes) in a reliable and reproducible manner, including their deployments.

Existing Nix integrations with process management

• NixOS builds entire machine configurations from a single declarative deployment specification and uses the Nix package manager to deploy and isolate all static artifacts of a system. It will also automatically generate and deploy systemd units for services defined in a NixOS configuration.
• nix-darwin can be used to specify a collection of services in a deployment specification and uses the Nix package manager to deploy all services and their corresponding launchd configuration files.

Docker functionality

• Process management. The life-cycle of a container is bound to the life-cycle of a root process that needs to be started or stopped.
• Dependency management. To ensure that applications have all the dependencies that they need and that no dependency is missing, Docker uses images containing a complete root filesystem with all required files to run an application.
• Resource isolation is heavily used for a variety of different reasons:
  
  • Foremost, to ensure that the root filesystem of the container does not conflict with the host system's root filesystem.
  • It is also used to prevent conflicts with other kinds of resources. For example, the isolated network interfaces allow services to bind to the same TCP ports that may also be in use by the host system or other containers.
  • It offers some degree of protection. For example, a malicious process will not be able to see or control a process belonging to the host system or a different container.
• Resource restriction can be used to limit the amount of system resources that a process can consume, such as the amount of RAM.
  
  Resource restriction can be useful for a variety of reasons, for example, to prevent a service from eating up all the system's resources affecting the stability of the system as a whole.
• Integrations with the host system (e.g. volumes) and other services.

Developing a Nix-based process management solution

• It uses Nix to deploy all required packages and other static artifacts (such as configuration files) that a service needs.
• It integrates with a variety of process managers on a variety of operating systems. So far, it can work with: sysvinit scripts, BSD rc scripts, supervisord, systemd, cygrunsrv and launchd.
  
  In addition to process managers, it can also automatically convert a processes model to deployment specifications that Disnix can consume.
• It uses declarative specifications to define functions that construct managed processes and process instances.
  
  Processes can be declared in a process-manager specific and process-manager agnostic way. The latter makes it possible to target all six supported process managers with the same declarative specification, albeit with a limited set of features.
• It allows you to run multiple instances of processes, by introducing a convention to cope with potential resource conflicts between process instances -- instance properties and potential conflicts can be configured with function parameters and can be changed in such a way that they do not conflict.
• It can facilitate unprivileged user deployments by using Nix's ability to perform unprivileged package deployments and introducing a convention that allows you to disable user switching.

{createManagedProcess, tmpDir}:
{port, instanceSuffix ? "", instanceName ? "webapp${instanceSuffix}"}:

let
  webapp = import ../../webapp;
in
createManagedProcess {
  name = instanceName;
  description = "Simple web application";
  inherit instanceName;

  process = "${webapp}/bin/webapp";
  daemonArgs = [ "-D" ];

  environment = {
    PORT = port;
    PID_FILE = "${tmpDir}/${instanceName}.pid";
  };
  user = instanceName;
  credentials = {
    groups = {
      "${instanceName}" = {};
    };
    users = {
      "${instanceName}" = {
        group = instanceName;
        description = "Webapp";
      };
    };
  };

  overrides = {
    sysvinit = {
      runlevels = [ 3 4 5 ];
    };
  };
}

• The outer function header (first line) refers to parameters that are common to all process instances: createManagedProcess is a function that can construct process manager configurations and tmpDir refers to the directory in which temp files are stored (which is /tmp in conventional Linux installations).
• The inner function header (second line) refers to instance parameters -- when it is desired to construct multiple instances of this process, we must make sure that we have configured these parameters in such as a way that they do not conflict with other processes.
  
  For example, when we assign a unique TCP port and a unique instance name (a property used by the daemon tool to create unique PID files) we can safely have multiple instances of this service co-existing on the same system.
• In the body, we invoke the createManagedProcess function to generate configurations files for a process manager.
• The process parameter specifies the executable that we need to run to start the process.
• The daemonArgs parameter specifies command-line instructions passed to the the process executable, when the process should daemonize itself (the -D parameter instructs the webapp process to daemonize).
• The environment parameter specifies all environment variables. Environment variables are used as a generic configuration facility for the service.
• The user parameter specifies the name the process should run as (each process instance has its own user and group with the same name as the instance).
• The credentials parameter is used to automatically create the group and user that the process needs.
• The overrides parameter makes it possible to override the parameters generated by the createManagedProcess function with process manager-specific overrides, to configure features that are not universally supported.
  
  In the example above, we use an override to configure the runlevels in which the service should run (runlevels 3-5 are typically used to boot a system that is network capable). Runlevels are a sysvinit-specific concept.

{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager
}:

let
  constructors = import ./constructors.nix {
    inherit pkgs stateDir runtimeDir logDir tmpDir;
    inherit forceDisableUserChange processManager;
  };
in
rec {
  webapp = rec {
    port = 5000;
    dnsName = "webapp.local";

    pkg = constructors.webapp {
      inherit port;
    };
  };

  nginxReverseProxy = rec {
    port = 8080;

    pkg = constructors.nginxReverseProxyHostBased {
      webapps = [ webapp ];
      inherit port;
    } {};
  };
}

• The first line is a function header in which the function parameters correspond to ajustable properties that apply to all process instances:
  • stateDir allows you to globally override the base directory in which all state is stored (the default value is: /var).
  • We can also change the locations of each individual state directories: tmpDir, cacheDir, logDir, runtimeDir etc.) if desired.
  • forceDisableUserChange can be enabled to prevent the process manager to change user permissions and create users and groups. This is useful to facilitate unprivileged user deployments in which the user typically has no rights to change user permissions.
  • The processManager parameter allows you to pick a process manager. All process configurations will be automatically generated for the selected process manager.
    
    For example, if we would pick: systemd then all configurations get translated to systemd units. supervisord causes all configurations to be translated to supervisord configuration files.
• To get access to constructor functions, we import a constructors expression that composes all constructor functions by calling them with their common parameters (not shown in this blog post).
  
  The constructors expression also contains a reference to the Nix expression that deploys the webapp service, shown in our previous example.
• The processes model defines two processes: a webapp instance that listens to TCP port 5000 and Nginx that acts as a reverse proxy forwarding requests to webapp process instances based on the virtual host name.
• webapp is declared a dependency of the nginxReverseProxy service (by passing webapp as a parameter to the constructor function of Nginx). This causes webapp to be activated before the nginxReverseProxy.

$ nixproc-sysvinit-switch --state-dir /home/sander/var \
  --force-disable-user-change processes.nix

$ nixproc-systemd-switch processes.nix

Integrating Docker into the process management framework

Using a shared Nix store

with import <nixpkgs> {};

let
  cmd = [ "${nginx}/bin/nginx" "-g" "daemon off;" "-c" ./nginx.conf ];
in
dockerTools.buildImage {
  name = "nginxexp";
  tag = "test";

  runAsRoot = ''
    ${dockerTools.shadowSetup}
    groupadd -r nogroup
    useradd -r nobody -g nogroup -d /dev/null
    mkdir -p /var/log/nginx /var/cache/nginx /var/www
    cp ${./index.html} /var/www/index.html
  '';

  config = {
    Cmd = map (arg: builtins.unsafeDiscardStringContext arg) cmd;
    Expose = {
      "80/tcp" = {};
    };
  };                                                                                                                                                                                                                          
}

$ docker run -p 8080:80 -v /nix/store:/nix/store -it nginxexp:latest

Using the host system's network

$ docker run -v /nix/store:/nix/store --network host -it nginxexp:latest

Mapping a base directory for storing state

$ docker run -v /nix/store:/nix/store \
  -v /var:/var --network host -it nginxexp:latest

Orchestrating containers

01-webapp-docker-priority
02-nginx-docker-priority
nginx-docker-cmd
nginx-docker-createparams
nginx-docker-settings
webapp-docker-cmd
webapp-docker-createparams
webapp-docker-settings

• The files that have a -docker-settings suffix contain general properties of the container, such as the image that needs to be used a template.
• The files that have a -docker-createparams suffix contain the command line parameters that are propagated to docker create to create the container. If a container with the same name already exists, the container creation is skipped and the existing instance is used instead.
• To prevent the Nix packages that a Docker container needs from being garbage collected the generator creates a file with a -docker-cmd suffix containing the Cmd instruction including the full Nix store paths of the packages that a container needs.
  
  Because the strings' contexts are not discarded in the generation process, the packages become a dependency of the configuration file. As long as this configuration file is deployed, the packages will not get garbage collected.
• To ensure that the containers are activated in the right order we have two files that are prefixed with two numeric digits that have a -container-priority suffix. The numeric digits determine in which order the containers should be activated -- in the above example the webapp process gets activated before Nginx (that acts as a reverse proxy).

$ nixproc-docker-switch processes.nix
55d833e07428: Loading layer [==================================================>]  46.61MB/46.61MB
Loaded image: webapp:latest
f020f5ecdc6595f029cf46db9cb6f05024892ce6d9b1bbdf9eac78f8a178efd7
nixproc-webapp
95b595c533d4: Loading layer [==================================================>]  46.61MB/46.61MB
Loaded image: nginx:latest
b195cd1fba24d4ec8542c3576b4e3a3889682600f0accc3ba2a195a44bf41846
nixproc-nginx

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES                                                   
b195cd1fba24        nginx:latest        "/nix/store/j3v4fz9h…"   15 seconds ago      Up 14 seconds                           nixproc-nginx                                           
f020f5ecdc65        webapp:latest       "/nix/store/b6pz847g…"   16 seconds ago      Up 15 seconds                           nixproc-webapp

Deploying Docker containers in a heteregenous and/or distributed environment

{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
}:

let
  constructors = import ./constructors.nix {
    inherit pkgs stateDir runtimeDir logDir tmpDir;
    inherit forceDisableUserChange;
    processManager = "docker";
  };
in
rec {
  webapp = rec {
    name = "webapp";

    port = 5000;
    dnsName = "webapp.local";

    pkg = constructors.webapp {
      inherit port;
    };

    type = "docker-container";
  };

  nginxReverseProxy = rec {
    name = "nginxReverseProxy";

    port = 8080;

    pkg = constructors.nginxReverseProxyHostBased {
      webapps = [ webapp ];
      inherit port;
    } {};

    type = "docker-container";
  };
}

{
  test1.properties.hostname = "test1";
}

{infrastructure}:

{
  webapp = [ infrastructure.test1 ];
  nginxReverseProxy = [ infrastructure.test1 ];
}

$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix

Deploying conventional Docker containers with Disnix

{dockerTools, stdenv, nginx}:

let
  dockerImage = dockerTools.buildImage {
    name = "nginxexp";
    tag = "test";
    contents = nginx;

    runAsRoot = ''
      ${dockerTools.shadowSetup}
      groupadd -r nogroup
      useradd -r nobody -g nogroup -d /dev/null
      mkdir -p /var/log/nginx /var/cache/nginx /var/www
      cp ${./index.html} /var/www/index.html
    '';

    config = {
      Cmd = [ "${nginx}/bin/nginx" "-g" "daemon off;" "-c" ./nginx.conf ];
      Expose = {
        "80/tcp" = {};
      };
    };
  };
in
stdenv.mkDerivation {
  name = "nginxexp";
  buildCommand = ''
    mkdir -p $out
    cat > $out/nginxexp-docker-settings <<EOF
    dockerImage=${dockerImage}
    dockerImageTag=nginxexp:test
    EOF

    cat > $out/nginxexp-docker-createparams <<EOF
    -p
    8080:80
    EOF
  '';
}

$ docker save nginx-debian -o nginx-debian.tar.gz

{dockerTools, stdenv, nginx}:

stdenv.mkDerivation {
  name = "nginxexp";
  buildCommand = ''
    mkdir -p $out
    cat > $out/nginxexp-docker-settings <<EOF
    dockerImage=${./nginx-debian.tar.gz}
    dockerImageTag=nginxexp:test
    EOF

    cat > $out/nginxexp-docker-createparams <<EOF
    -p
    8080:80
    EOF
  '';
}

Discussion

• Docker is heavily developed around Linux-specific concepts, such as namespaces and cgroups. As a result, it can only be used to deploy software built for Linux.
  
  The Nix process management framework should work on any operating system that is supported by the Nix package manager (e.g. Nix also has first class support for macOS, and can also be used on other UNIX-like operating systems such as FreeBSD). The same also applies to Disnix.
• The Nix process management framework can work with sysvinit, BSD rc and Disnix process scripts, that do not require any external service to manage a process' life-cycle. This is convenient for local unprivileged user deployments. To deploy Docker containers, you need to have the Docker daemon installed first.
• Docker has an experimental rootless deployment mode, but in the Nix process management framework facilitating unprivileged user deployments is a first class concept.

• Docker heavily relies on namespaces to prevent resource conflicts, such as overlapping TCP ports and global state directories. The Nix process management framework solves conflicts by avoiding them (i.e. configuring properties in such a way that they are unique). The conflict avoidance approach works as long as a service is well-specified. Unfortunately, preventing conflicts is not a hard guarantee that the tool can provide you.
• Docker also provides some degree of protection by using namespaces and cgroups. The Nix process management framework does not support this out of the box, because these concepts are not generalizable over all the process management backends it supports. (As a sidenote: it is still possible to use these concepts by defining process manager-specific overrides).

Availability

How to use Neovim (Neovim-Qt) under WSL 1/2 with the power of Nix

Intro

Well we all know that generally development on Linux is easier than on Windows, but sometimes you are forced to use Windows. But that does not mean that all those nice tools from Linux are not available to you, as we will see in this post.

Windows has some thing called WSL which enables you to run Linux tools natively in the Windows subsystem. Not all is without issues, you can not run graphical Linux applications because Windows does not run Xorg server, yeah you have Xorg ports that run there but that is in this case just one more unwanted layer, remember, building efficient solutions is what every engineer should strive to.

What I did is to use Windows pre-built binaries of Neovim-Qt and run the Neovim installed with Nix inside WSL.

Ok, you could say then, why not use VS Code with some Vim/Neovim plugin and use so called Remote-WSL plugin to access WSL… Well yes, but at least me I stumble upon few issues. First was that CPU usage was through the roof when Remote-WSL extension was in use on WSL1 (I could not just run Windows Update on client’s managed computer) and the fix was to install specific version of libc with dpkg (which is absurd in the first place because this is a good way to ruin your whole environment). Applying this fix did the trick for lowering the CPU usage. The second issue come right after, when I wanted to install some package with APT package manager, like I predicted, libc install did its damage, I could not install or un-install anything with APT. Nix comes again to the rescue.

By the way the sleep command forgot how to work under WSL and Ubuntu 20.04 Source.

Let’s see the solution

Neovim-Qt

Neovim-Qt has nicely built binaries on their GitHub page for Windows, so I just downloaded that zip and unpacked it into C:/Program Files/neovim-qt/. But any location could do.

WSL

Open PowerShell as Administrator and run:

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart

If you do not have up to date Windows for any kind of reason to install WLS2 then reboot now.

Reboot Time

You should have now enabled the WSL1 and you can proceed to install Ubuntu 20.04 (or any other Linux distro you like) from Microsoft store. Do not forget to click Launch after installing it (it will ask you to create a user).

Nix

To install Nix, you need to first open some terminal emulator and run wsl.exe, but you can also just run it from Start menu.

bash <(curl -L https://nixos.org/nix/install)

To finish you can just close the terminal and open wsl.exe again.

Thats it.

The Nix script

Now here is the absolutely most awesome part that connects everything together.

{ pkgs ? import <nixpkgs> {} }:
pkgs.writeScript "run-neovim-qt.sh" ''
    #!${pkgs.stdenv.shell}
    set -e

    # get random free port
    export NVIM_LISTEN="127.0.0.1:$(${pkgs.python3Packages.python}/bin/python -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1]); s.close()')"

    # use python's sleep, because coreutils' sleep does not function under Ubuntu 20.04 and WSL
    #   after delay start nvim-qt - so that nvim starts before the GUI
    { ${pkgs.python3Packages.python}/bin/python -c 'import time; time.sleep(1)'; "''${NVIM_QT_PATH}" --server "$NVIM_LISTEN"; } &

    # start nvim
    ${pkgs.neovim}/bin/nvim --listen "$NVIM_LISTEN" --headless "$@"
''

Save it to your drive or download with wget under WSL:

wget https://raw.githubusercontent.com/matejc/helper_scripts/master/nixes/neovim-qt.nix 

Then build the command with:

nix-build ./neovim-qt.nix

The resulting script is ./result.

Usage

First we need to tell the script where is the Neovim-qt located:

export NVIM_QT_PATH='/mnt/c/Program Files/neovim-qt/bin/nvim-qt.exe'

You can save this into .bashrc or .profile and restart the terminal so that you do not need to repeat the step every time you run wsl shell.

The final step is:

./result my/awesome/code.py

Conclusion

Too much work? You think? Well how much more time you would use using and configuring VS Code or Atom to work under similar environment? And what about Nix? You can install it without the use of native package managers (in case the native one is b0rked) and once you do, you have the power to install your favorite development environment with single command.

I like this solution, in my eyes its simple and efficient, what are your thoughts?

Until next time… I wish you happy hacking!

Links

High cpu usage of node process in Remote-WSL extension #2921 Neovim-Qt Releases WSL on Windows 10 Quick start with Nix

This is the third in a series of blog posts about Nix flakes. The first part motivated why we developed flakes — to improve Nix’s reproducibility, composability and usability — and gave a short tutorial on how to use flakes. The second part showed how flakes enable reliable caching of Nix evaluation results. In this post, we show how flakes can be used to manage NixOS systems in a reproducible and composable way.

What problems are we trying to solve?

Lack of reproducibility

One of the main selling points of NixOS is reproducibility: given a specification of a system, if you run nixos-rebuild to deploy it, you should always get the same actual system (modulo mutable state such as the contents of databases). For instance, we should be able to reproduce in a production environment the exact same configuration that we’ve previously validated in a test environment.

However, the default NixOS workflow doesn’t provide reproducible system configurations out of the box. Consider a typical sequence of commands to upgrade a NixOS system:

• You edit /etc/nixos/configuration.nix.
• You run nix-channel --update to get the latest version of the nixpkgs repository (which contains the NixOS sources).
• You run nixos-rebuild switch, which evaluates and builds a function in the nixpkgs repository that takes /etc/nixos/configuration.nix as an input.

In this workflow, /etc/nixos/configuration.nix might not be under configuration management (e.g. point to a Git repository), or if it is, it might be a dirty working tree. Furthermore, configuration.nix doesn’t specify what Git revision of nixpkgs to use; so if somebody else deploys the same configuration.nix, they might get a very different result.

Lack of traceability

The ability to reproduce a configuration is not very useful if you can’t tell what configuration you’re actually running. That is, from a running system, you should be able to get back to its specification. So there is a lack of traceability: the ability to trace derived artifacts back to their sources. This is an essential property of good configuration management, since without it, we don’t know what we’re actually running in production, so reproducing or fixing problems becomes much harder.

NixOS currently doesn’t not have very good traceability. You can ask a NixOS system what version of Nixpkgs it was built from:

$ nixos-version --json | jq -r .nixpkgsRevision
a84b797b28eb104db758b5cb2b61ba8face6744b

Unfortunately, this doesn’t allow you to recover configuration.nix or any other external NixOS modules that were used by the configuration.

Lack of composability

It’s easy to enable a package or system service in a NixOS configuration if it is part of the nixpkgs repository: you just add a line like environment.systemPackages = [ pkgs.hello ]; or services.postgresql.enable = true; to your configuration.nix. But what if we want to use a package or service that isn’t part of Nixpkgs? Then we’re forced to use mechanisms like $NIX_PATH, builtins.fetchGit, imports using relative paths, and so on. These are not standardized (since everybody uses different conventions) and are inconvenient to use (for example, when using $NIX_PATH, it’s the user’s responsibility to put external repositories in the right directories).

Put another way: NixOS is currently built around a monorepo workflow — the entire universe should be added to the nixpkgs repository, because anything that isn’t, is much harder to use.

It’s worth noting that any NixOS system configuration already violates the monorepo assumption: your system’s configuration.nix is not part of the nixpkgs repository.

Using flakes for NixOS configurations

In the previous post, we saw that flakes are (typically) Git repositories that have a file named flake.nix, providing a standardized interface to Nix artifacts. We saw flakes that provide packages and development environments; now we’ll use them to provide NixOS system configurations. This solves the problems described above:

• Reproducibility: the entire system configuration (including everything it depends on) is captured by the flake and its lock file. So if two people check out the same Git revision of a flake and build it, they should get the same result.
• Traceability: nixos-version prints the Git revision of the top-level configuration flake, not its nixpkgs input.
• Composability: it’s easy to pull in packages and modules from other repositories as flake inputs.

Prerequisites

Flake support has been added as an experimental feature to NixOS 20.03. However, flake support is not part of the current stable release of Nix (2.3). So to get a NixOS system that supports flakes, you first need to switch to the nixUnstable package and enable some experimental features. This can be done by adding the following to configuration.nix:

nix.package = pkgs.nixUnstable;
nix.extraOptions = ''
  experimental-features = nix-command flakes
'';

Creating a NixOS configuration flake

Let’s create a flake that contains the configuration for a NixOS container.

$ git init my-flake
$ cd my-flake
$ nix flake init -t templates#simpleContainer
$ git commit -a -m 'Initial version'

Note that the -t flag to nix flake init specifies a template from which to copy the initial contents of the flake. This is useful for getting started. To see what templates are available, you can run:

$ nix flake show templates

For reference, this is what the initial flake.nix looks like:

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-20.03";

  outputs = { self, nixpkgs }: {

    nixosConfigurations.container = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules =
        [ ({ pkgs, ... }: {
            boot.isContainer = true;

            # Let 'nixos-version --json' know about the Git revision
            # of this flake.
            system.configurationRevision = nixpkgs.lib.mkIf (self ? rev) self.rev;

            # Network configuration.
            networking.useDHCP = false;
            networking.firewall.allowedTCPPorts = [ 80 ];

            # Enable a web server.
            services.httpd = {
              enable = true;
              adminAddr = "morty@example.org";
            };
          })
        ];
    };

  };
}

That is, the flake has one input, namely nixpkgs - specifically the 20.03 branch. It has one output, nixosConfigurations.container, which evaluates a NixOS configuration for tools like nixos-rebuild and nixos-container. The main argument is modules, which is a list of NixOS configuration modules. This takes the place of the file configuration.nix in non-flake deployments. (In fact, you can write modules = [ ./configuration.nix ] if you’re converting a pre-flake NixOS configuration.)

Let’s create and start the container! (Note that nixos-container currently requires you to be root.)

# nixos-container create flake-test --flake /path/to/my-flake
host IP is 10.233.4.1, container IP is 10.233.4.2

# nixos-container start flake-test

To check whether the container works, let’s try to connect to it:

$ curl http://flake-test/
<html><body><h1>It works!</h1></body></html>

As an aside, if you just want to build the container without the nixos-container command, you can do so as follows:

$ nix build /path/to/my-flake#nixosConfigurations.container.config.system.build.toplevel

Note that system.build.toplevel is an internal NixOS option that evaluates to the “system” derivation that commands like nixos-rebuild, nixos-install and nixos-container build and activate. The symlink /run/current-system points to the output of this derivation.

Hermetic evaluation

One big difference between “regular” NixOS systems and flake-based NixOS systems is that the latter record the Git revisions from which they were built. We can query this as follows:

# nixos-container run flake-test -- nixos-version --json
{"configurationRevision":"9190c396f4dcfc734e554768c53a81d1c231c6a7"
,"nixosVersion":"20.03.20200622.13c15f2"
,"nixpkgsRevision":"13c15f26d44cf7f54197891a6f0c78ce8149b037"}

Here, configurationRevision is the Git revision of the repository /path/to/my-flake. Because evaluation is hermetic, and the lock file locks all flake inputs such as nixpkgs, knowing the revision 9190c39… allows you to completely reconstruct this configuration at a later point in time. For example, if you want to deploy this particular configuration to a container, you can do:

# nixos-container update flake-test \
    --flake /path/to/my-flake?rev=9190c396f4dcfc734e554768c53a81d1c231c6a7

Dirty configurations

It’s not required that you commit all changes to a configuration before deploying it. For example, if you change the adminAddr line in flake.nix to

adminAddr = "rick@example.org";

and redeploy the container, you will get:

# nixos-container update flake-test
warning: Git tree '/path/to/my-flake' is dirty
...
reloading container...

and the container will no longer have a configuration Git revision:

# nixos-container run flake-test -- nixos-version --json | jq .configurationRevision
null

While this may be convenient for testing, in production we really want to ensure that systems are deployed from clean Git trees. One way is to disallow dirty trees on the command line:

# nixos-container update flake-test --no-allow-dirty
error: --- Error -------------------- nix
Git tree '/path/to/my-flake' is dirty

Another is to require a clean Git tree in flake.nix, for instance by adding a check to the definition of system.configurationRevision:

system.configurationRevision =
  if self ? rev
  then self.rev
  else throw "Refusing to build from a dirty Git tree!";

Adding modules from third-party flakes

One of the main goals of flake-based NixOS is to make it easier to use packages and modules that are not included in the nixpkgs repository. As an example, we’ll add Hydra (a continuous integration server) to our container.

Here’s how we add it to our container. We specify it as an additional input:

  inputs.hydra.url = "github:NixOS/hydra";

and as a corresponding function argument to the outputs function:

  outputs = { self, nixpkgs, hydra }: {

Finally, we enable the NixOS module provided by the hydra flake:

      modules =
        [ hydra.nixosModules.hydraTest

          ({ pkgs, ... }: {
            ... our own configuration ...

            # Hydra runs on port 3000 by default, so open it in the firewall.
            networking.firewall.allowedTCPPorts = [ 3000 ];
          })
        ];

Note that we can discover the name of this module by using nix flake show:

$ nix flake show github:NixOS/hydra
github:NixOS/hydra/d0deebc4fc95dbeb0249f7b774b03d366596fbed
├───…
├───nixosModules
│   ├───hydra: NixOS module
│   ├───hydraProxy: NixOS module
│   └───hydraTest: NixOS module
└───overlay: Nixpkgs overlay

After committing this change and running nixos-container update, we can check whether hydra is working in the container by visiting http://flake-test:3000/ in a web browser.

Working with lock files

There are a few command line flags accepted by nix, nixos-rebuild and nixos-container that make updating lock file more convenient. A very common action is to update a flake input to the latest version; for example,

$ nixos-container update flake-test --update-input nixpkgs --commit-lock-file

updates the nixpkgs input to the latest revision on the nixos-20.03 branch, and commits the new lock file with a commit message that records the input change.

A useful flag during development is --override-input, which allows you to point a flake input to another location, completely overriding the input location specified by flake.nix. For example, this is how you can build the container against a local Git checkout of Hydra:

$ nixos-container update flake-test --override-input hydra /path/to/my/hydra

Adding overlays from third-party flakes

Similarly, we can add Nixpkgs overlays from other flakes. (Nixpkgs overlays add or override packages in the pkgs set.) For example, here is how you add the overlay provided by the nix flake:

  outputs = { self, nixpkgs, nix }: {
    nixosConfigurations.container = nixpkgs.lib.nixosSystem {
      ...
      modules =
        [
          ({ pkgs, ... }: {
            nixpkgs.overlays = [ nix.overlay ];
            ...
          })
        ];
    };
  };
}

Using nixos-rebuild

Above we saw how to manage NixOS containers using flakes. Managing “real” NixOS systems works much the same, except using nixos-rebuild instead of nixos-container. For example,

# nixos-rebuild switch --flake /path/to/my-flake#my-machine

builds and activates the configuration specified by the flake output nixosConfigurations.my-machine. If you omit the name of the configuration (#my-machine), nixos-rebuild defaults to using the current host name.

Pinning Nixpkgs

It’s often convenient to pin the nixpkgs flake to the exact version of nixpkgs used to build the system. This ensures that commands like nix shell nixpkgs#<package> work more efficiently since many or all of the dependencies of <package> will already be present. Here is a bit of NixOS configuration that pins nixpkgs in the system-wide flake registry:

nix.registry.nixpkgs.flake = nixpkgs;

Note that this only affects commands that reference nixpkgs without further qualifiers; more specific flake references like nixpkgs/nixos-20.03 or nixpkgs/348503b6345947082ff8be933dda7ebeddbb2762 are unaffected.

Conclusion

In this blog post we saw how Nix flakes make NixOS configurations hermetic and reproducible. In a future post, we’ll show how we can do the same for cloud deployments using NixOps.

Acknowledgment: The development of flakes was partially funded by Target Corporation.

Application domains

Nix is a powerful package manager for Linux and other Unix systems that makes package management reliable and reproducible. Share your development and build environments across different machines.

Docker is an open platform for developing, shipping, and running applications.

• Nix's chief responsibility is as its description implies: package management and provides a collection of software tools that automates the process of installing, upgrading, configuring, and removing computer programs for a computer's operating system in a consistent manner.
  
  There are two properties that set Nix apart from most other package management solutions. First, Nix is also a source-based package manager -- it can be used as a tool to construct packages from source code and their dependencies, by invoking build scripts in "pure build environments".
  
  Moreover, it borrows concepts from purely functional programming languages to make deployments reproducible, reliable and efficient.
• Docker's chief responsibility is much broader than package management -- Docker facilitates full process/service life-cycle management. Package management can be considered to be a sub problem of this domain, as I will explain later in this blog post.

Nix concepts

with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "file-5.38";

  src = fetchurl {
    url = "ftp://ftp.astron.com/pub/file/file-5.38.tar.gz";
    sha256 = "0d7s376b4xqymnrsjxi3nsv3f5v89pzfspzml2pcajdk5by2yg2r";
  };

  buildInputs = [ zlib ];

  meta = {
    homepage = https://darwinsys.com/file;
    description = "A program that shows the type of files";
  };
}

• The name parameter provides the package name.
• The src parameter invokes the fetchurl function that specifies where to download the source tarball from.
• buildInputs refers to the build-time dependencies that the package needs. The file package only uses one dependency: zlib that provides deflate compression support.
  
  The buildInputs parameter is used to automatically configure the build environment in such a way that zlib can be found as a library dependency by the build script.
• The meta parameter specifies the package's meta data. Meta data is used by Nix to provide information about the package, but it is not used by the build script.

$ nix-build file.nix

/nix/store/6rcg0zgqyn2v1ypd46hlvngaf5lgqk9g-file-5.38

• Each build will run as an unprivileged user, that do not have any write access to any directory but its own build directory and the designated output Nix store paths.
• On Linux, optionally a build can run in a chroot environment, that completely disables access to all global directories in the build process. In addition, all Nix store paths of all dependencies will be bind mounted, preventing the build process to still access undeclared dependencies in the Nix store (changes will be slim that you encounter such a build, but still...)
• On Linux kernels that support namespaces, the Nix build environment will use them to improve build purity.
  
  The network namespace helps the Nix builder to prevent a build process from accessing the network -- when a build process downloads an undeclared dependency from a remote location, we cannot be sure that we get a predictable result.
  
  In Nix, only builds that are so-called fixed output derivations (whose output hashes need to be known in advance) are allowed to download files from remote locations, because their output results can be verified.
  
  (As a sidenote: namespaces are also intensively used by Docker containers, as I will explain in the next section.)
• On macOS, builds can optionally be executed in an app sandbox, that can also be used to restrict access to various kinds of shared resources, such as network access.

$ nix-store -qR /nix/store/6rcg0zgqyn2v1ypd46hlvngaf5lgqk9g-file-5.38
/nix/store/y8n2b9nwjrgfx3kvi3vywvfib2cw5xa6-libunistring-0.9.10
/nix/store/fhg84pzckx2igmcsvg92x1wpvl1dmybf-libidn2-2.3.0
/nix/store/bqbg6hb2jsl3kvf6jgmgfdqy06fpjrrn-glibc-2.30
/nix/store/5x6l9xm5dp6v113dpfv673qvhwjyb7p5-zlib-1.2.11
/nix/store/6rcg0zgqyn2v1ypd46hlvngaf5lgqk9g-file-5.38

$ nix-store -qR /nix/store/bzm0mszhvbr6hp4gmar4czsn52hz07q1-cpio-2.13
/nix/store/y8n2b9nwjrgfx3kvi3vywvfib2cw5xa6-libunistring-0.9.10
/nix/store/fhg84pzckx2igmcsvg92x1wpvl1dmybf-libidn2-2.3.0
/nix/store/bqbg6hb2jsl3kvf6jgmgfdqy06fpjrrn-glibc-2.30
/nix/store/bzm0mszhvbr6hp4gmar4czsn52hz07q1-cpio-2.13

Docker concepts

When you use Docker, you are creating and using images, containers, networks, volumes, plugins, and other objects.

• Images. The overview page states: "An image is a read-only template with instructions for creating a Docker container.".
  
  To more accurately describe what this means is that images are created from build recipes called Dockerfiles. They produce self-contained root file systems containing all necessary files to run a program, such as binaries, libraries, configuration files etc. The resulting image itself is immutable (read only) and cannot change after it has been built.
• Containers. The overview gives the following description: "A container is a runnable instance of an image".
  
  More specifically, this means that the life-cycle (whether a container is in a started or stopped state) is bound to the life-cycle of a root process, that runs in a (somewhat) isolated environment using the content of a Docker image as its root file system.

• The mount namespace: This is in IMO the most important name space. After setting up a private mount namespace, every subsequent mount that we make will be visible in the container, but not to other containers/processes that are in a different mount name space.
  
  A private mount namespace is used to mount a new root file system (the contents of the Docker image) with all essential system software and other artifacts to run an application, that is different from the host system's root file system.
• The Process ID (PID) namespace facilitates process isolation. A process/container with a private PID namespace will not be able to see or control the host system's processes (the opposite is actually possible).
• The network namespace separates network interfaces from the host system. In a private network namespace, a container has one or more private network interfaces with their own IP addresses, port assignments and firewall settings.
  
  As a result, a service such as the Apache HTTP server in a Docker container can bind to port 80 without conflicting with another HTTP server that binds to the same port on the host system or in another container instance.
• The Inter-Process Communication (IPC) namespace separates the ability for processes to communicate with each other via the SHM family of functions to establish a range of shared memory between the two processes.
• The UTS namespace isolates kernel and version identifiers.

FROM debian:buster

RUN apt-get update
RUN apt-get install -y apache2
ADD index.html /var/www/html
CMD ["apachectl", "-D", "FOREGROUND"]
EXPOSE 80/tcp

• It takes the debian:buster image from Docker Hub as a base image.
• It updates the Debian package database (apt-get update) and installs the Apache HTTPD server package from the Debian package repository.
• It uploads an example page (index.html) to the document root folder.
• It executes the: apachectl -D FOREGROUND command-line instruction to start the Apache HTTP server in foreground mode. The container's life-cycle is bound to the life-cycle of this foreground process.
• It informs Docker that the container listens to TCP port: 80. Connecting to port 80 makes it possible for a user to retrieve the example index.html page.

$ docker build . -t debian-apache

$ docker history debian-nginx:latest
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
a72c04bd48d6        About an hour ago   /bin/sh -c #(nop)  EXPOSE 80/tcp                0B                  
325875da0f6d        About an hour ago   /bin/sh -c #(nop)  CMD ["apachectl" "-D" "FO…   0B                  
35d9a1dca334        About an hour ago   /bin/sh -c #(nop) ADD file:18aed37573327bee1…   129B                
59ee7771f1bc        About an hour ago   /bin/sh -c apt-get install -y apache2           112MB               
c355fe9a587f        2 hours ago         /bin/sh -c apt-get update                       17.4MB              
ae8514941ea4        33 hours ago        /bin/sh -c #(nop)  CMD ["bash"]                 0B                  
<missing>           33 hours ago        /bin/sh -c #(nop) ADD file:89dfd7d3ed77fd5e0…   114MB

FROM debian:buster

RUN apt-get update
RUN apt-get install -y nginx
ADD nginx.conf /etc
ADD index.html /var/www
CMD ["nginx", "-g", "daemon off;", "-c", "/etc/nginx.conf"]
EXPOSE 80/tcp

$ docker history debian-nginx:latest
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
b7ae6f38ae77        2 hours ago         /bin/sh -c #(nop)  EXPOSE 80/tcp                0B                  
17027888ce23        2 hours ago         /bin/sh -c #(nop)  CMD ["nginx" "-g" "daemon…   0B                  
41a50a3fa73c        2 hours ago         /bin/sh -c #(nop) ADD file:18aed37573327bee1…   129B                
0f5b2fdcb207        2 hours ago         /bin/sh -c #(nop) ADD file:f18afd18cfe2728b3…   189B                
e49bbb46138b        2 hours ago         /bin/sh -c apt-get install -y nginx             64.2MB              
c355fe9a587f        2 hours ago         /bin/sh -c apt-get update                       17.4MB              
ae8514941ea4        33 hours ago        /bin/sh -c #(nop)  CMD ["bash"]                 0B                  
<missing>           33 hours ago        /bin/sh -c #(nop) ADD file:89dfd7d3ed77fd5e0…   114MB

Some common misconceptions

A comparison of use cases

Managing services

$ docker run -p 8080:80 --name nginx-container -it debian-nginx

http://localhost:8080

$ docker rm nginx-container

Experimenting with packages

$ docker pull debian:buster

$ docker run --name myexperiment -it debian:buster /bin/sh
$ apt-get update
$ apt-get install -y file
# file --version
file-5.22
magic file from /etc/magic:/usr/share/misc/magic

FROM debian:buster

RUN apt-get update
RUN apt-get install -y file

$ docker build . -t file-experiment

$ docker run --name myexperiment -it debian:buster /bin/sh

$ docker run --name myexperiment -it file-experiment /bin/sh
# file --version
file-5.22
magic file from /etc/magic:/usr/share/misc/magic

$ nix-shell -p file
$ file --version
file-5.39
magic file from /nix/store/j4jj3slm15940mpmympb0z99a2ghg49q-file-5.39/share/misc/magic

Building development projects/arbitrary packages

#!/bin/bash -e

mkdir -p /build
cd /build

wget ftp://ftp.astron.com/pub/file/file-5.38.tar.gz

tar xfv file-5.38.tar.gz
cd file-5.38
./configure --prefix=/opt/file
make
make install

tar cfvz /out/file-5.38-binaries.tar.gz /opt/file

• Create a dedicated build directory.
• Download the source tarball from the FTP server
• Unpack the tarball
• Execute the standard GNU Autotools build procedure: ./configure; make; make install and install the binaries in an isolated folder (/opt/file).
• Create a binary tarball from the /opt/file folder and store it in the /out directory (that is a volume shared between the container and the host system).

FROM debian:buster

RUN apt-get update
RUN apt-get install -y wget gcc make libz-dev
ADD ./build.sh /
CMD /build.sh

$ docker build . -t buildenv

$ docker run -v $(pwd)/out:/out --rm -t buildenv

ls out/
file-5.38-binaries.tar.gz

• In the build script, we download the source tarball without checking its integrity. This might cause an impurity, because the tarball on the remote server could change (this could happen for non-mallicious as well as mallicous reasons).
• While running the build, we have unrestricted network access. The build script might unknowingly download all kinds of undeclared/unknown dependencies from external sites whose results are not deterministic.
• We do not reset any timestamps -- as a result, when performing the same build twice in a row, the second result might be slightly different because of the timestamps integrated in the build product.

Combined use cases

Experimenting with the Nix package manager in a Docker container

$ docker pull nixos/nix

$ docker run -it nixos/nix

$ nix-channel --add https://nixos.org/channels/nixpkgs-unstable nixpkgs
$ nix-channel --update
$ nix-env -f '<nixpkgs>' -iA file
$ file --version
file-5.39
magic file from /nix/store/bx9l7vrcb9izgjgwkjwvryxsdqdd5zba-file-5.39/share/misc/magic

Using the Nix package manager to deliver the required packages to construct an image

FROM nixos/nix

RUN nix-channel --add https://nixos.org/channels/nixpkgs-unstable nixpkgs
RUN nix-channel --update
RUN nix-env -f '<nixpkgs>' -iA nginx

RUN mkdir -p /var/log/nginx /var/cache/nginx /var/www
ADD nginx.conf /etc
ADD index.html /var/www

CMD ["nginx", "-g", "daemon off;", "-c", "/etc/nginx.conf"]
EXPOSE 80/tcp

Using Nix to build Docker images

with import <nixpkgs> {};

dockerTools.buildImage {
  name = "nginxexp";
  tag = "test";

  contents = nginx;

  runAsRoot = ''
    ${dockerTools.shadowSetup}
    groupadd -r nogroup
    useradd -r nobody -g nogroup -d /dev/null
    mkdir -p /var/log/nginx /var/cache/nginx /var/www
    cp ${./index.html} /var/www/index.html
  '';

  config = {
    Cmd = [ "${nginx}/bin/nginx" "-g" "daemon off;" "-c" ./nginx.conf ];
    Expose = {
      "80/tcp" = {};
    };
  };
}

• The name of the image is: nginxexp using the tag: test.
• The contents parameter specifies all Nix packages that should be installed in the Docker image.
• The runAsRoot refers to a script that runs as root user in a QEMU virtual machine. This virtual machine is used to provide the dynamic parts of a Docker image, setting up user accounts and configuring the state of the Nginx service.
• The config parameter specifies image configuration properties, such as the command to execute and which TCP ports should be exposed.

$ nix-build
/nix/store/qx9cpvdxj78d98rwfk6a5z2qsmqvgzvk-docker-image-nginxexp.tar.gz

$ docker load -i \
  /nix/store/qx9cpvdxj78d98rwfk6a5z2qsmqvgzvk-docker-image-nginxexp.tar.gz

$ docker run -p 8080:80/tcp -it nginxexp:test

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
nginxexp            test                cde8298f025f        50 years ago        61MB

• The first odd property is that the overview says that the image that was created 50 years ago. This is explainable: to make Nix builds pure and deterministic, time stamps are typically reset to 1 second after the epoch (Januarty 1st 1970), to ensure that we always get the same bit-identical build result.
• The second property is the size of the image: 61MB is considerably smaller than our Debian-based Docker image.
  
  To give you a comparison: the docker history command-line invocation (shown earlier in this blog post) that displays the layers of which the Debian-based Nginx image consists, shows that the base Linux distribution image consumes 114 MB, the update layer 17.4 MB and the layer that provides the Nginx package is 64.2 MB.

with import <nixpkgs> {};

dockerTools.buildLayeredImage {
  name = "nginxexp";
  tag = "test";

  contents = nginx;

  maxLayers = 100;

  extraCommands = ''
    mkdir -p var/log/nginx var/cache/nginx var/www
    cp ${./index.html} var/www/index.html
  '';

  config = {
    Cmd = [ "${nginx}/bin/nginx" "-g" "daemon off;" "-c" ./nginx.conf ];
    Expose = {
      "80/tcp" = {};
    };
  };
}

$ docker load -i $(nix-build layered.nix)

$ docker history nginxexp:test
IMAGE               CREATED             CREATED BY          SIZE                COMMENT
b91799a04b99        50 years ago                            1.47kB              store paths: ['/nix/store/snxpdsksd4wxcn3niiyck0fry3wzri96-nginxexp-customisation-layer']
<missing>           50 years ago                            200B                store paths: ['/nix/store/6npz42nl2hhsrs98bq45aqkqsndpwvp1-nginx-root.conf']
<missing>           50 years ago                            1.79MB              store paths: ['/nix/store/qsq6ni4lxd8i4g9g4dvh3y7v1f43fqsp-nginx-1.18.0']
<missing>           50 years ago                            71.3kB              store paths: ['/nix/store/n14bjnksgk2phl8n69m4yabmds7f0jj2-source']
<missing>           50 years ago                            166kB               store paths: ['/nix/store/jsqrk045m09i136mgcfjfai8i05nq14c-source']
<missing>           50 years ago                            1.3MB               store paths: ['/nix/store/4w2zbpv9ihl36kbpp6w5d1x33gp5ivfh-source']
<missing>           50 years ago                            492kB               store paths: ['/nix/store/kdrdxhswaqm4dgdqs1vs2l4b4md7djma-pcre-8.44']
<missing>           50 years ago                            4.17MB              store paths: ['/nix/store/6glpgx3pypxzb09wxdqyagv33rrj03qp-openssl-1.1.1g']
<missing>           50 years ago                            385kB               store paths: ['/nix/store/7n56vmgraagsl55aarx4qbigdmcvx345-libxslt-1.1.34']
<missing>           50 years ago                            324kB               store paths: ['/nix/store/1f8z1lc748w8clv1523lma4w31klrdpc-geoip-1.6.12']
<missing>           50 years ago                            429kB               store paths: ['/nix/store/wnrjhy16qzbhn2qdxqd6yrp76yghhkrg-gd-2.3.0']
<missing>           50 years ago                            1.22MB              store paths: ['/nix/store/hqd0i3nyb0717kqcm1v80x54ipkp4bv6-libwebp-1.0.3']
<missing>           50 years ago                            327kB               store paths: ['/nix/store/79nj0nblmb44v15kymha0489sw1l7fa0-fontconfig-2.12.6-lib']
<missing>           50 years ago                            1.7MB               store paths: ['/nix/store/6m9isbbvj78pjngmh0q5qr5cy5y1kzyw-libxml2-2.9.10']
<missing>           50 years ago                            580kB               store paths: ['/nix/store/2xmw4nxgfximk8v1rkw74490rfzz2gjp-libtiff-4.1.0']
<missing>           50 years ago                            404kB               store paths: ['/nix/store/vbxifzrl7i5nvh3h505kyw325da9k47n-giflib-5.2.1']
<missing>           50 years ago                            79.8kB              store paths: ['/nix/store/jc5bd71qcjshdjgzx9xdfrnc9hsi2qc3-fontconfig-2.12.6']
<missing>           50 years ago                            236kB               store paths: ['/nix/store/9q5gjvrabnr74vinmjzkkljbpxi8zk5j-expat-2.2.8']
<missing>           50 years ago                            482kB               store paths: ['/nix/store/0d6vl8gzwqc3bdkgj5qmmn8v67611znm-xz-5.2.5']
<missing>           50 years ago                            6.28MB              store paths: ['/nix/store/rmn2n2sycqviyccnhg85zangw1qpidx0-gcc-9.3.0-lib']
<missing>           50 years ago                            1.98MB              store paths: ['/nix/store/fnhsqz8a120qwgyyaiczv3lq4bjim780-freetype-2.10.2']
<missing>           50 years ago                            757kB               store paths: ['/nix/store/9ifada2prgfg7zm5ba0as6404rz6zy9w-dejavu-fonts-minimal-2.37']
<missing>           50 years ago                            1.51MB              store paths: ['/nix/store/yj40ch9rhkqwyjn920imxm1zcrvazsn3-libjpeg-turbo-2.0.4']
<missing>           50 years ago                            79.8kB              store paths: ['/nix/store/1lxskkhsfimhpg4fd7zqnynsmplvwqxz-bzip2-1.0.6.0.1']
<missing>           50 years ago                            255kB               store paths: ['/nix/store/adldw22awj7n65688smv19mdwvi1crsl-libpng-apng-1.6.37']
<missing>           50 years ago                            123kB               store paths: ['/nix/store/5x6l9xm5dp6v113dpfv673qvhwjyb7p5-zlib-1.2.11']
<missing>           50 years ago                            30.9MB              store paths: ['/nix/store/bqbg6hb2jsl3kvf6jgmgfdqy06fpjrrn-glibc-2.30']
<missing>           50 years ago                            209kB               store paths: ['/nix/store/fhg84pzckx2igmcsvg92x1wpvl1dmybf-libidn2-2.3.0']
<missing>           50 years ago                            1.63MB              store paths: ['/nix/store/y8n2b9nwjrgfx3kvi3vywvfib2cw5xa6-libunistring-0.9.10']

Conclusion

Related work

Future work

In this post I’m going to show how to setup, with Terraform, a Buildkite-based CI using your own workers that run on GCP. For reference, the complete Terraform configuration for this post is available in this repository.

• The setup gives you complete control on how fast your worker are.
• The workers come with Nix pre-installed, so you won’t need to spend time downloading the same docker container again and again on every push as would usually happen with most cloud CI providers.
• The workers come with a distributed Nix cache set up. So authors of CI scripts won’t have to bother about caching at all.

Secrets

We are going to need to import two secret resources:

resource "secret_resource" "buildkite_agent_token" {}
resource "secret_resource" "nix_signing_key" {}

To initialize the resources, execute the following from the root directory of your project:

$ terraform import secret_resource.<name> <value>

where:

• buildkite_agent_token is obtained from the Buildkite site.

• nix_signing_key can be generated by running:

  nix-store --generate-binary-cache-key <your-key-name> key.private key.public

  The key.private file will contain the value for the signing key. I’ll explain later in the post how to use the contents of the key.public file.

Custom NixOS image

The next step is to use the nixos_image_custom module to create a NixOS image with custom configuration.

resource "google_storage_bucket" "nixos_image" {
  name     = "buildkite-nixos-image-bucket-name"
  location = "EU"
}

module "nixos_image_custom" {
  source      = "git::https://github.com/tweag/terraform-nixos.git//google_image_nixos_custom?ref=40fedb1fae7df5bd7ad9defdd71eb06b7252810f"
  bucket_name = "${google_storage_bucket.nixos_image.name}"
  nixos_config = "${path.module}/nixos-config.nix"
}

The snippet above first creates a bucket nixos_image where the generated image will be uploaded, then it uses the nixos_image_custom module, which handles generation of the image using the configuration from the nixos-config.nix file. The file is assumed to be in the same directory as the Terraform configuration, hence ${path.module}/.

Service account and cache bucket

To control access to different resources we will also need a service account:

resource "google_service_account" "buildkite_agent" {
  account_id   = "buildkite-agent"
  display_name = "Buildkite agent"
}

We can use it to set access permissions for the storage bucket that will contain the Nix cache:

resource "google_storage_bucket" "nix_cache_bucket" {
  name     = "nix-cache-bucket-name"
  location = "EU"
  force_destroy = true
  retention_policy {
    retention_period = 7889238 # three months
  }
}

resource "google_storage_bucket_iam_member" "buildkite_nix_cache_writer" {
  bucket = "${google_storage_bucket.nix_cache_bucket.name}"
  role = "roles/storage.objectAdmin"
  member = "serviceAccount:${google_service_account.buildkite_agent.email}"
}

resource "google_storage_bucket_iam_member" "buildkite_nix_cache_reader" {
  bucket = "${google_storage_bucket.nix_cache_bucket.name}"
  role   = "roles/storage.objectViewer"
  member = "allUsers"
}

The bucket is configured to automatically delete objects that are older than 3 months. We give the service account the ability to write to and read from the bucket (the roles/storage.objectAdmin role). The rest of the world gets the ability to read from the bucket (the roles/storage.objectViewer role).

NixOS configuration

Here is the content of my nixos-config.nix. This NixOS configuration can serve as a starting point for writing your own. The numbered points refer to the notes below.

{ modulesPath, pkgs, ... }:
{
  imports = [
    "${modulesPath}/virtualisation/google-compute-image.nix"
  ];
  virtualisation.googleComputeImage.diskSize = 3000;
  virtualisation.docker.enable = true;

  services = {
    buildkite-agents.agent = {
      enable = true;
      extraConfig = ''
      tags-from-gcp=true
      '';
      tags = {
        os = "nixos";
        nix = "true";
      };
      tokenPath = "/run/keys/buildkite-agent-token"; # (1)
      runtimePackages = with pkgs; [
        bash
        curl
        gcc
        gnutar
        gzip
        ncurses
        nix
        python3
        xz
        # (2) extend as necessary
      ];
    };
    nix-store-gcs-proxy = {
      nix-cache-bucket-name = { # (3)
        address = "localhost:3000";
      };
    };
  };

  nix = {
    binaryCaches = [
      "https://cache.nixos.org/"
      "https://storage.googleapis.com/nix-cache-bucket-name" # (4)
    ];
    binaryCachePublicKeys = [
      "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
      "<insert your public signing key here>" # (5)
    ];
    extraOptions = ''
      post-build-hook = /etc/nix/upload-to-cache.sh # (6)
    '';
  };

  security.sudo.enable = true;
  services.openssh.passwordAuthentication = false;
  security.sudo.wheelNeedsPassword = false;
}

Notes:

1. This file will be created later by the startup script (see below).
2. The collection of packages that are available to the Buildkite script can be edited here.
3. Replace nix-cache-bucket-name by the name of the bucket used for the Nix cache.
4. Similarly to (3) replace nix-cache-bucket-name in the URL.
5. Insert the contents of the key.public file you generated earlier.
6. The file will be created later by the startup script.

Compute instances and startup script

The following snippet sets up an instance group manager which controls multiple (3 in this example) Buildkite agents. The numbered points refer to the notes below.

data "template_file" "buildkite_nixos_startup" { # (1)
  template = "${file("${path.module}/files/buildkite_nixos_startup.sh")}"

  vars = {
    buildkite_agent_token = "${secret_resource.buildkite_agent_token.value}"
    nix_signing_key = "${secret_resource.nix_signing_key.value}"
  }
}

resource "google_compute_instance_template" "buildkite_nixos" {
  name_prefix  = "buildkite-nixos-"
  machine_type = "n1-standard-8"

  disk {
    boot         = true
    disk_size_gb = 100
    source_image = "${module.nixos_image_custom.self_link}"
  }

  metadata_startup_script = "${data.template_file.buildkite_nixos_startup.rendered}"

  network_interface {
    network = "default"

    access_config = {}
  }

  metadata {
    enable-oslogin = true
  }

  service_account {
    email = "${google_service_account.buildkite_agent.email}"

    scopes = [
      "compute-ro",
      "logging-write",
      "storage-rw",
    ]
  }

  scheduling {
    automatic_restart   = false
    on_host_maintenance = "TERMINATE"
    preemptible         = true # (2)
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "google_compute_instance_group_manager" "buildkite_nixos" {
  provider           = "google-beta"
  name               = "buildkite-nixos"
  base_instance_name = "buildkite-nixos"
  target_size        = "3" # (3)
  zone               = "<your-zone>" # (4)

  version {
    name              = "buildkite_nixos"
    instance_template = "${google_compute_instance_template.buildkite_nixos.self_link}"
  }

  update_policy {
    type                  = "PROACTIVE"
    minimal_action        = "REPLACE"
    max_unavailable_fixed = 1
  }
}

Notes:

1. The file files/buildkite_nixos_startup.sh is shown below.
2. Because of the remote Nix cache, the nodes can be preemptible (short-lived, never lasting longer than 24 hours), which results in much lower GCP costs.
3. Changing target_size allows you to scale the system. This is the number of instances that are controlled by the instance group manager.
4. Insert your desired zone here.

Finally, here is the startup script:

# workaround https://github.com/NixOS/nixpkgs/issues/42344
chown root:keys /run/keys
chmod 750 /run/keys
umask 037
echo "${buildkite_agent_token}" > /run/keys/buildkite-agent-token
chown root:keys /run/keys/buildkite-agent-token
umask 077
echo '${nix_signing_key}' > /run/keys/nix_signing_key
chown root:keys /run/keys/nix-signing-key

cat <<EOF > /etc/nix/upload-to-cache.sh
#!/bin/sh

set -eu
set -f # disable globbing
export IFS=' '

echo "Uploading paths" $OUT_PATHS
exec nix copy --to http://localhost:3000?secret-key=/run/keys/nix_signing_key \$OUT_PATHS
EOF
chmod +x /etc/nix/upload-to-cache.sh

This script uses the Nix post build hook approach for uploading to the cache without polluting the CI script.

Conclusion

The setup allows us to run Nix builds in an environment where Nix tooling is available. It also provides a remote Nix cache which does not require that the authors of CI scripts set it up or, even, be aware of it at all. We use this setup on many of Tweag’s projects and found that both mental and performance overheads are minimal. A typical CI script looks like this:

steps:
  - label: Build and test
    command: nix-build -A distributed-closure --no-out-link

Builds with up-to-date cache that does not cause re-builds may finish in literally 1 second.

Nix evaluation is often quite slow. In this blog post, we’ll have a look at a nice advantage of the hermetic evaluation model enforced by flakes: the ability to cache evaluation results reliably. For a short introduction to flakes, see our previous blog post.

Why Nix evaluation is slow

Nix uses a simple, interpreted, purely functional language to describe package dependency graphs and NixOS system configurations. So to get any information about those things, Nix first needs to evaluate a substantial Nix program. This involves parsing potentially thousands of .nix files and running a Turing-complete language.

For example, the command nix-env -qa shows you which packages are available in Nixpkgs. But this is quite slow and takes a lot of memory:

$ command time nix-env -qa | wc -l
5.09user 0.49system 0:05.59elapsed 99%CPU (0avgtext+0avgdata 1522792maxresident)k
28012

Evaluating individual packages or configurations can also be slow. For example, using nix-shell to enter a development environment for Hydra, we have to wait a bit, even if all dependencies are present in the Nix store:

$ command time nix-shell --command 'exit 0'
1.34user 0.18system 0:01.69elapsed 89%CPU (0avgtext+0avgdata 434808maxresident)k

That might be okay for occasional use but a wait of one or more seconds may well be unacceptably slow in scripts.

Note that the evaluation overhead is completely independent from the time it takes to actually build or download a package or configuration. If something is already present in the Nix store, Nix won’t build or download it again. But it still needs to re-evaluate the Nix files to determine which Nix store paths are needed.

Caching evaluation results

So can’t we speed things up by caching evaluation results? After all, the Nix language is purely functional, so it seems that re-evaluation should produce the same result, every time. Naively, maybe we can keep a cache that records that attribute A of file X evaluates to derivation D (or whatever metadata we want to cache). Unfortunately, it’s not that simple; cache invalidation is, after all, one of the only two hard problems in computer science.

The reason this didn’t work is that in the past Nix evaluation was not hermetic. For example, a .nix file can import other Nix files through relative or absolute paths (such as ~/.config/nixpkgs/config.nix for Nixpkgs) or by looking them up in the Nix search path ($NIX_PATH). So unless we perfectly keep track of all the files used during evaluation, a cached result might be inconsistent with the current input.

(As an aside: for a while, Nix has had an experimental replacement for nix-env -qa called nix search, which used an ad hoc cache for package metadata. It had exactly this cache invalidation problem: it wasn’t smart enough to figure out whether its cache was up to date with whatever revision of Nixpkgs you were using. So it had a manual flag --update-cache to allow the user to force cache invalidation.)

Flakes to the rescue

Flakes solve this problem by ensuring fully hermetic evaluation. When you evaluate an output attribute of a particular flake (e.g. the attribute defaultPackage.x86_64-linux of the dwarffs flake), Nix disallows access to any files outside that flake or its dependencies. It also disallows impure or platform-dependent features such as access to environment variables or the current system type.

This allows the nix command to aggressively cache evaluation results without fear of cache invalidation problems. Let’s see this in action by running Firefox from the nixpkgs flake. If we do this with an empty evaluation cache, Nix needs to evaluate the entire dependency graph of Firefox, which takes a quarter of a second:

$ command time nix shell nixpkgs#firefox -c firefox --version
Mozilla Firefox 75.0
0.26user 0.05system 0:00.39elapsed 82%CPU (0avgtext+0avgdata 115224maxresident)k

But if we do it again, it’s almost instantaneous (and takes less memory):

$ command time nix shell nixpkgs#firefox -c firefox --version
Mozilla Firefox 75.0
0.01user 0.01system 0:00.03elapsed 93%CPU (0avgtext+0avgdata 25840maxresident)k

The cache is implemented using a simple SQLite database that stores the values of flake output attributes. After the first command above, the cache looks like this:

$ sqlite3 ~/.cache/nix/eval-cache-v1/302043eedfbce13ecd8169612849f6ce789c26365c9aa0e6cfd3a772d746e3ba.sqlite .dump
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE Attributes (
    parent      integer not null,
    name        text,
    type        integer not null,
    value       text,
    primary key (parent, name)
);
INSERT INTO Attributes VALUES(0,'',0,NULL);
INSERT INTO Attributes VALUES(1,'packages',3,NULL);
INSERT INTO Attributes VALUES(1,'legacyPackages',0,NULL);
INSERT INTO Attributes VALUES(3,'x86_64-linux',0,NULL);
INSERT INTO Attributes VALUES(4,'firefox',0,NULL);
INSERT INTO Attributes VALUES(5,'type',2,'derivation');
INSERT INTO Attributes VALUES(5,'drvPath',2,'/nix/store/7mz8pkgpl24wyab8nny0zclvca7ki2m8-firefox-75.0.drv');
INSERT INTO Attributes VALUES(5,'outPath',2,'/nix/store/5x1i2gp8k95f2mihd6aj61b5lydpz5dy-firefox-75.0');
INSERT INTO Attributes VALUES(5,'outputName',2,'out');
COMMIT;

In other words, the cache stores all the attributes that nix shell had to evaluate, in particular legacyPackages.x86_64-linux.firefox.{type,drvPath,outPath,outputName}. It also stores negative lookups, that is, attributes that don’t exist (such as packages).

The name of the SQLite database, 302043eedf….sqlite in this example, is derived from the contents of the top-level flake. Since the flake’s lock file contains content hashes of all dependencies, this is enough to efficiently and completely capture all files that might influence the evaluation result. (In the future, we’ll optimise this a bit more: for example, if the flake is a Git repository, we can simply use the Git revision as the cache name.)

The nix search command has been updated to use the new evaluation cache instead of its previous ad hoc cache. For example, searching for Blender is slow the first time:

$ command time nix search nixpkgs blender
* legacyPackages.x86_64-linux.blender (2.82a)
  3D Creation/Animation/Publishing System
5.55user 0.63system 0:06.17elapsed 100%CPU (0avgtext+0avgdata 1491912maxresident)k

but the second time it is pretty fast and uses much less memory:

$ command time nix search nixpkgs blender
* legacyPackages.x86_64-linux.blender (2.82a)
  3D Creation/Animation/Publishing System
0.41user 0.00system 0:00.42elapsed 99%CPU (0avgtext+0avgdata 21100maxresident)k

The evaluation cache at this point is about 10.9 MiB in size. The overhead for creating the cache is fairly modest: with the flag --no-eval-cache, nix search nixpkgs blender takes 4.9 seconds.

Caching and store derivations

There is only one way in which cached results can become “stale”, in a way. Nix evaluation produces store derivations such as /nix/store/7mz8pkgpl24wyab8nny0zclvca7ki2m8-firefox-75.0.drv as a side effect. (.drv files are essentially a serialization of the dependency graph of a package.) These store derivations may be garbage-collected. In that case, the evaluation cache points to a path that no longer exists. Thus, Nix checks whether the .drv file still exist, and if not, falls back to evaluating normally.

Future improvements

Currently, the evaluation cache is only created and used locally. However, Nix could automatically download precomputed caches, similar to how it has a binary cache for the contents of store paths. That is, if we need a cache like 302043eedf….sqlite, we could first check if it’s available on cache.nixos.org and if so fetch it from there. In this way, when we run a command such as nix shell nixpkgs#firefox, we could even avoid the need to fetch the actual source of the flake!

Another future improvement is to populate and use the cache in the evaluator itself. Currently the cache is populated and cached in the user interface (that is, the nix command). The command nix shell nixpkgs#firefox will create a cache entry for firefox, but not for the dependencies of firefox; thus a subsequent nix shell nixpkgs#thunderbird won’t see a speed improvement even though it shares most of its dependencies. So it would be nice if the evaluator had knowledge of the evaluation cache. For example, the evaluation of thunks that represent attributes like nixpkgs.legacyPackages.x86_64-linux.<package name> could check and update the cache.

As of today, nixbuild.net will automatically select resources (CPU count and memory amount) for builds submitted to it. Based on historic build data, nixbuild.net calculates a resource allocation that will make your build as performant as possible, while wasting minimal CPU time. This means nixbuild.net users get faster and cheaper builds, while also taking away the user’s burden of figuring out what resource settings to use for each individual build.

Previously, all builds were assigned 4 CPUs unless the user configured resource selection differently. However, configuring different resource settings for individual builds was difficult, since Nix has no notion of such settings. Additionally, it is really tricky to know wether a build will gain anything from being allocated many CPUs, or if it just makes the build more expensive. It generally requires the user to try out the build with different settings, which is time-consuming for a single build and almost insurmountable for a large set of builds with different characteristics.

Now, each individual build will be analyzed and can be assigned between 1 and 16 CPUs, depending on how well the build utilizes multiple CPUs. The memory allocation will be adapted to minimize the amount of unused memory.

The automatic resource optimization has been tested both internally and by a selected number of beta users, and the results have been very positive so far. We’re happy to make this feature available to all nixbuild.net users, since it aligns perfectly with the service’s core idea of being simple, cost-effective and performant.

How Does it Work?

The automatic resource optimization works in two steps:

1. When a Nix derivation is submitted to nixbuild.net, we look for similar derivations that have been built on nixbuild.net before. A heuristic approach is used, where derivations are compared based on package names and version numbers. This approach can be improved in the future, by looking at more parts of the derivations, like dependencies and build scripts.

2. A number of the most recent, most similar derivations are selected. We then analyze the build data of those derivations. Since we have developed a secure sandbox specifically for running Nix builds, we’re also able to collect a lot of data about the builds. One metric that is collected is CPU utilization, and that lets us make predictions about how well a build would scale, performance-wise, if it was given more CPUs.

   We also look at metrics about the historic memory usage, and make sure the new build is allocated enough memory.

Reproducible builds and deployments — this is the ambition that Nix proclaims for itself. This ambition comes, however, with a fine print: builds are reproducible only if the original source code still exists. Otherwise, Nix can’t download, build and deploy it. The community maintains binary caches like cache.nixos.org, but these don’t preserve anything — caches are ephemeral. After all, preserving source code is not Nix’s primary mission!

Software Heritage, on the other hand, aspires to preserve software forever. Software Heritage is an independent foundation, grounded in academia, supported by public money, and backed by many of the world’s largest software companies. It can thus reliably pursue the tedious task of collecting and archiving public code and making it available through its website.

Quite naturally, this situation suggested an opportunity for collaboration: can we combine Nix’s reproducible build definitions with Software Heritage’s long-term archive? Will this collaboration bring us closer to the dream of forever replicable and successful builds? We thought that this was an effort worth pursuing, so a partnership started.

This is a story about the challenges that we encountered when trying to make this happen, what we already achieved, and the lessons that we have learned.

Archiving all sources in Nixpkgs

Whenever a Nix user installs a package that isn’t in the binary cache, Nix tries to download the original source and build it from scratch. When these sources don’t exist anymore, or don’t correspond anymore to what Nix expects them to be, the experience can be frustrating — instead of a reproducible build, the user ends up with a “file not found” HTTP error message.

With a long-term source code archive at hand, we could simply redirect the download request to it and move on with the build process. In other words, we would like to make Nix fall back on the Software Heritage archive to download the missing source from there.

For this to work, we must ensure that the source code has been fed to the Software Heritage archive previously. The best way to do it is simply to tell Software Heritage which source code we would like to have archived. Indeed, Software Heritage’s long term goal is to archive all source code produced in the world, and is quite eager to get pointed to locations where to get more!

The first step of this joint effort was to compile a list of all source code URLs required by Nixpkgs, and make them available to Software Heritage. A Nix community project is in charge of generating the list of source code URLs required by a Nixpkgs build, and it is available here. This list indexes every tarball used by the Nixpkgs master branch, with other sources such as patches, JAR’s, or git clones being currently excluded, and hence not archived.

A source list is a simple JSON file that looks like this:

{
  "sources": [
    {
      "type": "url",
      "urls": [
        "https://ftpmirror.gnu.org/hello/hello-2.10.tar.gz",
      ],
      "integrity": "sha256-MeBmE3qWJnbon2nRtlOC3pWn732RS4y5VvQepy4PUWs=",
    },
    ...
  ],
  "version": 1,
  "revision": "cc4e04c26672dd74e5fd0fecb78b435fb55368f7"
}

Here,

• version is the version of this file’s format,
• revision is a Nixpkgs commit ID,
• type is the type of the source. Only the url type is currently supported,
• integrity is the hash of the Nix fixed output derivation.

We then implemented a Software Heritage loader which fetches this JSON file once per day and archives all listed tarballs in a snapshot.

You can see an example snapshot here. This snapshot was created the 03 June 2020, points to the Nixpkgs commit 46fcaf3c8a13f32e2c147fd88f97c4ad2d3b0f27 and contains 21,100 branches, each one corresponding to the tarballs used by this Nixpkgs revision.

Every day, Software Heritage is now archiving more than 21,000 tarballs used by Nixpkgs. But how we can plug them back into Nix?

Falling back on Software Heritage

Two issues need to be addressed to make this happen:

First, we need to find the correct source, the one that Nix tried to download unsuccessfully, in the Software Heritage archive. We cannot simply use the original source URL to query the Software Heritage archive because the URL alone doesn’t identify uniquely the content that is behind it. What if the code that a URL points to has changed over time? To check this, Nix associates a hash with each downloaded artifact. Such a build step is called a fixed output derivation because Nix verifies that the output hash remains immitigably unchanged. It thus ensures that the content that it downloads is always the same, and emits an error otherwise. What we therefore really want is querying Software Heritage with the contents, that is, the hash of the source code artifact itself.

Let’s do it then! Does this mean that the problem is solved? Unfortunately not, and the reason for this are the design decisions behind Nix, Software Heritage and other package repositories, that are not fully compatible. And tarballs play a central role in this.

Content Hash vs Tarball Hash

When Software Heritage archives a tarball, it first unpacks it and then stores all files and directories. This makes it possible to deduplicate files: when two tarballs contain the same file it is only stored once. This is a nice solution, but unfortunately, this new tarball may differ from the original one — for example, if file permissions or timestamps were not preserved in this reassembly procedure. Since Nix computes the checksum of the tarball, it will detect that it differs from the hash of the original, and has no means to ensure that the tarball downloaded from the Software Heritage archive corresponds to the one it expects.

Nix, being really stubborn with respect to reproducibility, computes the checksum of the tarball, and detects that it differs from the hash of the original. Nix simply cannot ensure that the tarball downloaded from the Software Heritage archive corresponds to the one it expects, and rightly so because this could easily lead to a different build.

Nix already deals with some situations where the hash of a source tarball can change without affecting build reproducibility. For example, fetchFromGitHub unpacks the tarball before computing the hash of the unpacked contents. This is because GitHub produces release tarballs on the fly, and a change in the compression algorithm can invalidate the hash expected by Nix. It happens that when Nix uses the fetchFromGithub strategy, it manages to download tarballs from Software Heritage. Currently, this is the case for about 6,000 sources in Nixpkgs.

For all other sources, we would have to modify the Nixpkgs fetchers to compute checksums on the unpacked tarball contents. This seems reasonable, since we want to make sure that the source code is the same, and we don’t care much about the format in it is transferred (which can also be important for security reasons). However, using the hash of the tarball directly is often more convenient, since it is exposed by many repositories such as Pypi and Hackage. When updating a package in Nixpkgs, the maintainer can just pick the checksum provided by the package repository, instead of recalculating it locally.

Ideally, Nix fetchers and these package repositories would provide checksums of the unpacked contents of their packages as much as possible. This would give us the freedom to identify content independently of the way it is packed, transferred and stored. We haven’t yet decided how to move on, but rest assured that we will continue to work on it.

Wrap up

Thanks to Software Heritage, a significant part of the sources used by Nixpkgs are now archived forever. Moreover, about 6,000 out of the 21,000 tarballs used by Nixpkgs can already be used by the future Nix fallback mechanism. We now want to increase the number of archived source code tarballs, add support for Git sources and start to implement the fallback mechanism in Nix. At the same time, we aim to increase the number of fixed output derivations whose hash is computed on the unpacked tarball.

We want build reproducibility to become the standard in the software world. For this reason, the Software Heritage loader archiving Nixpkgs source code tarballs has been designed to be easily reused by other actors. For example, the Guix project already started publishing and archiving their sources using the same component! Finally, we want to thank NLnet for the funding that makes this work possible.

• They are self contained -- they do not rely on any external services to operate. This makes it very easy to run a collection of processes for local experimentation.
• They have a standard means to notify the caller that the service is ready. By convention, the executable that spawns the daemon process is only supposed to terminate when the daemon has been successfully initialized. For example, foreground processes that are managed by systemd, should invoke the non-standard sd_notify() function to notify systemd that they are ready.

Specifying a process configuration

{writeTextFile, mydaemon}:

writeTextFile {
  name = "mydaemon";
  text = ''
    process=${mydaemon}/bin/mydaemon
    pidFile=/var/run/mydaemon.pid
  '';
  destination = "/etc/dysnomia/process";
}

• The process field specifies the path to executable to start (that in turn spawns a deamon process that keeps running in the background).
• The pidFile field indicates the location of the PID file containing the process ID of the daemon process, so that it can be reliably terminated.
  

• If no pidFile is provided, then the deployment system assumes that the daemon generates a PID file with the same name as the executable and resides in the directory that is commonly used for storing PID files: /var/run.
• If a package provides only a single executable in the bin/ sub folder, then it is also not required to specify a process.

{stdenv, writeTextFile, writeScript, daemon, myForegroundService}:

let
  myForegroundServiceWrapper = writeScript {
    name = "myforegroundservice-wrapper";
    text = ''
      #! ${stdenv.shell} -e

      mkdir -p /var/lib/myservice
      exec ${daemon}/bin/daemon -U -F /var/run/mydaemon.pid -- \
        ${myForegroundService}/bin/myservice
    '';
  };
in
writeTextFile {
  name = "mydaemon";
  text = ''
    process=${myForegroundServiceWrapper}
    pidFile=/var/run/mydaemon.pid
  '';
  destination = "/etc/dysnomia/process";
}

• First, it creates the required state directory: /var/lib/myservice so that the service can work properly.
• Then it invokes libslack's daemon command to automatically daemonize the service. The daemon command will automatically store a PID file containing the daemon's process ID, so that the configuration system knows how to terminate it. The value of the -F parameter passed to the daemon executable and the pidFile configuration property are the same.

{createManagedProcess, runtimeDir}:
{port}:

let
  webapp = import ../../webapp;
in
createManagedProcess rec {
  name = "webapp";
  description = "Simple web application";

  # This expression can both run in foreground or daemon mode.
  # The process manager can pick which mode it prefers.
  process = "${webapp}/bin/webapp";
  daemonArgs = [ "-D" ];

  environment = {
    PORT = port;
    PID_FILE = "${runtimeDir}/${name}.pid";
  };
}

Managing daemons with Disnix

$ kill $(cat $pidFile)

Translation to a Disnix deployment specification

{ pkgs ? import  { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager ? "disnix"
}:

let
  constructors = import ./constructors.nix {
    inherit pkgs stateDir runtimeDir logDir tmpDir forceDisableUserChange processManager;
  };
in
rec {
  webapp = rec {
    port = 5000;
    dnsName = "webapp.local";

    pkg = constructors.webapp {
      inherit port;
    };
  };

  nginxReverseProxy = rec {
    port = 8080;

    pkg = constructors.nginxReverseProxyHostBased {
      webapps = [ webapp ];
      inherit port;
    } {};
  };
}

• The webapp process is the web application process described earlier that runs an embedded HTTP server and serves a static HTML page.
• The nginxReverseProxy is an Nginx web server that acts as a reverse proxy server for the webapp process. To make this service to work properly, it needs to be activated after the webapp process is activated. To ensure that the activation is done in the right order, webapp is passed as a process dependency to the nginxReverseProxyHostBased constructor function.

nginxReverseProxy = rec {
  name = "nginxReverseProxy";
  port = 8080;

  pkg = constructors.nginxReverseProxyHostBased {
    webapps = [ webapp ];
    inherit port;
  } {};

  activatesAfter = {
    inherit webapp;
  };

  type = "process";
};

• A name property because this is a mandatory field for every service.
• In the process management framework all process instances are managed by the same process manager, but in Disnix services can have all kinds of shapes and formes and require a plugin to manage their life-cycles.
  
  To allow Disnix to manage daemons, we specify the type property to refer to our process Dysnomia module that starts and terminates a daemon from a simple textual specification.
• The process dependencies are translated to Disnix inter-dependencies by using the activatesAfter property.
  
  In Disnix, inter-dependency parameters serve two purposes -- they provide the inter-dependent services with configuration parameters and they ensure the correct activation ordering.
  
  The activatesAfter parameter disregards the first inter-dependency property, because we are already using the process management framework's convention for propagating process dependencies.

An example deployment scenario

$ nixproc-disnix-switch --state-dir /home/sander/var \
  --force-disable-user-change processes.nix

$ disnix-visualize > out.dot
$ dot -Tpng out.dot > out.png

Discussion

Availability

• All units of which a system consists are defined as Nix expressions declaring a function. Each function parameter refers to a dependency or configuration property required to construct the unit from its sources.
• To compose a particular variant of a unit, we must invoke the function that builds and configures the unit with parameters providing the dependencies and configuration properties that the unit needs.
• To make all units conveniently accessible from a single location, the content of the configuration units is typically blended into a symlink tree called Nix profiles.

Motivating example: deploying a Java-based web application and web service system

Configuring the client interface for local deployment

$ export DISNIX_CLIENT_INTERFACE=disnix-client

$ export DISNIX_CLIENT_INTERFACE=disnix-runactivity

Deploying the example system locally

{
  localhost.properties.hostname = "localhost";
}

{infrastructure}:

{
  simpleAppservingTomcat = [ infrastructure.localhost ];
  axis2 = [ infrastructure.localhost ];

  HelloService = [ infrastructure.localhost ];
  HelloWorldService = [ infrastructure.localhost ];
  HelloWorld = [ infrastructure.localhost ];
}

$ disnix-env -s services.nix -i infrastructure-local.nix -d distribution-local.nix

Deploying the example system locally as an unprivileged user

{ pkgs, system, distribution, invDistribution
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager ? "systemd"
}:

let
  processType =
    if processManager == null then "managed-process"
    else if processManager == "sysvinit" then "sysvinit-script"
    else if processManager == "systemd" then "systemd-unit"
    else if processManager == "supervisord" then "supervisord-program"
    else if processManager == "bsdrc" then "bsdrc-script"
    else if processManager == "cygrunsrv" then "cygrunsrv-service"
    else if processManager == "launchd" then "launchd-daemon"
    else throw "Unknown process manager: ${processManager}";

  constructors = import ../../../nix-processmgmt/examples/service-containers-agnostic/constructors.nix {
    inherit pkgs stateDir runtimeDir logDir cacheDir tmpDir forceDisableUserChange processManager;
  };

  customPkgs = import ../top-level/all-packages.nix {
    inherit system pkgs stateDir;
  };
in
rec {
  simpleAppservingTomcat = constructors.simpleAppservingTomcat {
    httpPort = 8080;
    type = processType;
  };
  ...
}

$ disnix-env -s services.nix -i infrastructure-local.nix -d distribution-local.nix \
  --extra-params '{
  stateDir = "/home/sander/var";
  processManager = "sysvinit";
  forceDisableUserChange = true;
}'

Working with predeployed container services

$ disnix-capture-infra infrastructure.nix > \
  infrastructure-captured.nix

$ disnix-env -s services.nix -i infrastructure-captured.nix -d distribution-local.nix

Undeploying a system

$ disnix-env --undeploy -i infrastructure.nix

Availability

This is the first in a series of blog posts intended to provide a gentle introduction to flakes, a new Nix feature that improves reproducibility, composability and usability in the Nix ecosystem. This blog post describes why flakes were introduced, and give a short tutorial on how to use them.

Flakes were developed at Tweag and funded by Target Corporation and Tweag.

What problems do flakes solve?

Once upon a time, Nix pioneered reproducible builds: it tries hard to ensure that two builds of the same derivation graph produce an identical result. Unfortunately, the evaluation of Nix files into such a derivation graph isn’t nearly as reproducible, despite the language being nominally purely functional.

For example, Nix files can access arbitrary files (such as ~/.config/nixpkgs/config.nix), environment variables, Git repositories, files in the Nix search path ($NIX_PATH), command-line arguments (--arg) and the system type (builtins.currentSystem). In other words, evaluation isn’t as hermetic as it could be. In practice, ensuring reproducible evaluation of things like NixOS system configurations requires special care.

Furthermore, there is no standard way to compose Nix-based projects. It’s rare that everything you need is in Nixpkgs; consider for instance projects that use Nix as a build tool, or NixOS system configurations. Typical ways to compose Nix files are to rely on the Nix search path (e.g. import <nixpkgs>) or to use fetchGit or fetchTarball. The former has poor reproducibility, while the latter provides a bad user experience because of the need to manually update Git hashes to update dependencies.

There is also no easy way to deliver Nix-based projects to users. Nix has a “channel” mechanism (essentially a tarball containing Nix files), but it’s not easy to create channels and they are not composable. Finally, Nix-based projects lack a standardized structure. There are some conventions (e.g. shell.nix or release.nix) but they don’t cover many common use cases; for instance, there is no way to discover the NixOS modules provided by a repository.

Flakes are a solution to these problems. A flake is simply a source tree (such as a Git repository) containing a file named flake.nix that provides a standardized interface to Nix artifacts such as packages or NixOS modules. Flakes can have dependencies on other flakes, with a “lock file” pinning those dependencies to exact revisions to ensure reproducible evaluation.

The flake file format and semantics are described in a NixOS RFC, which is currently the best reference on flakes.

Trying out flakes

Flakes are currently implemented in an experimental branch of Nix. If you want to play with flakes, you can get this version of Nix from Nixpkgs:

$ nix-shell -I nixpkgs=channel:nixos-20.03 --packages nixFlakes

Since flakes are an experimental feature, you also need to add the following line to ~/.config/nix/nix.conf:

experimental-features = nix-command flakes

or pass the flag --experimental-features 'nix-command flakes' whenever you call the nix command.

Using flakes

To see flakes in action, let’s start with a simple Unix package named dwarffs (a FUSE filesystem that automatically fetches debug symbols from the Internet). It lives in a GitHub repository at https://github.com/edolstra/dwarffs; it is a flake because it contains a file named flake.nix. We will look at the contents of this file later, but in short, it tells Nix what the flake provides (such as Nix packages, NixOS modules or CI tests).

The following command fetches the dwarffs Git repository, builds its default package and runs it.

$ nix shell github:edolstra/dwarffs --command dwarffs --version
dwarffs 0.1.20200406.cd7955a

The command above isn’t very reproducible: it fetches the most recent version of dwarffs, which could change over time. But it’s easy to ask Nix to build a specific revision:

$ nix shell github:edolstra/dwarffs/cd7955af31698c571c30b7a0f78e59fd624d0229 ...

Nix tries very hard to ensure that the result of building a flake from such a URL is always the same. This requires it to restrict a number of things that Nix projects could previously do. For example, the dwarffs project requires a number of dependencies (such as a C++ compiler) that it gets from the Nix Packages collection (Nixpkgs). In the past, you might use the NIX_PATH environment variable to allow your project to find Nixpkgs. In the world of flakes, this is no longer allowed: flakes have to declare their dependencies explicitly, and these dependencies have to be locked to specific revisions.

In order to do so, dwarffs’s flake.nix file declares an explicit dependency on Nixpkgs, which is also a flake. We can see the dependencies of a flake as follows:

$ nix flake list-inputs github:edolstra/dwarffs
github:edolstra/dwarffs/d11b181af08bfda367ea5cf7fad103652dc0409f
├───nix: github:NixOS/nix/3aaceeb7e2d3fb8a07a1aa5a21df1dca6bbaa0ef
│   └───nixpkgs: github:NixOS/nixpkgs/b88ff468e9850410070d4e0ccd68c7011f15b2be
└───nixpkgs: github:NixOS/nixpkgs/b88ff468e9850410070d4e0ccd68c7011f15b2be

So the dwarffs flake depends on a specific version of the nixpkgs flake (as well as the nix flake). As a result, building dwarffs will always produce the same result. We didn’t specify this version in dwarffs’s flake.nix. Instead, it’s recorded in a lock file named flake.lock that is generated automatically by Nix and committed to the dwarffs repository.

Flake outputs

Another goal of flakes is to provide a standard structure for discoverability within Nix-based projects. Flakes can provide arbitrary Nix values, such as packages, NixOS modules or library functions. These are called its outputs. We can see the outputs of a flake as follows:

$ nix flake show github:edolstra/dwarffs
github:edolstra/dwarffs/d11b181af08bfda367ea5cf7fad103652dc0409f
├───checks
│   ├───aarch64-linux
│   │   └───build: derivation 'dwarffs-0.1.20200409'
│   ├───i686-linux
│   │   └───build: derivation 'dwarffs-0.1.20200409'
│   └───x86_64-linux
│       └───build: derivation 'dwarffs-0.1.20200409'
├───defaultPackage
│   ├───aarch64-linux: package 'dwarffs-0.1.20200409'
│   ├───i686-linux: package 'dwarffs-0.1.20200409'
│   └───x86_64-linux: package 'dwarffs-0.1.20200409'
├───nixosModules
│   └───dwarffs: NixOS module
└───overlay: Nixpkgs overlay

While a flake can have arbitrary outputs, some of them, if they exist, have a special meaning to certain Nix commands and therefore must have a specific type. For example, the output defaultPackage.<system> must be a derivation; it’s what nix build and nix shell will build by default unless you specify another output. The nix CLI allows you to specify another output through a syntax reminiscent of URL fragments:

$ nix build github:edolstra/dwarffs#checks.aarch64-linux.build

By the way, the standard checks output specifies a set of derivations to be built by a continuous integration system such as Hydra. Because flake evaluation is hermetic and the lock file locks all dependencies, it’s guaranteed that the nix build command above will evaluate to the same result as the one in the CI system.

The flake registry

Flake locations are specified using a URL-like syntax such as github:edolstra/dwarffs or git+https://github.com/NixOS/patchelf. But because such URLs would be rather verbose if you had to type them all the time on the command line, there also is a flake registry that maps symbolic identifiers such as nixpkgs to actual locations like https://github.com/NixOS/nixpkgs. So the following are (by default) equivalent:

$ nix shell nixpkgs#cowsay --command cowsay Hi!
$ nix shell github:NixOS/nixpkgs#cowsay --command cowsay Hi!

It’s possible to override the registry locally. For example, you can override the nixpkgs flake to your own Nixpkgs tree:

$ nix registry add nixpkgs ~/my-nixpkgs

or pin it to a specific revision:

$ nix registry add nixpkgs github:NixOS/nixpkgs/5272327b81ed355bbed5659b8d303cf2979b6953

Writing your first flake

Unlike Nix channels, creating a flake is pretty simple: you just add a flake.nix and possibly a flake.lock to your project’s repository. As an example, suppose we want to create our very own Hello World and distribute it as a flake. Let’s create this project first:

$ git init hello
$ cd hello
$ echo 'int main() { printf("Hello World"); }' > hello.c
$ git add hello.c

To turn this Git repository into a flake, we add a file named flake.nix at the root of the repository with the following contents:

{
  description = "A flake for building Hello World";

  inputs.nixpkgs.url = github:NixOS/nixpkgs/nixos-20.03;

  outputs = { self, nixpkgs }: {

    defaultPackage.x86_64-linux =
      # Notice the reference to nixpkgs here.
      with import nixpkgs { system = "x86_64-linux"; };
      stdenv.mkDerivation {
        name = "hello";
        src = self;
        buildPhase = "gcc -o hello ./hello.c";
        installPhase = "mkdir -p $out/bin; install -t $out/bin hello";
      };

  };
}

The command nix flake init creates a basic flake.nix for you.

Note that any file that is not tracked by Git is invisible during Nix evaluation, in order to ensure hermetic evaluation. Thus, you need to make flake.nix visible to Git:

$ git add flake.nix

Let’s see if it builds!

$ nix build
warning: creating lock file '/home/eelco/Dev/hello/flake.lock'
warning: Git tree '/home/eelco/Dev/hello' is dirty

$ ./result/bin/hello
Hello World

or equivalently:

$ nix shell --command hello
Hello World

It’s also possible to get an interactive development environment in which all the dependencies (like GCC) and shell variables and functions from the derivation are in scope:

$ nix dev-shell
$ eval "$buildPhase"
$ ./hello
Hello World

So what does all that stuff in flake.nix mean?

• The description attribute is a one-line description shown by nix flake info.
• The inputs attribute specifies other flakes that this flake depends on. These are fetched by Nix and passed as arguments to the outputs function.

• The outputs attribute is the heart of the flake: it’s a function that produces an attribute set. The function arguments are the flakes specified in inputs.

  The self argument denotes this flake. Its primarily useful for referring to the source of the flake (as in src = self;) or to other outputs (e.g. self.defaultPackage.x86_64-linux).

• The attributes produced by outputs are arbitrary values, except that (as we saw above) there are some standard outputs such as defaultPackage.${system}.
• Every flake has some metadata, such as self.lastModifiedDate, which is used to generate a version string like hello-20191015.

You may have noticed that the dependency specification github:NixOS/nixpkgs/nixos-20.03 is imprecise: it says that we want to use the nixos-20.03 branch of Nixpkgs, but doesn’t say which Git revision. This seems bad for reproducibility. However, when we ran nix build, Nix automatically generated a lock file that precisely states which revision of nixpkgs to use:

$ cat flake.lock
{
  "nodes": {
    "nixpkgs": {
      "info": {
        "lastModified": 1587398327,
        "narHash": "sha256-mEKkeLgUrzAsdEaJ/1wdvYn0YZBAKEG3AN21koD2AgU="
      },
      "locked": {
        "owner": "NixOS",
        "repo": "nixpkgs",
        "rev": "5272327b81ed355bbed5659b8d303cf2979b6953",
        "type": "github"
      },
      "original": {
        "owner": "NixOS",
        "ref": "nixos-20.03",
        "repo": "nixpkgs",
        "type": "github"
      }
    },
    "root": {
      "inputs": {
        "nixpkgs": "nixpkgs"
      }
    }
  },
  "root": "root",
  "version": 5
}

Any subsequent build of this flake will use the version of nixpkgs recorded in the lock file. If you add new inputs to flake.nix, when you run any command such as nix build, Nix will automatically add corresponding locks to flake.lock. However, it won’t replace existing locks. If you want to update a locked input to the latest version, you need to ask for it:

$ nix flake update --update-input nixpkgs
$ nix build

To wrap things up, we can now commit our project and push it to GitHub, after making sure that everything is in order:

$ nix flake check
$ git commit -a -m 'Initial version'
$ git remote add origin git@github.com:edolstra/hello.git
$ git push -u origin master

Other users can then use this flake:

$ nix shell github:edolstra/hello -c hello

Next steps

In the next blog post, we’ll talk about typical uses of flakes, such as managing NixOS system configurations, distributing Nixpkgs overlays and NixOS modules, and CI integration.