NixOS Planet

September 10, 2019

Munich NixOS Meetup

NixOS 19.09 Release Sprint

photoMunich NixOS Meetup

The next stable NixOS release 19.09 'Loris' is going to happen at the end of September. The goal of this sprint is to fix critical issues before the release. Some maintainers will be attending and are available for guidance and feedback.

• Blocking issues: https://github.com/Ni...

• All 19.09 issues: https://github.com/Ni...

ZHF issue: https://github.com/Ni...

The sprint will be held at the Mayflower office in Munich on Friday starting at 11:00. Drinks will be provided.

München 80687 - Germany

Friday, September 13 at 11:00 AM

15

https://www.meetup.com/Munich-NixOS-Meetup/events/264400018/

September 10, 2019 03:57 PM

August 30, 2019

Hercules Labs

Native support for import-from-derivation

Today we are releasing a new feature we’ve been working on the last couple of weeks.

Generating Nix expressions

As a developer you often bump a dependency version or add a new dependency.

Every time your package files change, you need to regenerate the Nix expressions that describe how the project is built.

There are two ways to regenerate Nix expressions in that case:

  1. Outside the Nix domain, possibly with an automated script and commands like bundix, cabal2nix, yarn2nix. This quickly grows from a nuisance to a maintenance headache as your git repository grows in size due to generated artifacts. It requires special care when diffing, merging, etc.

  2. Let Nix generate Nix expressions during the build. Sounds simple, but it’s quite subtle.

Additionally, Nixpkgs builds forbid option (2), which leads to manual work.

As of today Hercules natively supports option (2), let’s dig into the subtleties.

Evaluation and realization

The Nix language describes how software is built, which happens in two phases.

The first phase is called evaluation:

Evaluation

Evaluation takes a Nix expression and results into a dependency tree of derivations.

A derivation is a set of instructions how to build software.

The second phase is called realization:

Evaluation

Realizing a derivation is the process of building. The builder is usually a shell script, although any executable can be specified.

Since a derivation describes all the necessary inputs, the result is guaranteed to be deterministic.

Derivations

This begs the question, why have intermediate representation (derivations)? There are a couple of reasons:

  • Evaluation can include significant computation. It can range from a couple of seconds, to typically minutes, or even an hour for huge projects. We want to evaluate only once and then distribute derivations to multiple machines for speedup and realize them as we traverse the graph of dependencies.

  • Evaluation can produce derivations that are built on different platforms or require some specific hardware. By copying the derivations to these machines, we don’t need to worry about running evaluation on those specific machines.

  • In case of a build failure, it allows the machine to retry immediately instead of re-evaluating again.

All in all, derivation files save us computation compared to evaluating more than once.

Interleaving evaluation and realization

Sometimes it’s worth mixing the two phases.

A build produces Nix expressions that we now would like to evaluate, but we’re already in the realization phase, so we have:

  1. Evaluate to get the derivation that will output a Nix file
  2. Realize that derivation
  3. Continue evaluating by importing the derivation containing the Nix file
  4. Realize the final derivation set

This is called Import-From-Derivation or shortly, IFD.

A minimal example

let
  pkgs = import <nixpkgs> {};
  getHello = pkgs.runCommand "get-hello.nix" {} ''
    # Call any command here to generate an expression. A simple example:
    echo 'pkgs: pkgs.hello' > $out
  '';
in import getHello pkgs

In the last line we’re importing from getHello, which is a Nix derivation that we need to build before evaluation can continue to use pkgs: pkgs.hello Nix expression in the output.

Haskell.nix example

haskell.nix is an alternative Haskell infrastructure for Nixpkgs.

Given a Haskell project with a Cabal file (Haskell’s package manager), drop the following default.nix into root of your repository:

let
  pkgs = import (import ./nix/sources.nix).nixpkgs {};
  haskell = import (import ./nix/sources.nix)."haskell.nix" { inherit pkgs; };
  plan = haskell.callCabalProjectToNix
              { index-state = "2019-08-26T00:00:00Z"; src = pkgs.lib.cleanSource ./.;};

  pkgSet = haskell.mkCabalProjectPkgSet {
    plan-pkgs = import plan;
    pkg-def-extras = [];
    modules = [];
  };
in pkgSet.config.hsPkgs.mypackage.components.all

Once you replace mypackage with the name from your Cabal file, your whole dependency tree is deterministic by pinning the package index to a timestamp using index-state and hash of your local folder using ./..

Haskell.nix will generate all expressions how to build each package on the fly via import from derivation.

Native support in CI

Using different platforms (typically Linux and macOS) during IFD is one of the reasons why upstream forbids IFD, since their evaluator is running on Linux and it can’t build for macOS.

Our CI dispatches all builds during IFD back to our scheduler, so it’s able to dispatch those builds to either specific platform or specific hardware.

IFD support is seamless. There’s nothing extra to configure.

In case of build errors during evaluation UI will show you all the details including build log:

IFD attribute error

In order to use IFD support you will need to upgrade to hercules-ci-agent-0.4.0.

Future work

Some Nix tools already embrace IFD, such as haskell.nix, yarn2nix (Node.js), pnpm2nix (Node.js) and opam2nix (OCaml).

We encourage more language tools to take advantage of this feature.

Currently Nix evaluation is single threaded and IFD evaluation is blocking until the builds are done. We have some ideas to make IFD concurrent.

We believe this is a huge step forward to simplify day-to-day Nix development.

What we do

Automated hosted infrastructure for Nix, reliable and reproducible developer tooling, to speed up adoption and lower integration cost. We offer Continuous Integration and Binary Caches.

Updates

2019-09-08: Add opam2nix

August 30, 2019 12:00 AM

August 22, 2019

Hercules Labs

Pre-commit git hooks with Nix

pre-commit manages a set of hooks that are executed by git before committing code:

pre-commit.png

Common hooks range from static analysis or linting to source formatting.

Since we’re managing quite a couple of repositories, maintaining the duplicated definitions became a burden.

Hence we created:

nix-pre-commit-hooks

The goal is to manage these hooks with Nix and solve the following:

  • Simpler integration into Nix projects, doing the wiring up behind the scenes

  • Provide a low-overhead build of all the tooling available for the hooks to use (calling nix-shell for every check does bring some latency when committing)

  • Common package set of hooks for popular languages like Haskell, Elm, etc.

  • Two trivial Nix functions to run hooks as part of development and on your CI

Currently the following hooks are provided:

Nix

  • canonix: a Nix formatter (currently incomplete, requiring some manual formatting as well)

Haskell

Elm

Shell

We encourage everyone to contribute additional hooks.

Installation

See project’s README for latest up-to-date installation steps.


What we do

Automated hosted infrastructure for Nix, reliable and reproducible developer tooling, to speed up adoption and lower integration cost. We offer Continuous Integration and binary caches.

August 22, 2019 12:00 AM

August 21, 2019

Matthew Bauer

All the versions with Nix

1 Background

In Channel Changing with Nix, I described how to move between channels in Nix expressions. This provides an easy way to work with multiple versions of Nixpkgs. I was reminded of this post after seeing a comment by jolmg on Hacker News. The comment suggested we should have a way to use multiple versions of packages seamlessly together. It suggested that we should use commits to differentiate versions, but I suggested that stable channels would work much better.

So, as a follow up to my Channel Changing post, I want to show how you can use a quick Nix mockup to accomplish this. Like the previous post, it will come with a Nix snippet that you can try out yourself.

2 Code

So, what follows is some code that I wrote up that lets you find package versions in an easier way. It is also available for download at https://matthewbauer.us/generate-versions.nix.

{ channels ? [ "19.03" "18.09" "18.03" "17.09" "17.03"
         "16.09" "16.03" "15.09" "14.12" "14.04" "13.10" ]
, attrs ? builtins.attrNames (import <nixpkgs> {})
, system ? builtins.currentSystem
, args ? { inherit system; }
}: let

  getSet = channel:
    (import (builtins.fetchTarball "channel:nixos-${channel}") args).pkgs;

  getPkg = name: channel: let
    pkgs = getSet channel;
    pkg = pkgs.${name};
    version = (builtins.parseDrvName pkg.name).version;
  in if builtins.hasAttr name pkgs && pkg ? name then {
    name = version;
    value = pkg;
  } else null;

in builtins.listToAttrs (map (name: {
  inherit name;
  value = builtins.listToAttrs
    (builtins.filter (x: x != null)
      (map (getPkg name) channels));
}) attrs)

This Nix expression generates an index of each package from all 11 releases of Nixpkgs that have occurred since October 2010. For every package, each version that came with a release is included and put into a map. The map uses the version as a key and the package as its value, preferring the newer release when versions conflict.

This is all done lazily, because that’s how Nix works. Still, it will take a little while at first to evaluate because we need to parse all 11 releases! Remarkably, this expression uses only Nix builtins, and requires no special library function.

3 Usage

Working with this Nix expression is extremely interesting, and I’ve included some examples of how to work with it. They should all be usable on a Linux machine (or maybe macOS) with Nix installed.

3.1 Query package versions

You can query what package versions are available through Nix’s builtins.attrNames function. For example,

$ nix eval "(builtins.attrNames (import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).emacs)"
[ "24.3" "24.4" "24.5" "25.3" "26.1" ]

This shows us that there are 5 versions of Emacs. This is kind of interesting because it means that there were at least 6 duplicate versions of Emacs between our release channels. Unfortunately, a few versions of Emacs are notably missing including Emacs 25.1 and Emacs 25.2. Emacs 24.2 was released almost a year before the first stable Nixpkgs release! As time goes on, we should collect more of these releases.

3.2 Running an old version

A shown above, there are 5 versions of Emacs available to us. We can run Emacs 24.3 with a fairly short command:

$ LC_ALL=C nix run "(import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).emacs.\"24.3\"" -c emacs

LC_ALL=C is needed on Linux to avoid the old Glibc trying to load the newer, incompatible locales that may be included with your system. This is an unfortunate problem with Glibc including breaking changes between releases. It also makes me want use to switch to Musl some time soon! I’ve also noticed some incompatibilities with GTK icons that appear to come from the gdk-pixbuf module. More investigation is needed on why this is the case.

This will not work on macOS because we did not have Emacs working on macOS back then! macOS users can try Emacs 25.3. This looks very similar to the above:

$ nix run "(import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).emacs.\"25.3\"" -c emacs

3.3 Firefox

Another example using Firefox is pretty neat. The code is very similar to Emacs:

$ nix eval "(builtins.attrNames (import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).firefox)"
[ "25.0.1" "34.0.5" "39.0.3" "45.0" "48.0.2" "51.0.1" "55.0.3" "59.0.2" "63.0.3" "66.0.3" "68.0.2" ]

We get all 11 releases with unique Firefox versions this time.

You can run Firefox 25.0.1 using this command:

$ LC_ALL=C nix run "(import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).firefox.\"25.0.1\"" -c firefox

Amazing how notably Firefox has changed since then!

3.4 Blender

Another example using Blender. The code is very similar to the two above:

$ nix eval "(builtins.attrNames (import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).blender)"
[ "2.67" "2.70" "2.72b" "2.75a" "2.77a" "2.78c" "2.79" "2.79a" "2.79b" ]

You can run Blender 2.67 using this command:

$ LC_ALL=C nix run "(import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).blender.\"26.7\"" -c blender

4 Rationale

The reason that channels work better than commits is because every commit in Nixpkgs is not guaranteed to work on its own. Some may be missing security patches, configuration changes, or worse may just not work with other versions of packages. In addition, there are just too many commits to work with effectively. On the other hand, Nixpkgs release stable channels every 6 months, and we have a long vetting process of ensuring the stabilized channel works well.

The main drawback the 6-month channels have is that we don’t have every version released of package. If the version you want is missing in a release, you are out of luck. But, the 6-month window tends to pick up a lot of packages and we end up with almost every major version of popular software. My philosophy is not all releases are worth keeping. Some contain critical security flaws, contain major bugs, and might not work well with other software. The 6-month window is good enough for me. Perhaps in the future we can increase Nixpkgs release cadence to 3-month or 1-month, but the maintainers are not quite ready for that yet.

5 Conclusion

This has hopefully shown how Nix’s functional dependency model makes it very easy to switch between versions of packages. This is builtin to Nix, but you need some scripts to really use this well. Our 6-month release window is an arbitrary choice, but tends to pick up a lot of useful versions in the mean time.

August 21, 2019 12:00 AM

August 19, 2019

Mayflower

The NixOS RFC Process

History The NixOS RFC process was established in March 2017, initiated by zimbatm, teh and Moretea in order to standardise a process to find an agreement on larger changes to Nix/NixOS/Nixpkgs and the ecosystem in general. Over the following one and a half years a few uncontroversial RFCs were merged but for most of the RFCs that needed further discussion nobody felt responsible to make a decision. That is the reason RFC 36 was written in collaboration with most core Nix/Nixpkgs members at the last NixCon and merged in December 2018 in order to streamline the process and define clearer responsibilities.

August 19, 2019 02:30 PM

August 11, 2019

Sander van der Burg

A new input model transformation pipeline for Disnix

As explained in earlier blog posts, Disnix (as well as other tools in the Nix project) are driven by declarative specifications -- instead of describing the activities that need to be carried out to deploy a system (such as building and distributing packages), we specify all the relevant properties of a service-oriented system:

  • The services model describes all the services that can be deployed to target machines in a network, how they can be built from their sources, how they depend on each other and what their types are, so that the deployment system knows how they can be activated.
  • The infrastructure model captures all target machines in the network, their properties, and the containers they provide. Containers in a Disnix-context are services that manage the life-cycle of a component, such as an application server, service manager or database management service (DBMS).
  • The distribution model maps services to containers on the target machines.

By running the following command-line instruction:

$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix

Disnix infers all the activities that need to be executed to get the system in a running state, such as building packages from source code (or downloading substitutes from a binary cache), the distribution of packages, the activation of a system and taking and restoring state snapshots.

Conceptually, this approach may sound very simple but the implementation that infers the deployment process is not. Whilst the input models are declarative, they are not executable -- there is not a one-on-one mapping between properties in the input models and the activities that Disnix needs to carry out.

To be able to execute deployment activities, Disnix transforms the three input models into a single declarative specification (called a deployment manifest file) that contains one-on-one mappings between deployment artifacts (e.g. Nix profiles, Nix packages and snapshots) and deployment targets (the target machines and/or container services). The transformation pipeline fills in the blanks with default settings, and transforms the input models into several intermediate representations, before it gets transformed into the manifest file.

So far, the intermediate representations and final result were never well defined. Instead, they have organically evolved and were heavily revised several times. As a result of adding new features and not having well defined representations, it became very hard to make changes and reason about the correctness of the models.

In my previous blog post, I have developed libnixxml to make the integration between a data model defined in the Nix expression language and external tools (that implement deployment activities that Nix does not support) more convenient. I am primarily using this library to simplify the integration of manifest files with Disnix tools.

As an additional improvement, I have revised the transformation pipeline, with well-defined intermediate representations. Besides a better quality transformation pipeline with well-defined intermediate stages, the Disnix toolset can now also take the intermediate model representations as input parameters, which is quite convenient for integration with external tooling and experimentation purposes. Furthermore, a new input model has been introduced.

In the blog post, I will describe the steps in the transformation pipeline, and the intermediate representations of the deployment models.

Separated concerns: services, infrastructure, distribution models


As explained earlier in this blog post, Disnix deployment are primarily driven by three input models: the services, infrastructure and distribution models. The reason why I have picked three input models (as opposed to a single configuration file) is to separate concerns and allow these concerns to be reused in different kinds of deployment scenarios.

For example, we can write a simple services model (services.nix) that describes two services that have an inter-dependency on each other:

{distribution, invDistribution, system, pkgs}:

let customPkgs = import ../top-level/all-packages.nix {
inherit system pkgs;
};
in
rec {
HelloMySQLDB = {
name = "HelloMySQLDB";
pkg = customPkgs.HelloMySQLDB;
dependsOn = {};
type = "mysql-database";
};

HelloDBService = {
name = "HelloDBService";
pkg = customPkgs.HelloDBServiceWrapper;
dependsOn = {
inherit HelloMySQLDB;
};
type = "tomcat-webapplication";
};
}

The above services model captures two services with the following properties:

  • The HelloMySQLDB services refers to a MySQL database backend that stores data. The type property: mysql-database specifies which Dysnomia module should be used to manage the lifecycle of the service. For example, the mysql-database Dysnomia module will create the database on initial startup.
  • The HelloDBService is a web service that exposes the data stored in the database backend to the outside it world. Since it requires the presence of a MySQL database backend and needs to know where it has been deployed, the database backend been declared as an inter-dependency of the service (by means of the dependsOn attribute).

    The tomcat-webapplication type specifies that Disnix should use the Apache Tomcat Dysnomia module, to activate the corresponding Java-based web service inside the Apache Tomcat servlet container.

The services model captures the aspects of a service-oriented system from a functional perspective, without exposing much of the details of the environments they may run in. This is intentional -- the services are meant to be deployed to a variety of environments. Target agnostic services make it possible, for example, to write an infrastructure model of a test environment (infrastructure-test.nix):

{
test1 = {
properties = {
hostname = "test1.example.org";
};

containers = {
tomcat-webapplication = {
tomcatPort = 8080;
};
};
};

test2 = {
properties = {
hostname = "test2.example.org";
};

containers = {
tomcat-webapplication = {
tomcatPort = 8080;
};

mysql-database = {
mysqlPort = 3306;
mysqlUsername = "mysqluser";
mysqlPassword = builtins.readFile ./mysqlpw;
};
};
};
}

and a distribution model that maps the services to the target machines in the infrastructure model (distribution-test.nix):

{infrastructure}:

{
HelloMySQLDB = [ infrastructure.test2 ];
HelloDBService = [ infrastructure.test1 ];
}

With these three deployment models, we can deploy a system to a test environment, by running:

$ disnix-env -s services.nix \
-i infrastructure-test.nix \
-d distribution-test.nix

and later switch to a production environment using the same functional services model, after the system has been properly validated in the test environment:

$ disnix-env -s services.nix \
-i infrastructure-prod.nix \
-d distribution-prod.nix

Similarly, we can adjust the distribution model to only deploy a sub set of the services of a system for, say, experimentation purposes.

Unifying the input models into a single specification: the deployment architecture model


The first step in transforming the input models into a single executable specification, is unifying the specifications into one single declarative specification, that I will call the deployment architecture model. The name is derived from the concept of deployment architectures in software architecture terminology:
a description that specifies the distribution of software components over hardware nodes.

A Disnix deployment architecture model may look as follows:

{system, pkgs}:

let customPkgs = import ../top-level/all-packages.nix {
inherit system pkgs;
};
in
rec {
services = rec {
HelloMySQLDB = {
name = "HelloMySQLDB";
pkg = customPkgs.HelloMySQLDB;
dependsOn = {};
type = "mysql-database";

targets = [ infrastructure.test2 ];
};

HelloDBService = {
name = "HelloDBService";
pkg = customPkgs.HelloDBServiceWrapper;
dependsOn = {
inherit HelloMySQLDB;
};
type = "tomcat-webapplication";

targets = [ infrastructure.test1 ];
};
};

infrastructure = {
test1 = {
properties = {
hostname = "test1.example.org";
};

containers = {
tomcat-webapplication = {
tomcatPort = 8080;
};
};
};

test2 = {
properties = {
hostname = "test2.example.org";
};

containers = {
tomcat-webapplication = {
tomcatPort = 8080;
};

mysql-database = {
mysqlPort = 3306;
mysqlUsername = "mysqluser";
mysqlPassword = builtins.readFile ./mysqlpw;
};
};
};
}

The above deployment architecture defines has the following properties:

  • The services and infrastructure models are unified into a a single attribute set in which the services attribute refers to the available services and infrastructure attribute to the available deployment targets.
  • The separated distribution concern is completely eliminated -- the mappings in the distribution models are augmented to the corresponding services, by means of the targets attribute. The transformation step basically checks whether no targets property was specified already, and if there is not -- it will consider the targets in the distribution model the deployment targets of the service.

    The fact that the targets attribute will not be overridden, also makes it possible to already specify the targets in the services model, if desired.

In addition to the three deployment models, it is now also possible as an end-user to write a deployment architecture model and use that to automate deployments. The following command-line instruction will deploy a service-oriented system from a deployment architecture model:

$ disnix-env -A architecture.nix

Normalizing the deployment architecture model


Unifying models into a single deployment architecture specification is a good first step in producing an executable specification, but more needs to be done to fully reach that goal.

There are certain deployment properties that are unspecified in the examples shown earlier. For some configuration properties, Disnix provides reasonable default values, such as:

  • Each service can indicate whether they want their state to be managed by Dysnomia (with the property deployState), so that data will automatically be migrated when moving the service from one machine to another. The default setting is false and can be overridden with the --deploy-state parameter.

    If a service does not specify this property then Disnix will automatically propagate the default setting as a parameter.
  • Every target machine in the infrastructure model also has specialized settings for connecting to the target machines, building packages and running tasks concurrently:

    test2 = {
    properties = {
    hostname = "test2.example.org";
    };

    containers = {
    tomcat-webapplication = {
    tomcatPort = 8080;
    };

    mysql-database = {
    mysqlPort = 3306;
    mysqlUsername = "mysqluser";
    mysqlPassword = builtins.readFile ./mysqlpw;
    };

    clientInterface = "disnix-ssh-client";
    targetProperty = "hostname";
    numOfCores = 1;
    system = "x86_64-linux";
    };
    };

    If none of these advanced settings are provided, Disnix will assume that the every target machine has the same system architecture (system) as the coordinator machine (so that the Nix package manager does not have to delegate a build to a machine that has a compatible architecture), we use the Disnix SSH client (disnix-ssh-client) interface executable (clientInterface) to connect to the target machine (using the hostname property as a connection string) and we only run one activity per target machine concurrently: numOfCores.

In addition to unspecified properties (that need to be augmented with default values), we also have properties that are abstract specifications. These specifications need to be translated into more concrete representations:

  • As explained in an older blog post, the targets property -- that maps services to targets -- does not only map services to machines, but also to container services hosted on that machine. In most cases, you will only use one container instance per service type -- for example, running two MySQL DBMS services (e.g. one on TCP port 3306 and another on 3307) is far less common use case scenario.

    If no container mapping is provided, Disnix will do an auto-mapping to a container service that corresponds to the service's type property.

    The MySQLDBService's targets property shown in the last deployment architecture model gets translated into the following property:

    {system, pkgs}:

    rec
    {
    services = rec {
    HelloMySQLDB = {
    name = "HelloMySQLDB";
    ...

    targets = [
    rec {
    selectedContainer = "mysql-database";

    container = {
    mysqlPort = 3306;
    mysqlUsername = "mysqluser";
    mysqlPassword = builtins.readFile ./mysqlpw;
    };

    properties = {
    hostname = "test2.example.org";
    };

    clientInterface = "disnix-ssh-client";
    targetProperty = "hostname";
    numOfCores = 1;
    system = "x86_64-linux";
    }
    ];
    };
    };

    infrastructure = ...
    }

    As may be observed, the target provides a selectedContainer property to indicate to what container the service needs to be deployed. The properties of all the containers that the service does not need to know about are discarded.
  • Another property that needs to be extended is the inter-dependency specifications (dependsOn and connectsTo). Typically, inter-dependency specifications are only specified on a functional level -- a service typically only specifies that it depends on another service disregarding the location where that service may have been deployed.

    If no target location is specified, then Disnix will assume that the service has an inter-dependency on all possible locations where that dependency may be deployed. If an inter-dependency is redundantly deployed, then that service also has an inter-dependency on all redundant replicas.

    The fact that it is also possible to specify the targets of the inter-dependencies, makes it also possible to optimize certain deployments. For example, you can also optimize a service's performance by forcing it to bind to an inter-dependency that is deployed to the same target machine, so that it will not be affected by slow network connectivity.

    The dependsOn property of the HelloDBService will translate to:

    dependsOn = {
    HelloMySQLDB = {
    name = "HelloMySQLDB";
    pkg = customPkgs.HelloMySQLDB;
    dependsOn = {};
    type = "mysql-database";

    targets = [
    {
    selectedContainer = "mysql-database";

    container = {
    mysqlPort = 3306;
    mysqlUsername = "mysqluser";
    mysqlPassword = builtins.readFile ./mysqlpw;
    };

    properties = {
    hostname = "test2.example.org";
    };
    }
    ];
    };
    };

    In the above code fragment, the inter-dependency has been augmented with a targets property corresponding to the targets where that inter-dependency has been deployed to.


The last ingredient to generate an executable specification is building the services from source code so that we can map their build results to the target machines. To accomplish this, Disnix generates two invisible helper attributes for each service:

HelloDBService = {
name = "HelloDBService";
pkg = customPkgs.HelloDBServiceWrapper;
dependsOn = {
inherit HelloMySQLDB;
};
type = "tomcat-webapplication";

...

_systemsPerTarget = [ "x86_64-linux" "x86_64-darwin" ];
_pkgsPerSystems = {
"x86_64-linux" = "/nix/store/91abq...-HelloDBService";
"x86_64-darwin" = "/nix/store/f1ap2...-HelloDBService";
};
};

The above code example shows the two "hidden" properties augmented to the HelloDBService:

  • The _systemsPerTarget specifies for which CPU architecture/operating systems the service must be built. Normally, services are target agnostic and should always yield the same Nix store path (with a build that is nearly bit-identical), but the system architecture of the target machine is an exception to deviate from this property -- it is also possible to deploy the same service to different CPU architectures/operating systems. In such cases the build result could be different.
  • The _pkgsPerSystem specifies for each system architecture, the Nix store path to the build result. A side effect of evaluating the Nix store path is the service also gets built from source code.

Finally, it will compose a deployment architecture model attribute named: targetPackages that refers to a list of Nix store paths to be distributed to each machine in the network:

{
targetPackages = {
test1 = [
"/nix/store/91abq...-HelloDBService"
];

test2 = [
"/nix/store/p9af1...-HelloMySQLDB"
];
};

services = ...
infrastructure = ...
}

The targetPackages attribute is useful for a variety of reasons, as we will see later.

Generating a deployment model


With a normalized architecture model, we can generate an executable specification that I will call a deployment model. The deployment model can be used for executing all remaining activities after the services have been built.

An example of a deployment model could be:

{
profiles = {
test1 = "/nix/store/...-test1";
test2 = "/nix/store/...-test2";
};

services = {
"ekfekrerw..." = {
name = "HelloMySQLDB";
pkg = "/nix/store/...";
type = "mysql-database";
dependsOn = [
];
connectsTo = [
];
};

"dfsjs9349..." = {
name = "HelloDBService";
pkg = "/nix/store/...";
type = "tomcat-webapplication";
dependsOn = [
{ target = "test1";
container = "mysql-database";
service = "ekfekrerw...";
}
];
connectsTo = [
];
};
};

infrastructure = {
test1 = {
properties = {
hostname = "test1.example.org";
};
containers = {
apache-webapplication = {
documentRoot = "/var/www";
};
};
system = "x86_64-linux";
numOfCores = 1;
clientInterface = "disnix-ssh-client";
targetProperty = "hostname";
};
test2 = {
properties = {
hostname = "test2.example.org";
};
containers = {
mysql-database = {
mysqlPort = "3306";
};
};
system = "x86_64-linux";
numOfCores = 1;
clientInterface = "disnix-ssh-client";
targetProperty = "hostname";
};
};

serviceMappings = [
{ service = "ekfekrerw...";
target = "test2";
container = "mysql-database";
}
{ service = "dfsjs9349...";
target = "test1";
container = "tomcat-webapplication";
}
];

snapshotMappings = [
{ service = "ekfekrerw...";
component = "HelloMySQLDB";
container = "mysql-database";
target = "test2";
}
];
}

  • The profiles attribute refers to Nix profiles mapped to target machines and is derived from the targetPackages property in the normalized deployment architecture model. From the profiles property Disnix derives all steps of the distribution phase in which all packages and their intra-dependencies are copied to machines in the network.
  • The services attribute refers to all services that can be mapped to machines. The keys in this attribute set are SHA256 hash codes recursively computed from the Nix store path of the package, the type, and all the inter-dependency mappings. Using hash codes to identify the services makes it possible to easily see whether a service is identical to another or not (by comparing hash codes), so that upgrades can be done more efficiently.
  • The infrastructure attribute is unchanged compared to the deployment architecture model and still stores target machine properties.
  • The serviceMappings attribute maps services in the services attribute set, to target machines in the network stored in the infrastructure attribute set and containers hosted on the target machines.

    From these mappings, Disnix can derive the steps to activate and deactivate the services of which a system is composed, ensure that all dependencies are present and that the services are activated or deactivated in the right order.
  • The snapshotMappings attribute state that for each services mapped to a target machines and container, we also want to migrate the state (by taking and restoring snapshots) if the service gets moved from one machine to another.

Although a deployment model is quite low-level, it is now also possible to manually write one, and deploy it by running:

$ disnix-env -D deployment.nix

disnix-env invokes an external executable called: disnix-deploy that executes the remaining activities of deployment process after the build process succeeds. disnix-depoy as well as the tools that execute individual deployment activities are driven by a manifest files. A manifest file is simply a one-on-one translation of the deployment model in the Nix expression language to XML following the NixXML convention.

Generating a build model


To build the services from source code, Disnix simply uses Nix's build facilities to execute the build. If nothing special has been configured, all builds will be executed on the coordinator machine, but this may not always be desired.

Disnix also facilitates heterogeneous architecture support. For example, if the coordinator machine is a Linux machine and a target machine is macOS (which is not compatible with the Linux system architecture), then Nix should delegate the build to a remote machine that is capable of building it. This is not something that Disnix handles for you out of the box -- you must configure Nix yourself to allow builds to be delegated.

It is also possible to optionally let Disnix delegate builds to the target machines in the network. To make build delegation work, Disnix generates a build model from a normalized deployment architecture model:

{
derivations = [
{ "/nix/store/HelloMySQLDB-....drv"; interface = "test1"; }
{ "/nix/store/HelloDBService-....drv"; interface = "test2"; }
];

interfaces = {
test1 = {
targetAddress = "test1.example.org";
clientInterface = "disnix-ssh-client";
};

test2 = {
targetAddress = "test2.example.org";
clientInterface = "disnix-ssh-client";
};
};
}

The build model shown above defines the following properties:

  • The derivations attribute maps Nix store derivation files (low-level Nix specifications that capture build procedures and dependencies) to machines in the network that should perform the build. This information is used by Disnix to delegate store derivation closure to target machines, use Nix to build the packages remotely, and fetch the build results back to the coordinator machine.
  • The interfaces attribute is a sub set of the infrastructure model that contains the connectivity settings for each target machine.

By running the following command, you can execute a build model to delegate builds to remote machines and fetch their results back:

$ disnix-delegate -B build.nix

If the build delegation option is enabled (for example, by passing --build-on-targets parameter to disnix-env) then Disnix will work a so-called distributed derivation file. Similar to a manifest file, a distributed derivation file is a one-on-one translation from the build model written in the Nix expression language to XML using the NixXML convention.

Packages model


In the normalized architecture model and deployment model, we generate a targetPackages property that we can use to compose Nix profiles with packages from.

For a variety of reasons, I thought it would also be interesting to give the user direct control to use this property. A new feature in Disnix is that you can now also write a packages model:

{pkgs, system}:

{
test1 = [
pkgs.mc
];

test2 = [
pkgs.wget
pkgs.curl
];
}

The above packages model says that we should distribute the Midnight Commander package to the test1 machine, and wget and curl to the test2 machine.

Running the following command will deploy the packages to the target machines in the network:

$ disnix-env -i infrastructure.nix -P pkgs.nix

You can also combine the three common Disnix models with a package model:

$ disnix-env -s services.nix \
-i infrastructure.nix \
-d distribution.nix \
-P pkgs.nix

then Disnix will deploy the services that are distributed to target machines and the supplemental packages defined in the packages model.

The packages model is useful for a variety of reasons:

  • Although it is already possible to use Disnix as a simple package deployer (by setting the types of services to: package), the packages model approach makes it even easier. Furthermore, you also more easily specify sets of packages for target machines. The only thing you cannot do is deploying packages that have inter-dependencies on services, e.g. a client that is preconfigured to connect to a service.
  • The hybrid approach makes it possible to more smooth make a transition to Disnix when automating the deployment process of a system. You can start by managing the dependencies with Nix, then package pieces of the project as Nix packages, then use Disnix to deploy them to remote machines, and finally turn pieces of the system into services that can be managed by Disnix.

Conclusion


In this blog post, I have described a new transformation pipeline in Disnix with well-defined intermediate steps that transforms the input models to a deployment model that is consumable by the tools that implement the deployment activities.

The following diagram summarizes the input models, intermediate models and output models:


The new transformation pipeline has the following advantages over the old infrastructure:

  • The implementation is much easier to maintain and we can more easily reason about its correctness
  • We have access to a broader range of configuration properties. For example, it was previously not possible to select the targets of the inter-dependencies.
  • The output models: deployment and build models are much more easily consumable by the Disnix tools that execute the remainder of the deployment activities. The domain models in the code, also closely resemble the structure of the build and deployment models. This can also be partially attributed to libnixxml that I have described in my previous blog post.
  • We can more easily implement new input models, such as the packages model.
  • The implementation of the disnix-reconstruct tool that reconstructs the manifest on the coordinator machine from metadata stored on the target machines also has become much simpler -- we can get rid of most of the custom code and generate a deployment model instead.

Availability


The new pipeline is available in the current development version of Disnix and will become available for general use in the next Disnix release.

The deployment models described in this blog post are incompatible with the manifest file format used in the last stable release of Disnix. This means that after upgrading Disnix, you need to convert any previous deployment configuration by running the disnix-convert tool.

by Sander van der Burg (noreply@blogger.com) at August 11, 2019 10:31 PM

August 09, 2019

Ollie Charles

Who Authorized These Ghosts!?

Recently at CircuitHub we’ve been making some changes to how we develop our APIs. We previously used Yesod with a custom router, but we’re currently exploring Servant for API modelling, in part due to it’s potential for code generation for other clients (e.g., our Elm frontend). Along the way, this is requiring us to rethink and reinvent previously established code, and one of those areas is authorization.

To recap, authorization is

the function of specifying access rights/privileges to resources related to information security and computer security in general and to access control in particular.

This is in contrast to authentication, which is the act of showing that someone is who they claim to be.

Authorization is a very important process, especially in a business like CircuitHub where we host many confidential projects. Accidentally exposing this data could be catastrophic to both our business and customers, so we take it very seriously.

Out of the box, Servant has experimental support for authorization, which is a good start. servant-server gives us Servant.Server.Experimental.Auth which makes it a doddle to plug in our existing authorization mechanism (cookies & Redis). But that only shows that we know who is asking for resources, how do we check that they are allowed to access the resources?

As a case study, I want to have a look at a particular end-point, /projects/:id/price. This endpoint calculates the pricing options CircuitHub can offer a project, and there are few important points to how this endpoint works:

  1. The pricing for a project depends on the user viewing it. This is because some users can consign parts so CircuitHub won’t order them. Naturally, this affects the price, so pricing is viewer dependent.
  2. Some projects are owned by organizations, and should be priced by the organization as a whole. If a user is a member of the organization that owns the project pricing has been requested for, return the pricing for the organization. If the user is not in the organization, return their own custom pricing.
  3. Private projects should only expose their pricing to superusers, the owner of the project, and any members of the project’s organization (if it’s owned by an organization).

This specification is messy and complicated, but that’s just reality doing it’s thing.

Our first approach was to try and represent this in Servant’s API type. We start with the “vanilla” route, with no authentication or authorization:

Next, we add authorization:

At this point, we’re on our own - Servant offers no authorization primitives (though there are discussions on this topic).

My first attempt to add authorization to this was:

There are two new routing combinators here: AuthorizeWith and CanView. The idea is AuthorizeWith somehow captures the result of authenticating, and provides that information to CanView. CanView itself does some kind of authorization using a type class based on its argument - here Capture "id" ProjectId. The result is certainly something that worked, but I was unhappy with both the complexity to implement it (which is scope to get it wrong), and the lack of actual evidence of authorization.

The latter point needs some expanding. What I mean by “lacking evidence” is that with the current approach, the authorization is essentially like writing the following code:

If I later add more resource access into doThings, what will hold me accountable to checking authorization on those resources? The answer is… nothing! This is similar to boolean blindless - we performed logical check, only to throw all the resulting evidence away immediately.

At this point I wanted to start exploring some different options. While playing around with ideas, I was reminded of the wonderful paper “Ghosts of Departed Proofs”, and it got me thinking… can we use these techniques for authorization?

Ghosts of Departed Proofs

The basic idea of GDP is to name values using higher-rank quantification, and then - in trusted modules - produce proofs that refer to these names. To name values, we introduce a Named type, and the higher-ranked function name to name things:

Note that the only way to construct a Named value outside of this module is to use name, which introduces a completely distinct name for a limited scope. Within this scope, we can construct proofs that refer to these names. As a basic example, we could use GDP to prove that a number is prime:

Here we have our first proof witness - IsPrime. We can witness whether or not a named Int is prime using checkPrime - like the boolean value isPrime this determines if a number is or isn’t prime, but we get evidence that we’ve checked a specific value for primality.

This is the whirlwind tour of GDP, I highly recommend reading the paper for a more thorough explanation. Also, the library justified-containers explores these ideas in the context of maps, where we have proofs that specific items are in the map (giving us total lookups, rather than partial lookups).

GDP and Authorization

This is all well and good, but how does this help with authorization? The basic idea is that authorization is itself a proof - a proof that we can view or interact with resources in a particular way. First, we have to decide which functions need authorization - these functions will be modified to require proof values the refer to the function arguments. In this example, we’ll assume our Servant handler is going to itself make a call to the price :: ProjectId -> UserId -> m Price function. However, given the specification above, we need to make sure that user and project are compatible. To do this, we’ll name the arguments, and then introduce a proof that the user in question can view the project:

But what is this CanViewProject proof?

A first approximation is to treat it as some kind of primitive or axiom. A blessed function can postulate this proof with no further evidence:

This is a good start! Our price function can only be called with a CanViewProject that matches the named arguments, and the only way to construct such a value is to use canViewProject. Of course we could get the implementation of this wrong, so we should focus our testing efforts to make sure it’s doing the right thing.

However, the Agda programmer in me is a little unhappy about just blindly postulating CanViewProject at the end. We’ve got a bit of vision back from our boolean blindness, but the landscape is still blurry. Fortunately, all we have to do is recruit more of the same machinery so far to subdivide this proof into smaller ones:

Armed with these smaller authorization primitives, we can build up our richer authorization scheme:

Now canViewProject just calls out to the other authorization routines to build it’s proof. Furthermore, there’s something interesting here. CanViewProject doesn’t postulate anything - everything is attached with a proof of the particular authorization case. This means that we can actually open up the whole CanViewProject module to the world - there’s no need to keep anything private. By doing this and allowing people to pattern match on CanViewProject, authorization results become reusable - if something else only cares that a user is a super user, we might be able to pull this directly out of CanViewProject - no need for any redundant database checks!

In fact, this very idea can help us implement the final part of our original specification:

Some projects are owned by organizations, and should be priced by the organization as a whole. If a user is a member of the organization that owns the project pricing has been requested for, return the pricing for the organization. If the user is not in the organization, return their own custom pricing.

If we refine our UserBelongsToProjectOrganization proof, we can actually maintain a bit of extra evidence:

Now whenever we have a proof UserBelongsToProjectOrganization, we can pluck out the actual organization that we’re talking about. We also have evidence that the organization owns the project, so we can easily construct a new CanViewProject proof - proofs generate more proofs!

Relationship to Servant

At the start of this post, I mentioned that the goal was to integrate this with Servant. So far, we’ve looked at adding authorization to a single function, so how does this interact with Servant? Fortunately, it requires very little to change. The Servant API type is authorization free, but does mention authentication.

It’s only when we need to call our price function do we need to have performed some authorization, and this happens in the server-side handler. We do this by naming the respective arguments, witnessing the authorization proof, and then calling price:

Conclusion

That’s where I’ve got so far. It’s early days so far, but the approach is promising. What I really like is there is almost a virtual slider between ease and rigour. It can be easy to get carried away, naming absolutely everything and trying to find the most fundamental proofs possible. I’ve found so far that it’s better to back off a little bit - are you really going to get some set membership checks wrong? Maybe. But a property check is probably gonig to be enough to keep that function in check. We’re not in a formal proof engine setting, pretending we are just makes things harder than they need to be.

by Oliver Charles at August 09, 2019 12:00 AM

July 11, 2019

Mayflower

Leveraging NixOS Tests in your Project

NixOS contains infrastructure for building integration tests based on QEMU/KVM virtual machines running NixOS. Tests built on this infrastructure are continuously run on new nixpkgs versions to ensure that NixOS continues to install and boot and that various services continue to operate correctly. This post illustrates how one may test a simple web service using NixOS tests. To have a simple enough example at hand, we wrote a small service in PHP—a classical guestbook in which visitors can leave a message that will then be written to a database and subsequently shown to later visitors of the same site.

July 11, 2019 03:00 PM

July 09, 2019

Hercules Labs

Hercules CI #5 update requiredSystemFeatures, Cachix and Darwin support

What’s new?

We’ve released hercules-ci-agent 0.3 , which brings in Cachix and Darwin (macOS) support alongside with requiredSystemFeatures.

hercules-agent-0.3.0

TOML configuration

Previously the agent was configured via CLI options. Those are now all part of a configuration file formatted using TOML.

Support for binary caches

Added support for Cachix binary caches to share resulting binaries either with the public and/or between developers and/or just multiple agents.

Multi-agent and Darwin support

With binary caches to share derivations and binaries between machines, you’re now able to have multiple agents running.

Sharing binaries between machines takes time (bandwidth) so we recommend upgrading agent hardware over adding more agents.

In addition to Linux, Darwin (macOS) also became a supported deployment platform for the agent.

requiredSystemFeatures support

Derivations are now dispatched also based on requiredSystemFeatures derivation attribute that allows dispatching specific derivations to specific agents.

Cachix 0.2.1

Upgrade via the usual:

$ nix-env -iA cachix -f https://cachix.org/api/v1/install  

The most notable improvement is default compression has been lowered to increase throughput and it’s overridable via ``–compression-level`.

See Changelog for more details.

What’s next?

Known issues we’re resolving:

  • Builds that are in progress while agent is restarted won’t be re-queued. We’re prioritizing this one. Expect a bugfix in next deployment.
  • Evaluation and building is slower with Cachix. We’re going to add bulk query support and upstream caches to mitigate that.
  • Having a lot of failed derivations (>10k) will get frontend unresponsive.
  • Cachix auth tokens for private binary caches are personal. We’ll add support to create tokens specific to a cache.

If you notice any other bugs or annoyances please let us know.

Preview phase

The preview phase will now extend to all subscribers, which is the final phase before we’re launching publicly.

You can also receive our latest updates via Twitter or read the previous development update.

July 09, 2019 12:00 AM

July 04, 2019

Munich NixOS Meetup

NixOS Munich Community Meetup

photoMunich NixOS Meetup

The theme of this meetup is: How do you use the Nix ecosystem in your projects?

We (Mayflower) will showcase how we use Gitlab, Hydra & nixops for Continuous Integration and Deployment. If you want to share your setup with us, just show up and show us. :-)

ATTENTION: The Mayflower Munich office has moved to Laim! Please note the new address!

Food and beverages will be provided. We will BBQ on our new rooftop terrace!

München 80687 - Germany

Thursday, July 4 at 6:30 PM

25

https://www.meetup.com/Munich-NixOS-Meetup/events/262224658/

July 04, 2019 03:24 PM

May 15, 2019

Hercules Labs

gitignore for Nix

Abstract

Nix, when used as a development build tool, needs to solve the same problem that git does: ignore some files. We’ve extended nix-gitignore so that Nix can more reliably use the configuration that you’ve already written for git.

Introduction

When you tell Nix to build your project, you need to tell it which source files to build. This is done by using path syntax in a derivation or string interpolation.

mkDerivation {
  src = ./vendored/cowsay;
  postPatch = ''
    # Contrived example of using a file in string interpolation
    # The patch file is put in /nix/store and the interpolation
    # produces the appropriate store path.
    patch -lR ${./cowsay-remove-alpaca.patch}
  '';
  # ...
}

This works well, until you find that Nix unexpectedly rebuilds your derivation because a temporary, hidden file has changed. One of those files you filtered out of your git tree with a ‘gitignore’ file…

Nix, as a build tool or package manager, was not designed with any specific version control system in mind. In fact it predates any dominance of git, because Nix’s general solution to the file ignoring problem, filterSource, was already implemented in 2007.

Over the last two to three years, various people have written functions to reuse these gitignore files. We have been using an implementation by @siers over the last couple of months and it has served us well, until we had a gitignore file that wasn’t detected because it was in a parent directory of the source directory we wanted to use.

I was nerd sniped.

Two months later, I finally got around to the implementation and I’m happy to announce that it solves some other problems as well. It reuses the tested rules by siers, doesn’t use import from derivation and can read all the files that it needs to.

Usage

You can import the gitignoreSource function from the repo like below, or use your favorite pinning method.

{ pkgs ? import <nixpkgs> {} }
let
  inherit (pkgs.stdenv) mkDerivation;
  inherit (import (builtins.fetchTarball "https://github.com/hercules-ci/gitignore/archive/master.tar.gz") { }) gitignoreSource;
in
mkDerivation {
  src = gitignoreSource ./vendored/cowsay;
  postPatch = ''
    patch -lR ${./cowsay-remove-alpaca.patch}
  '';
  # ...
}

That’s all there is to it.

It also composes with cleanSourceWith if you like to filter out some other files as well.

Comparison

Here’s a comparison with the pre-existing implementation I found.

The latest up to date comparison table is available on the repo.

Feature \ Implementation cleanSource siers siers recursive icetan Profpatsch numtide this project
Ignores .git ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
No special Nix configuration ✔️ ✔️ ✔️ ✔️ ✔️   ✔️
No import from derivation ✔️ ✔️   ✔️ ✔️ ✔️ ✔️
Uses subdirectory gitignores     ✔️     ✔️ ✔️
Uses parent gitignores           ✔️ ? ✔️
Uses user gitignores           ✔️ ✔️
Has a test suite   ✔️ ✔️ ✔️   ? ✔️
Works with restrict-eval / Hydra ✔️ ✔️   ✔️ ✔️   ✔️
Included in nixpkgs ✔️ ✔️ ✔️        
  Legend
✔️ Supported
✔️ ? Probably supported
  Not supported
? Probably not supported
- Not applicable or depends

Inclusion in Nixpkgs

I think it would be really nice to have this function in Nixpkgs, but it needs to be tested in practice first. This is where you can help out! Please give the project (GitHub) a spin and leave a thumbs up if it worked for you (issue).

Closing thoughts

I am happy to contribute to the friendly and inventive Nix community. Even though this gitignore project is just a small contribution, it wouldn’t have been possible without the ideas and work of siers, icetan, and everyone behind Nix and Nixpkgs in general.

As a company we are working hard to make good products to support the community and companies that want to use Nix. One of our goals is to keep making contributions like this, so please try our binary cache as a service, which is free for open source and just as easy to set up privately for companies. If you have an interest in our Nix CI, please subscribe.

– Robert

May 15, 2019 12:00 AM

May 14, 2019

Hercules Labs

Hercules CI #3 development update

What’s new?

Precise derivations improvements

Dependency failure tree

If a dependency failed for an attribute, you can now explore the dependency stack down to the actual build failure.

There’s also a rebuild button to retry the build for the whole stack, from the failed dependency down up to and including the build you clicked. We’ve addressed some of the styling issues visible on smaller screens.

Fixed an issue where users would end up being logged out

hercules-ci-agent 0.2

  • use gitignore instead of nix-gitignore
  • fix build on Darwin
  • limit internal concurrency to max eight OS threads for beefier machines
  • show version on --help
  • build against NixOS 19.03 as default
  • propagate agent information to agent view: Nix version, substituters, platform and Nix features

Focus for the next sprint

Cachix and thus Darwin support

The last bits missing (besides testing) are sharing derivations and artifacts between agents using cachix and the ease of Darwin agent deployment with accompanying documentation.

Stuck jobs when restarting the agent

Currently when you restart an agent that is doing work, jobs claimed by the agent will appear stuck in the queue. This sprint is planned to ship a way to remedy the issue manually via the UI. Later on it will be automatically handled by agent ping-alive.

Preview phase

Once we’re done with Darwin and Cachix support, we’ll hand out preview access to everyone who will have signed up for preview access.

You can also receive our latest updates via Twitter or read the previous development update.

May 14, 2019 12:00 AM

May 06, 2019

Matthew Bauer

Nixpkgs macOS Stdenv Updates

Over the past couple of months, I have been working on updating the macOS stdenv in Nixpkgs. This has significant impact on users of Nix/Nixpkgs on macOS. So, I want to explain what’s being updated, what the benefits are, and how we can minimize breakages.

1 macOS/Darwin stdenv changes

First, to summarize the changes that impact stdenv and the Darwin infrastructure. The PR is available at NixOS/nixpkgs PR #56744. This update has been in the works for the last few months, and is currently in the staging-next branch, waiting to be merged in NixOS/nixpkgs PR #60491. It should land on master and nixpkgs-unstable in the next few days. The main highlights are —

  • Change default LLVM toolchain from 5 to 7. LLVM 5 stdenv is still available through llvmPackages_5.stdenv attribute path.
  • Upgrade Apple SDK from 10.10 to 10.12.
  • Update libSystem symbols from 10.10 (XNU 3789.1.32) to 10.12 (XNU 3789.1.32).
  • Removed old patches to support old stdenv in Qt 5 and elsewhere.

These macOS SDK upgrades are equivalent to setting -mmacosx-version-min to 10.12 in XCode. As a result, we will break compatibility with macOS before 10.12.

2 Why do we need to set a minimum macOS version?

Without knowing internals of Nixpkgs, it might not be clear why we need to set a minimum macOS version. For instance with Linux, we are able to support any Linux kernel in Nixpkgs without any problem. The answer to this requires some understanding of how the kernel and userspace function.

Nixpkgs is able to support multiple Linux kernels because we can use multiple Libc’s at one time. For any executable, a Nix closure will include both its own Libc and the dynamic linker in its closure. This works in Linux where multiple Libc’s can be used, but not on macOS where only one Libc is available.

In short, Linux and macOS deal with compatibility between built binaries in different ways. They represent two opposite ends in how Unix-like kernels maintain compatibility with their userspace binaries.

2.1 Linux syscall compatibility

The kernel is responsible for managing core operating system functions such as start-up, memory management, device abstractions, and process isolation. For it to function, the kernel needs to interact with the rest of the operating system which is collectively referred to as “userspace”. Executables in userspace use “syscalls” to tell the kernel what to do. These syscalls are very low-level and usually not called directly by a process. Instead, an abstraction layer is provided by the standard C library, or Libc.

Linux is unique among operating systems due to the fact that the Kernel and Libc are developed independently. Linux is maintained by creator Linus Torvalds and a community of contributors. Glibc, the most popular Libc for Linux, is maintained by the GNU project. As a result, Linux has a strong separation between Syscalls and Libc.

Linux does not tie itself to any specific Libc. Even though Glibc is used in almost all distros, many alternatives are available. For instance, Musl provides a more lightweight version of Glibc, while Bionic is the Libc used in the Android operating system. In addition, multiple versions of each of these Libc’s can be used on any one kernel, even at the same time. This can become very common when using multiple Nixpkgs versions at one time.

To accomplish this, Linux provides a stable list of syscalls that it has maintained across many versions. This is specified for i386 at arch/x86/entry/syscalls/syscall_32.tbl in the kernel tree. The syscalls specified here are the interface through which the Libc communicates with the kernel. As a result, applications built in 1992 can run on a modern kernel, provided it comes with copies of all its libraries1.

2.2 macOS Libc compatibility

The macOS Libc is called libSystem. It is available on all macOS systems at /usr/lib/libSystem.B.dylib. This library is the main interface that binary compatibility is maintained in macOS. Unlike Linux, macOS maintains a stable interface in libSystem that all executables are expected to link to. This interface is guaranteed by Apple to be stable between versions.

In Nixpkgs, we maintain this compatibility through a list of symbols that are exported by libSystem. This is a simple text list and is available for viewing at NixOS/nixpkgs/pkgs/os-specific/darwin/apple-source-releases/Libsystem/system_c_symbols. The symbol list is created by listing symbols (nm) on the minimum macOS version that we support (for my PR, 10.12). We do some linking tricks to ensure that everything that we build in Nixpkgs only contains those symbols. This means that we can reproducibly build on newer versions of macOS, while maintaining compatibility with older macOS versions. Unfortunately, newer symbols introduced in later versions cannot be used even on systems that have those symbols.

A side effect of macOS design, is that fully static executables are not supported in macOS as they are on Linux. Without a stable syscall interface, there is nothing to provide compatibility between versions. As a result, Apple does not support this type of linking2.

There is no mandated reason why we need to use libSystem directly. In fact, some languages like Go have attempted to instead use the syscall interface directly. There is no reason why this couldn’t work, however, upgrades between versions will almost certainly break binaries. Go eventually abandoned this scheme in time for Go 1.12 (proposed by Nixpkgs macOS contributor copumpkin!)

2.3 Others

Some other examples may be useful. They mostly fall on one side or the other of the Syscall / Libc divide —

  • FreeBSD - breaks syscall compatibility between major releases, should use Libc for longterm binary compatibility.
  • OpenBSD - similarly, changes syscall interface, perhaps even more often than FreeBSD3.
  • NetBSD - apparently has maintained syscall compatibility since 1992. 4
  • Windows, Solaris, Fuchsia - I cannot find any information on these and how they handle binary compatibility.

2.4 LLVM triple

As a side note, this difference can be clearly seen in how we specify target systems. The LLVM triple is a 3 or 4-part string specifying what we want to build for. The parts of the triple correspond to:

<cpu>-<vendor>-<kernel>-<abi>
  • <cpu> — the CPU architecture that we are building for. Examples include x86_64, aarch64, armv7l, etc.
  • <vendor> — an arbitrary string specifying the vendor for the toolchain. In Nixpkgs, this should always be unknown.
  • <kernel> — the kernel to build for (linux).
  • <abi> — the kernel ABI to use. On Linux, this corresponds to the Libc we are using (gnu for Glibc, musl for Musl, android for Bionic).

When building for Linux, we can build for any version of Linux at one time. No version information is required. In addition, we must specify what “ABI” we want to use. In Nix, this is not very important because the Libc is provided by the closure. In fact, Nix has its own version of the LLVM triple called a Nix system tuple that omits the <abi> portion altogether! It corresponds to <cpu>-<kernel> from the LLVM triple.

In comparison, when building for BSDs, we must specify which version of the kernel we are building for. In addition, we leave off the <abi> portion, because there is only one Libc available for these platforms. They are even included in the same tree as the kernel. Examples of BSD triples include,

  • aarch64-apple-darwin16.0.0
  • x86_64-unknown-freebsd12.0
  • i386-unknown-openbsd5.8
  • armv7l-unknown-netbsd7.99

3 Compatibility table

Looking through the old versions, I’ve compiled a list of what I think are the corresponding macOS versions for each Nixpkgs release. As you can see, we try to support at least 3 previous macOS releases. This also happens to be about what Apple supports through security updates5.

Nixpkgs release macOS version
19.09 10.12, 10.13, 10.14, 10.15?
19.03 10.116, 10.12, 10.13, 10.14
18.09 10.11, 10.12, 10.13, 10.14
18.03 10.11, 10.12, 10.13, 10.14
17.09 10.10, 10.11, 10.12, 10.13
17.03 10.10, 10.11, 10.12
16.09 10.10, 10.11, 10.12
16.03 10.9?, 10.10, 10.11, 10.12

We know that some users are stuck on older versions of macOS due to reasons outside of their control. As a result, we will try to support the 19.03 branch for a little bit longer than is usually done. If your organization uses 10.11, it might be a good idea to update to a newer version along with your update to Nixpkgs 19.09.

4 Conclusion

My main goal has been to show better how Nixpkgs and macOS system interact. I got a little bit sidetracked exploring differences in binary compatibility between different operating systems. But, this should help users to better understand the differences in how macOS and Linux works in relation to Nixpkgs.

Footnotes:

1

It would be interesting to test this in practice. Finding a Libc that would work might be the hardest part. Even better if we could use Nix’s closures!

3

According to the_why_of_y on Hacker News, https://news.ycombinator.com/item?id=14011662

5

macOS updates come out about every year and Apple offers about 3 months support. More information is available at https://apple.stackexchange.com/questions/47664/what-is-apples-policy-for-supporting-security-updates-on-older-versions-of-os-x.

6

There is an issue with building on 10.11 with the new swift-corelibs derivation. As a result, you need to use prebuilt version to avoid this issue.

May 06, 2019 12:00 AM

April 30, 2019

Hercules Labs

Sprint #2 development update

What’s new in sprint #2?

Precise derivations

Agent 0.2 will communicate the structure of the derivation closure to the service, which allows us to traverse the derivation tree and dispatch each derivation to multiple agents.

Neither source or binary data used by Nix on the agent is ever sent to the service.

We will release agent 0.2 after more testing and UI improvements. as we’re still doing testing and UI improvements.

Git-rich metadata

Each job now includes a branch name, commit message and the committer:

Job rich metadata

Job page information

The job page shows information that triggered the build and timing information:

Job page

Focus for sprint #3

Cachix support

We’ve already made progress of parsing Cachix configuration, but there’s the actual pushing of derivations to be done.

Darwin / Features support

Now that precise derivations are working, they need to get aware of platforms for Darwin support. Same goes for infamous Nix “features”, which work like labels that can be used to dispatch individual derivations to specific groups of agents.

Preview phase

You’re still in time to sign up for preview access or follow us on Twitter as we will be expanding access in the coming weeks.

April 30, 2019 12:00 AM

April 16, 2019

Hercules Labs

Sprint #1 development update

Two weeks ago we launched preview access of our CI for Nix users. Thank you all for giving us the feedback through the poll and individually. We are overwhelmed with the support we got.

Focus of the preview launch was to build a fast, reliable, easy to use CI. Today you can connect your github repository in a few clicks, deploy an agent and all your commits are being tested with their status reported to GitHub.

In our latest sprint we have fixed a few issues, mainly centered around usability and clarity of what’s going on with your projects.

The following features were shipped:

The following bugs fixes were shipped:

  • When there is no agent available, enqueued jobs will show instructions to setup one

  • To prevent CSRF we’ve tightened SameSite cookie from Lax to Strict

  • CDN used to serve stale assets due to caching misconfiguration

  • Numerous fixes to the UI:

    • breadcrumbs now allow user to switch account or just navigate to it
    • no more flickering when switching pages
    • some jobs used to be stuck in Building phase
    • more minor improvements

In our upcoming sprint, #2 we will focus on:

  • Fine-grained dispatch of individual derivations (instead of just top-level derivation closures from attributes as we shipped in the preview) - what follows is testing and presenting derivations in the UI

  • Currently we only store the git revision for each job, which will be expanded to include more metadata like branch name, commit message, author, etc

  • If time allows, preliminary cachix support

You’re still in time to sign up for preview access as we will be expanding access in the following weeks.

April 16, 2019 12:00 AM