NixOS Planet

August 22, 2019

Hercules Labs

Pre-commit git hooks with Nix

pre-commit manages a set of hooks that are executed by git before committing code:

pre-commit.png

Common hooks range from static analysis or linting to source formatting.

Since we’re managing quite a couple of repositories, maintaining the duplicated definitions became a burden.

Hence we created:

nix-pre-commit-hooks

The goal is to manage these hooks with Nix and solve the following:

  • Simpler integration into Nix projects, doing the wiring up behind the scenes

  • Provide a low-overhead build of all the tooling available for the hooks to use (calling nix-shell for every check does bring some latency when committing)

  • Common package set of hooks for popular languages like Haskell, Elm, etc.

  • Two trivial Nix functions to run hooks as part of development and on your CI

Currently the following hooks are provided:

Nix

  • canonix: a Nix formatter (currently incomplete, requiring some manual formatting as well)

Haskell

Elm

Shell

We encourage everyone to contribute additional hooks.

Installation

See project’s README for latest up-to-date installation steps.


What we do

Automated hosted infrastructure for Nix, reliable and reproducible developer tooling, to speed up adoption and lower integration cost. We offer Continuous Integration and binary caches.

August 22, 2019 12:00 AM

August 21, 2019

Matthew Bauer

All the versions with Nix

1 Background

In Channel Changing with Nix, I described how to move between channels in Nix expressions. This provides an easy way to work with multiple versions of Nixpkgs. I was reminded of this post after seeing a comment by jolmg on Hacker News. The comment suggested we should have a way to use multiple versions of packages seamlessly together. It suggested that we should use commits to differentiate versions, but I suggested that stable channels would work much better.

So, as a follow up to my Channel Changing post, I want to show how you can use a quick Nix mockup to accomplish this. Like the previous post, it will come with a Nix snippet that you can try out yourself.

2 Code

So, what follows is some code that I wrote up that lets you find package versions in an easier way. It is also available for download at https://matthewbauer.us/generate-versions.nix.

{ channels ? [ "19.03" "18.09" "18.03" "17.09" "17.03"
         "16.09" "16.03" "15.09" "14.12" "14.04" "13.10" ]
, attrs ? builtins.attrNames (import <nixpkgs> {})
, system ? builtins.currentSystem
, args ? { inherit system; }
}: let

  getSet = channel:
    (import (builtins.fetchTarball "channel:nixos-${channel}") args).pkgs;

  getPkg = name: channel: let
    pkgs = getSet channel;
    pkg = pkgs.${name};
    version = (builtins.parseDrvName pkg.name).version;
  in if builtins.hasAttr name pkgs && pkg ? name then {
    name = version;
    value = pkg;
  } else null;

in builtins.listToAttrs (map (name: {
  inherit name;
  value = builtins.listToAttrs
    (builtins.filter (x: x != null)
      (map (getPkg name) channels));
}) attrs)

This Nix expression generates an index of each package from all 11 releases of Nixpkgs that have occurred since October 2010. For every package, each version that came with a release is included and put into a map. The map uses the version as a key and the package as its value, preferring the newer release when versions conflict.

This is all done lazily, because that’s how Nix works. Still, it will take a little while at first to evaluate because we need to parse all 11 releases! Remarkably, this expression uses only Nix builtins, and requires no special library function.

3 Usage

Working with this Nix expression is extremely interesting, and I’ve included some examples of how to work with it. They should all be usable on a Linux machine (or maybe macOS) with Nix installed.

3.1 Query package versions

You can query what package versions are available through Nix’s builtins.attrNames function. For example,

$ nix eval "(builtins.attrNames (import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).emacs)"
[ "24.3" "24.4" "24.5" "25.3" "26.1" ]

This shows us that there are 5 versions of Emacs. This is kind of interesting because it means that there were at least 6 duplicate versions of Emacs between our release channels. Unfortunately, a few versions of Emacs are notably missing including Emacs 25.1 and Emacs 25.2. Emacs 24.2 was released almost a year before the first stable Nixpkgs release! As time goes on, we should collect more of these releases.

3.2 Running an old version

A shown above, there are 5 versions of Emacs available to us. We can run Emacs 24.3 with a fairly short command:

$ LC_ALL=C nix run "(import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).emacs.\"24.3\"" -c emacs

LC_ALL=C is needed on Linux to avoid the old Glibc trying to load the newer, incompatible locales that may be included with your system. This is an unfortunate problem with Glibc including breaking changes between releases. It also makes me want use to switch to Musl some time soon! I’ve also noticed some incompatibilities with GTK icons that appear to come from the gdk-pixbuf module. More investigation is needed on why this is the case.

This will not work on macOS because we did not have Emacs working on macOS back then! macOS users can try Emacs 25.3. This looks very similar to the above:

$ nix run "(import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).emacs.\"25.3\"" -c emacs

3.3 Firefox

Another example using Firefox is pretty neat. The code is very similar to Emacs:

$ nix eval "(builtins.attrNames (import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).firefox)"
[ "25.0.1" "34.0.5" "39.0.3" "45.0" "48.0.2" "51.0.1" "55.0.3" "59.0.2" "63.0.3" "66.0.3" "68.0.2" ]

We get all 11 releases with unique Firefox versions this time.

You can run Firefox 25.0.1 using this command:

$ LC_ALL=C nix run "(import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).firefox.\"25.0.1\"" -c firefox

Amazing how notably Firefox has changed since then!

3.4 Blender

Another example using Blender. The code is very similar to the two above:

$ nix eval "(builtins.attrNames (import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).blender)"
[ "2.67" "2.70" "2.72b" "2.75a" "2.77a" "2.78c" "2.79" "2.79a" "2.79b" ]

You can run Blender 2.67 using this command:

$ LC_ALL=C nix run "(import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).blender.\"26.7\"" -c blender

4 Rationale

The reason that channels work better than commits is because every commit in Nixpkgs is not guaranteed to work on its own. Some may be missing security patches, configuration changes, or worse may just not work with other versions of packages. In addition, there are just too many commits to work with effectively. On the other hand, Nixpkgs release stable channels every 6 months, and we have a long vetting process of ensuring the stabilized channel works well.

The main drawback the 6-month channels have is that we don’t have every version released of package. If the version you want is missing in a release, you are out of luck. But, the 6-month window tends to pick up a lot of packages and we end up with almost every major version of popular software. My philosophy is not all releases are worth keeping. Some contain critical security flaws, contain major bugs, and might not work well with other software. The 6-month window is good enough for me. Perhaps in the future we can increase Nixpkgs release cadence to 3-month or 1-month, but the maintainers are not quite ready for that yet.

5 Conclusion

This has hopefully shown how Nix’s functional dependency model makes it very easy to switch between versions of packages. This is builtin to Nix, but you need some scripts to really use this well. Our 6-month release window is an arbitrary choice, but tends to pick up a lot of useful versions in the mean time.

August 21, 2019 12:00 AM

August 19, 2019

Mayflower

The NixOS RFC Process

History The NixOS RFC process was established in March 2017, initiated by zimbatm, teh and Moretea in order to standardise a process to find an agreement on larger changes to Nix/NixOS/Nixpkgs and the ecosystem in general. Over the following one and a half years a few uncontroversial RFCs were merged but for most of the RFCs that needed further discussion nobody felt responsible to make a decision. That is the reason RFC 36 was written in collaboration with most core Nix/Nixpkgs members at the last NixCon and merged in December 2018 in order to streamline the process and define clearer responsibilities.

August 19, 2019 02:30 PM

August 11, 2019

Sander van der Burg

A new input model transformation pipeline for Disnix

As explained in earlier blog posts, Disnix (as well as other tools in the Nix project) are driven by declarative specifications -- instead of describing the activities that need to be carried out to deploy a system (such as building and distributing packages), we specify all the relevant properties of a service-oriented system:

  • The services model describes all the services that can be deployed to target machines in a network, how they can be built from their sources, how they depend on each other and what their types are, so the the deployment system knows how they can be activated.
  • The infrastructure model captures all target machines in the network, their properties, and the containers they provide. Containers in a Disnix-context are services that manage the life-cycle of a component, such as an application server, service manager or database management service (DBMS).
  • The distribution model maps services to containers on the target machines.

By running the following command-line instruction:

$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix

Disnix infers all the activities that need to be executed to get the system in a running state, such as building packages from source code (or downloading substitutes from a binary cache), the distribution of packages, the activation of a system and taking and restoring state snapshots.

Conceptually, this approach may sound very simple but the implementation that infers the deployment process is not. Whilst the input models are declarative, they are not executable -- there is not a one-on-one mapping between properties in the input models and the activities that Disnix needs to carry out.

To be able to execute deployment activities, Disnix transforms the three input models into a single declarative specification (called a deployment manifest file) that contains one-on-one mappings between deployment artifacts (e.g. Nix profiles, Nix packages and snapshots) and deployment targets (the target machines and/or container services). The transformation pipeline fills in the blanks with default settings, and transforms the input models into several intermediate representations, before it gets transformed into the manifest file.

So far, the intermediate representations and final result were never well defined. Instead, they have organically evolved and were heavily revised several times. As a result of adding new features and not having well defined representations, it became very hard to make changes and reason about the correctness of the models.

In my previous blog post, I have developed libnixxml to make the integration between a data model defined in the Nix expression language and external tools (that implement deployment activities that Nix does not support) more convenient. I am primarily using this library to simplify the integration of manifest files with Disnix tools.

As an additional improvement, I have revised the transformation pipeline, with well-defined intermediate representations. Besides a better quality transformation pipeline with well-defined intermediate stages, the Disnix toolset can now also take the intermediate model representations as input parameters, which is quite convenient for integration with external tooling and experimentation purposes. Furthermore, a new input model has been introduced.

In the blog post, I will describe the steps in the transformation pipeline, and the intermediate representations of the deployment models.

Separated concerns: services, infrastructure, distribution models


As explained earlier in this blog post, Disnix deployment are primarily driven by three input models: the services, infrastructure and distribution models. The reason why I have picked three input models (as opposed to a single configuration file) is to separate concerns and allow these concerns to be reused in different kinds of deployment scenarios.

For example, we can write a simple services model (services.nix) that describes two services that have an inter-dependency on each other:

{distribution, invDistribution, system, pkgs}:

let customPkgs = import ../top-level/all-packages.nix {
inherit system pkgs;
};
in
rec {
HelloMySQLDB = {
name = "HelloMySQLDB";
pkg = customPkgs.HelloMySQLDB;
dependsOn = {};
type = "mysql-database";
};

HelloDBService = {
name = "HelloDBService";
pkg = customPkgs.HelloDBServiceWrapper;
dependsOn = {
inherit HelloMySQLDB;
};
type = "tomcat-webapplication";
};
}

The above services model captures two services with the following properties:

  • The HelloMySQLDB services refers to a MySQL database backend that stores data. The type property: mysql-database specifies which Dysnomia module should be used to manage the lifecycle of the service. For example, the mysql-database Dysnomia module will create the database on initial startup.
  • The HelloDBService is a web service that exposes the data stored in the database backend to the outside it world. Since it requires the presence of a MySQL database backend and needs to know where it has been deployed, the database backend been declared as an inter-dependency of the service (by means of the dependsOn attribute).

    The tomcat-webapplication type specifies that Disnix should use the Apache Tomcat Dysnomia module, to activate the corresponding Java-based web service inside the Apache Tomcat servlet container.

The services model captures the aspects of a service-oriented system from a functional perspective, without exposing much of the details of the environments they may run in. This is intentional -- the services are meant to be deployed to a variety of environments. Target agnostic services make it possible, for example, to write an infrastructure model of a test environment (infrastructure-test.nix):

{
test1 = {
properties = {
hostname = "test1.example.org";
};

containers = {
tomcat-webapplication = {
tomcatPort = 8080;
};
};
};

test2 = {
properties = {
hostname = "test2.example.org";
};

containers = {
tomcat-webapplication = {
tomcatPort = 8080;
};

mysql-database = {
mysqlPort = 3306;
mysqlUsername = "mysqluser";
mysqlPassword = builtins.readFile ./mysqlpw;
};
};
};
}

and a distribution model that maps the services to the target machines in the infrastructure model (distribution-test.nix):

{infrastructure}:

{
HelloMySQLDB = [ infrastructure.test2 ];
HelloDBService = [ infrastructure.test1 ];
}

With these three deployment models, we can deploy a system to a test environment, by running:

$ disnix-env -s services.nix \
-i infrastructure-test.nix \
-d distribution-test.nix

and later switch to a production environment using the same functional services model, after the system has been properly validated in the test environment:

$ disnix-env -s services.nix \
-i infrastructure-prod.nix \
-d distribution-prod.nix

Similarly, we can adjust the distribution model to only deploy a sub set of the services of a system for, say, experimentation purposes.

Unifying the input models into a single specification: the deployment architecture model


The first step in transforming the input models into a single executable specification, is unifying the specifications into one single declarative specification, that I will call the deployment architecture model. The name is derived from the concept of deployment architectures in software architecture terminology:
a description that specifies the distribution of software components over hardware nodes.

A Disnix deployment architecture model may look as follows:

{system, pkgs}:

let customPkgs = import ../top-level/all-packages.nix {
inherit system pkgs;
};
in
rec {
services = rec {
HelloMySQLDB = {
name = "HelloMySQLDB";
pkg = customPkgs.HelloMySQLDB;
dependsOn = {};
type = "mysql-database";

targets = [ infrastructure.test2 ];
};

HelloDBService = {
name = "HelloDBService";
pkg = customPkgs.HelloDBServiceWrapper;
dependsOn = {
inherit HelloMySQLDB;
};
type = "tomcat-webapplication";

targets = [ infrastructure.test1 ];
};
};

infrastructure = {
test1 = {
properties = {
hostname = "test1.example.org";
};

containers = {
tomcat-webapplication = {
tomcatPort = 8080;
};
};
};

test2 = {
properties = {
hostname = "test2.example.org";
};

containers = {
tomcat-webapplication = {
tomcatPort = 8080;
};

mysql-database = {
mysqlPort = 3306;
mysqlUsername = "mysqluser";
mysqlPassword = builtins.readFile ./mysqlpw;
};
};
};
}

The above deployment architecture defines has the following properties:

  • The services and infrastructure models are unified into a a single attribute set in which the services attribute refers to the available services and infrastructure attribute to the available deployment targets.
  • The separated distribution concern is completely eliminated -- the mappings in the distribution models are augmented to the corresponding services, by means of the targets attribute. The transformation step basically checks whether no targets property was specified already, and if there is not -- it will consider the targets in the distribution model the deployment targets of the service.

    The fact that the targets attribute will not be overridden, also makes it possible to already specify the targets in the services model, if desired.

In addition to the three deployment models, it is now also possible as an end-user to write a deployment architecture model and use that to automate deployments. The following command-line instruction will deploy a service-oriented system from a deployment architecture model:

$ disnix-env -A architecture.nix

Normalizing the deployment architecture model


Unifying models into a single deployment architecture specification is a good first step in producing an executable specification, but more needs to be done to fully reach that goal.

There are certain deployment properties that are unspecified in the examples shown earlier. For some configuration properties, Disnix provides reasonable default values, such as:

  • Each service can indicate whether they want their state to be managed by Dysnomia (with the property deployState), so that data will automatically be migrated when moving the service from one machine to another. The default setting is false and can be overridden with the --deploy-state parameter.

    If a service does not specify this property then Disnix will automatically propagate the default setting as a parameter.
  • Every target machine in the infrastructure model also has specialized settings for connecting to the target machines, building packages and running tasks concurrently:

    test2 = {
    properties = {
    hostname = "test2.example.org";
    };

    containers = {
    tomcat-webapplication = {
    tomcatPort = 8080;
    };

    mysql-database = {
    mysqlPort = 3306;
    mysqlUsername = "mysqluser";
    mysqlPassword = builtins.readFile ./mysqlpw;
    };

    clientInterface = "disnix-ssh-client";
    targetProperty = "hostname";
    numOfCores = 1;
    system = "x86_64-linux";
    };
    };

    If none of these advanced settings are provided, Disnix will assume that the every target machine has the same system architecture (system) as the coordinator machine (so that the Nix package manager does not have to delegate a build to a machine that has a compatible architecture), we use the Disnix SSH client (disnix-ssh-client) interface executable (clientInterface) to connect to the target machine (using the hostname property as a connection string) and we only run one activity per target machine concurrently: numOfCores.

In addition to unspecified properties (that need to be augmented with default values), we also have properties that are abstract specifications. These specifications need to be translated into more concrete representations:

  • As explained in an older blog post, the targets property -- that maps services to targets -- does not only map services to machines, but also to container services hosted on that machine. In most cases, you will only use one container instance per service type -- for example, running two MySQL DBMS services (e.g. one on TCP port 3306 and another on 3307) is far less common use case scenario.

    If no container mapping is provided, Disnix will do an auto-mapping to a container service that corresponds to the service's type property.

    The MySQLDBService's targets property shown in the last deployment architecture model gets translated into the following property:

    {system, pkgs}:

    rec
    {
    services = rec {
    HelloMySQLDB = {
    name = "HelloMySQLDB";
    ...

    targets = [
    rec {
    selectedContainer = "mysql-database";

    container = {
    mysqlPort = 3306;
    mysqlUsername = "mysqluser";
    mysqlPassword = builtins.readFile ./mysqlpw;
    };

    properties = {
    hostname = "test2.example.org";
    };

    clientInterface = "disnix-ssh-client";
    targetProperty = "hostname";
    numOfCores = 1;
    system = "x86_64-linux";
    }
    ];
    };
    };

    infrastructure = ...
    }

    As may be observed, the target provides a selectedContainer property to indicate to what container the service needs to be deployed. The properties of all the containers that the service does not need to know about are discarded.
  • Another property that needs to be extended is the inter-dependency specifications (dependsOn and connectsTo). Typically, inter-dependency specifications are only specified on a functional level -- a service typically only specifies that it depends on another service disregarding the location where that service may have been deployed.

    If no target location is specified, then Disnix will assume that the service has an inter-dependency on all possible locations where that dependency may be deployed. If an inter-dependency is redundantly deployed, then that service also has an inter-dependency on all redundant replicas.

    The fact that it is also possible to specify the targets of the inter-dependencies, makes it also possible to optimize certain deployments. For example, you can also optimize a service's performance by forcing it to bind to an inter-dependency that is deployed to the same target machine, so that it will not be affected by slow network connectivity.

    The dependsOn property of the HelloDBService will translate to:

    dependsOn = {
    HelloMySQLDB = {
    name = "HelloMySQLDB";
    pkg = customPkgs.HelloMySQLDB;
    dependsOn = {};
    type = "mysql-database";

    targets = [
    {
    selectedContainer = "mysql-database";

    container = {
    mysqlPort = 3306;
    mysqlUsername = "mysqluser";
    mysqlPassword = builtins.readFile ./mysqlpw;
    };

    properties = {
    hostname = "test2.example.org";
    };
    }
    ];
    };
    };

    In the above code fragment, the inter-dependency has been augmented with a targets property corresponding to the targets where that inter-dependency has been deployed to.


The last ingredient to generate an executable specification is building the services from source code so that we can map their build results to the target machines. To accomplish this, Disnix generates two invisible helper attributes for each service:

HelloDBService = {
name = "HelloDBService";
pkg = customPkgs.HelloDBServiceWrapper;
dependsOn = {
inherit HelloMySQLDB;
};
type = "tomcat-webapplication";

...

_systemsPerTarget = [ "x86_64-linux" "x86_64-darwin" ];
_pkgsPerSystems = {
"x86_64-linux" = "/nix/store/91abq...-HelloDBService";
"x86_64-darwin" = "/nix/store/f1ap2...-HelloDBService";
};
};

The above code example shows the two "hidden" properties augmented to the HelloDBService:

  • The _systemsPerTarget specifies for which CPU architecture/operating systems the service must be built. Normally, services are target agnostic and should always yield the same Nix store path (with a build that is nearly bit-identical), but the system architecture of the target machine is an exception to deviate from this property -- it is also possible to deploy the same service to different CPU architectures/operating systems. In such cases the build result could be different.
  • The _pkgsPerSystem specifies for each system architecture, the Nix store path to the build result. A side effect of evaluating the Nix store path is the service also gets built from source code.

Finally, it will compose a deployment architecture model attribute named: targetPackages that refers to a list of Nix store paths to be distributed to each machine in the network:

{
targetPackages = {
test1 = [
"/nix/store/91abq...-HelloDBService"
];

test2 = [
"/nix/store/p9af1...-HelloMySQLDB"
];
};

services = ...
infrastructure = ...
}

The targetPackages attribute is useful for a variety of reasons, as we will see later.

Generating a deployment model


With a normalized architecture model, we can generate an executable specification that I will call a deployment model. The deployment model can be used for executing all remaining activities after the services have been built.

An example of a deployment model could be:

{
profiles = {
test1 = "/nix/store/...-test1";
test2 = "/nix/store/...-test2";
};

services = {
"ekfekrerw..." = {
name = "HelloMySQLDB";
pkg = "/nix/store/...";
type = "mysql-database";
dependsOn = [
];
connectsTo = [
];
};

"dfsjs9349..." = {
name = "HelloDBService";
pkg = "/nix/store/...";
type = "tomcat-webapplication";
dependsOn = [
{ target = "test1";
container = "mysql-database";
service = "ekfekrerw...";
}
];
connectsTo = [
];
};
};

infrastructure = {
test1 = {
properties = {
hostname = "test1.example.org";
};
containers = {
apache-webapplication = {
documentRoot = "/var/www";
};
};
system = "x86_64-linux";
numOfCores = 1;
clientInterface = "disnix-ssh-client";
targetProperty = "hostname";
};
test2 = {
properties = {
hostname = "test2.example.org";
};
containers = {
mysql-database = {
mysqlPort = "3306";
};
};
system = "x86_64-linux";
numOfCores = 1;
clientInterface = "disnix-ssh-client";
targetProperty = "hostname";
};
};

serviceMappings = [
{ service = "ekfekrerw...";
target = "test2";
container = "mysql-database";
}
{ service = "dfsjs9349...";
target = "test1";
container = "tomcat-webapplication";
}
];

snapshotMappings = [
{ service = "ekfekrerw...";
component = "HelloMySQLDB";
container = "mysql-database";
target = "test2";
}
];
}

  • The profiles attribute refers to Nix profiles mapped to target machines and is derived from the targetPackages property in the normalized deployment architecture model. From the profiles property Disnix derives all steps of the distribution phase in which all packages and their intra-dependencies are copied to machines in the network.
  • The services attribute refers to all services that can be mapped to machines. The keys in this attribute set are SHA256 hash codes are recursively computed from the Nix store path of the package, the type, and all the inter-dependency mappings. Using hash codes to identify the services makes it possible to easily see whether a service is identical to another or not (by comparing hash codes), so that upgrades can be done more efficiently.
  • The infrastructure attribute is unchanged compared to the deployment architecture model and still stores target machine properties.
  • The serviceMappings attribute maps services in the services attribute set, to target machines in the network stored in the infrastructure attribute set and containers hosted on the target machines.

    From these mappings, Disnix can derive the steps to activate and deactivate the services of which a system is composed, ensure that all dependencies are present and that the services are activated or deactivated in the right order.
  • The snapshotMappings attribute state that for each services mapped to a target machines and container, we also want to migrate the state (by taking and restoring snapshots) if the service gets moved from one machine to another.

Although a deployment model is quite low-level, it is now also possible to manually write one, and deploy it by running:

$ disnix-env -D deployment.nix

disnix-env invokes an external executable called: disnix-deploy that executes the remaining activities of deployment process after the build process succeeds. disnix-depoy as well as the tools that execute individual deployment activities are driven by a manifest files. A manifest file is simply a one-on-one translation of the deployment model in the Nix expression language to XML following the NixXML convention.

Generating a build model


To build the services from source code, Disnix simply uses Nix's build facilities to execute the build. If nothing special has been configured, all builds will be executed on the coordinator machine, but this may not always be desired.

Disnix also facilitates heterogeneous architecture support. For example, if the coordinator machine is a Linux machine and a target machine is macOS (which is not compatible with the Linux system architecture), then Nix should delegate the build to a remote machine that is capable of building it. This is not something that Disnix handles for you out of the box -- you must configure Nix yourself to allow builds to be delegated.

It is also possible to optionally let Disnix delegate builds to the target machines in the network. To make build delegation work, Disnix generates a build model from a normalized deployment architecture model:

{
derivations = [
{ "/nix/store/HelloMySQLDB-....drv"; interface = "test1"; }
{ "/nix/store/HelloDBService-....drv"; interface = "test2"; }
];

interfaces = {
test1 = {
targetAddress = "test1.example.org";
clientInterface = "disnix-ssh-client";
};

test2 = {
targetAddress = "test2.example.org";
clientInterface = "disnix-ssh-client";
};
};
}

The build model shown above defines the following properties:

  • The derivations attribute maps Nix store derivation files (low-level Nix specifications that capture build procedures and dependencies) to machines in the network that should perform the build. This information is used by Disnix to delegate store derivation closure to target machines, use Nix to build the packages remotely, and fetch the build results back to the coordinator machine.
  • The interfaces attribute is a sub set of the infrastructure model that contains the connectivity settings for each target machine.

By running the following command, you can execute a build model to delegate builds to remote machines and fetch their results back:

$ disnix-delegate -B build.nix

If the build delegation option is enabled (for example, by passing --build-on-targets parameter to disnix-env) then Disnix will work a so-called distributed derivation file. Similar to a manifest file, a distributed derivation file is a one-on-one translation from the build model written in the Nix expression language to XML using the NixXML convention.

Packages model


In the normalized architecture model and deployment model, we generate a targetPackages property that we can use to compose Nix profiles with packages from.

For a variety of reasons, I thought it would also be interesting to give the user direct control to use this property. A new feature in Disnix is that you can now also write a packages model:

{pkgs, system}:

{
test1 = [
pkgs.mc
];

test2 = [
pkgs.wget
pkgs.curl
];
}

The above packages model says that we should distribute the Midnight Commander package to the test1 machine, and wget and curl to the test2 machine.

Running the following command will deploy the packages to the target machines in the network:

$ disnix-env -i infrastructure.nix -P pkgs.nix

You can also combine the three common Disnix models with a package model:

$ disnix-env -s services.nix \
-i infrastructure.nix \
-d distribution.nix \
-P pkgs.nix

then Disnix will deploy the services that are distributed to target machines and the supplemental packages defined in the packages model.

The packages model is useful for a variety of reasons:

  • Although it is already possible to use Disnix as a simple package deployer (by setting the types of services to: package), the packages model approach makes it even easier. Furthermore, you also more easily specify sets of packages for target machines. The only thing you cannot do is deploying packages that have inter-dependencies on services, e.g. a client that is preconfigured to connect to a service.
  • The hybrid approach makes it possible to more smooth make a transition to Disnix when automating the deployment process of a system. You can start by managing the dependencies with Nix, then package pieces of the project as Nix packages, then use Disnix to deploy them to remote machines, and finally turn pieces of the system into services that can be managed by Disnix.

Conclusion


In this blog post, I have described a new transformation pipeline in Disnix with well-defined intermediate steps that transforms the input models to a deployment model that is consumable by the tools that implement the deployment activities.

The following diagram summarizes the input models, intermediate models and output models:


The new transformation pipeline has the following advantages over the old infrastructure:

  • The implementation is much easier to maintain and we can more easily reason about its correctness
  • We have access to a broader range of configuration properties. For example, it was previously not possible to select the targets of the inter-dependencies.
  • The output models: deployment and build models are much more easily consumable by the Disnix tools that execute the remainder of the deployment activities. The domain models in the code, also closely resemble the structure of the build and deployment models. This can also be partially attributed to libnixxml that I have described in my previous blog post.
  • We can more easily implement new input models, such as the packages model.
  • The implementation of the disnix-reconstruct tool that reconstructs the manifest on the coordinator machine from metadata stored on the target machines also has become much simpler -- we can get rid of most of the custom code and generate a deployment model instead.

Availability


The new pipeline is available in the current development version of Disnix and will become available for general use in the next Disnix release.

The deployment models described in this blog post are incompatible with the manifest file format used in the last stable release of Disnix. This means that after upgrading Disnix, you need to convert any previous deployment configuration by running the disnix-convert tool.

by Sander van der Burg (noreply@blogger.com) at August 11, 2019 10:31 PM

August 09, 2019

Ollie Charles

Who Authorized These Ghosts!?

Recently at CircuitHub we’ve been making some changes to how we develop our APIs. We previously used Yesod with a custom router, but we’re currently exploring Servant for API modelling, in part due to it’s potential for code generation for other clients (e.g., our Elm frontend). Along the way, this is requiring us to rethink and reinvent previously established code, and one of those areas is authorization.

To recap, authorization is

the function of specifying access rights/privileges to resources related to information security and computer security in general and to access control in particular.

This is in contrast to authentication, which is the act of showing that someone is who they claim to be.

Authorization is a very important process, especially in a business like CircuitHub where we host many confidential projects. Accidentally exposing this data could be catastrophic to both our business and customers, so we take it very seriously.

Out of the box, Servant has experimental support for authorization, which is a good start. servant-server gives us Servant.Server.Experimental.Auth which makes it a doddle to plug in our existing authorization mechanism (cookies & Redis). But that only shows that we know who is asking for resources, how do we check that they are allowed to access the resources?

As a case study, I want to have a look at a particular end-point, /projects/:id/price. This endpoint calculates the pricing options CircuitHub can offer a project, and there are few important points to how this endpoint works:

  1. The pricing for a project depends on the user viewing it. This is because some users can consign parts so CircuitHub won’t order them. Naturally, this affects the price, so pricing is viewer dependent.
  2. Some projects are owned by organizations, and should be priced by the organization as a whole. If a user is a member of the organization that owns the project pricing has been requested for, return the pricing for the organization. If the user is not in the organization, return their own custom pricing.
  3. Private projects should only expose their pricing to superusers, the owner of the project, and any members of the project’s organization (if it’s owned by an organization).

This specification is messy and complicated, but that’s just reality doing it’s thing.

Our first approach was to try and represent this in Servant’s API type. We start with the “vanilla” route, with no authentication or authorization:

Next, we add authorization:

At this point, we’re on our own - Servant offers no authorization primitives (though there are discussions on this topic).

My first attempt to add authorization to this was:

There are two new routing combinators here: AuthorizeWith and CanView. The idea is AuthorizeWith somehow captures the result of authenticating, and provides that information to CanView. CanView itself does some kind of authorization using a type class based on its argument - here Capture "id" ProjectId. The result is certainly something that worked, but I was unhappy with both the complexity to implement it (which is scope to get it wrong), and the lack of actual evidence of authorization.

The latter point needs some expanding. What I mean by “lacking evidence” is that with the current approach, the authorization is essentially like writing the following code:

If I later add more resource access into doThings, what will hold me accountable to checking authorization on those resources? The answer is… nothing! This is similar to boolean blindless - we performed logical check, only to throw all the resulting evidence away immediately.

At this point I wanted to start exploring some different options. While playing around with ideas, I was reminded of the wonderful paper “Ghosts of Departed Proofs”, and it got me thinking… can we use these techniques for authorization?

Ghosts of Departed Proofs

The basic idea of GDP is to name values using higher-rank quantification, and then - in trusted modules - produce proofs that refer to these names. To name values, we introduce a Named type, and the higher-ranked function name to name things:

Note that the only way to construct a Named value outside of this module is to use name, which introduces a completely distinct name for a limited scope. Within this scope, we can construct proofs that refer to these names. As a basic example, we could use GDP to prove that a number is prime:

Here we have our first proof witness - IsPrime. We can witness whether or not a named Int is prime using checkPrime - like the boolean value isPrime this determines if a number is or isn’t prime, but we get evidence that we’ve checked a specific value for primality.

This is the whirlwind tour of GDP, I highly recommend reading the paper for a more thorough explanation. Also, the library justified-containers explores these ideas in the context of maps, where we have proofs that specific items are in the map (giving us total lookups, rather than partial lookups).

GDP and Authorization

This is all well and good, but how does this help with authorization? The basic idea is that authorization is itself a proof - a proof that we can view or interact with resources in a particular way. First, we have to decide which functions need authorization - these functions will be modified to require proof values the refer to the function arguments. In this example, we’ll assume our Servant handler is going to itself make a call to the price :: ProjectId -> UserId -> m Price function. However, given the specification above, we need to make sure that user and project are compatible. To do this, we’ll name the arguments, and then introduce a proof that the user in question can view the project:

But what is this CanViewProject proof?

A first approximation is to treat it as some kind of primitive or axiom. A blessed function can postulate this proof with no further evidence:

This is a good start! Our price function can only be called with a CanViewProject that matches the named arguments, and the only way to construct such a value is to use canViewProject. Of course we could get the implementation of this wrong, so we should focus our testing efforts to make sure it’s doing the right thing.

However, the Agda programmer in me is a little unhappy about just blindly postulating CanViewProject at the end. We’ve got a bit of vision back from our boolean blindness, but the landscape is still blurry. Fortunately, all we have to do is recruit more of the same machinery so far to subdivide this proof into smaller ones:

Armed with these smaller authorization primitives, we can build up our richer authorization scheme:

Now canViewProject just calls out to the other authorization routines to build it’s proof. Furthermore, there’s something interesting here. CanViewProject doesn’t postulate anything - everything is attached with a proof of the particular authorization case. This means that we can actually open up the whole CanViewProject module to the world - there’s no need to keep anything private. By doing this and allowing people to pattern match on CanViewProject, authorization results become reusable - if something else only cares that a user is a super user, we might be able to pull this directly out of CanViewProject - no need for any redundant database checks!

In fact, this very idea can help us implement the final part of our original specification:

Some projects are owned by organizations, and should be priced by the organization as a whole. If a user is a member of the organization that owns the project pricing has been requested for, return the pricing for the organization. If the user is not in the organization, return their own custom pricing.

If we refine our UserBelongsToProjectOrganization proof, we can actually maintain a bit of extra evidence:

Now whenever we have a proof UserBelongsToProjectOrganization, we can pluck out the actual organization that we’re talking about. We also have evidence that the organization owns the project, so we can easily construct a new CanViewProject proof - proofs generate more proofs!

Relationship to Servant

At the start of this post, I mentioned that the goal was to integrate this with Servant. So far, we’ve looked at adding authorization to a single function, so how does this interact with Servant? Fortunately, it requires very little to change. The Servant API type is authorization free, but does mention authentication.

It’s only when we need to call our price function do we need to have performed some authorization, and this happens in the server-side handler. We do this by naming the respective arguments, witnessing the authorization proof, and then calling price:

Conclusion

That’s where I’ve got so far. It’s early days so far, but the approach is promising. What I really like is there is almost a virtual slider between ease and rigour. It can be easy to get carried away, naming absolutely everything and trying to find the most fundamental proofs possible. I’ve found so far that it’s better to back off a little bit - are you really going to get some set membership checks wrong? Maybe. But a property check is probably gonig to be enough to keep that function in check. We’re not in a formal proof engine setting, pretending we are just makes things harder than they need to be.

by Oliver Charles at August 09, 2019 12:00 AM

July 11, 2019

Mayflower

Leveraging NixOS Tests in your Project

NixOS contains infrastructure for building integration tests based on QEMU/KVM virtual machines running NixOS. Tests built on this infrastructure are continuously run on new nixpkgs versions to ensure that NixOS continues to install and boot and that various services continue to operate correctly. This post illustrates how one may test a simple web service using NixOS tests. To have a simple enough example at hand, we wrote a small service in PHP—a classical guestbook in which visitors can leave a message that will then be written to a database and subsequently shown to later visitors of the same site.

July 11, 2019 03:00 PM

July 09, 2019

Hercules Labs

Hercules CI #5 update requiredSystemFeatures, Cachix and Darwin support

What’s new?

We’ve released hercules-ci-agent 0.3 , which brings in Cachix and Darwin (macOS) support alongside with requiredSystemFeatures.

hercules-agent-0.3.0

TOML configuration

Previously the agent was configured via CLI options. Those are now all part of a configuration file formatted using TOML.

Support for binary caches

Added support for Cachix binary caches to share resulting binaries either with the public and/or between developers and/or just multiple agents.

Multi-agent and Darwin support

With binary caches to share derivations and binaries between machines, you’re now able to have multiple agents running.

Sharing binaries between machines takes time (bandwidth) so we recommend upgrading agent hardware over adding more agents.

In addition to Linux, Darwin (macOS) also became a supported deployment platform for the agent.

requiredSystemFeatures support

Derivations are now dispatched also based on requiredSystemFeatures derivation attribute that allows dispatching specific derivations to specific agents.

Cachix 0.2.1

Upgrade via the usual:

$ nix-env -iA cachix -f https://cachix.org/api/v1/install  

The most notable improvement is default compression has been lowered to increase throughput and it’s overridable via ``–compression-level`.

See Changelog for more details.

What’s next?

Known issues we’re resolving:

  • Builds that are in progress while agent is restarted won’t be re-queued. We’re prioritizing this one. Expect a bugfix in next deployment.
  • Evaluation and building is slower with Cachix. We’re going to add bulk query support and upstream caches to mitigate that.
  • Having a lot of failed derivations (>10k) will get frontend unresponsive.
  • Cachix auth tokens for private binary caches are personal. We’ll add support to create tokens specific to a cache.

If you notice any other bugs or annoyances please let us know.

Preview phase

The preview phase will now extend to all subscribers, which is the final phase before we’re launching publicly.

You can also receive our latest updates via Twitter or read the previous development update.

July 09, 2019 12:00 AM

July 04, 2019

Munich NixOS Meetup

NixOS Munich Community Meetup

photoMunich NixOS Meetup

The theme of this meetup is: How do you use the Nix ecosystem in your projects?

We (Mayflower) will showcase how we use Gitlab, Hydra & nixops for Continuous Integration and Deployment. If you want to share your setup with us, just show up and show us. :-)

ATTENTION: The Mayflower Munich office has moved to Laim! Please note the new address!

Food and beverages will be provided. We will BBQ on our new rooftop terrace!

München 80687 - Germany

Thursday, July 4 at 6:30 PM

25

https://www.meetup.com/Munich-NixOS-Meetup/events/262224658/

July 04, 2019 03:24 PM

May 15, 2019

Hercules Labs

gitignore for Nix

Abstract

Nix, when used as a development build tool, needs to solve the same problem that git does: ignore some files. We’ve extended nix-gitignore so that Nix can more reliably use the configuration that you’ve already written for git.

Introduction

When you tell Nix to build your project, you need to tell it which source files to build. This is done by using path syntax in a derivation or string interpolation.

mkDerivation {
  src = ./vendored/cowsay;
  postPatch = ''
    # Contrived example of using a file in string interpolation
    # The patch file is put in /nix/store and the interpolation
    # produces the appropriate store path.
    patch -lR ${./cowsay-remove-alpaca.patch}
  '';
  # ...
}

This works well, until you find that Nix unexpectedly rebuilds your derivation because a temporary, hidden file has changed. One of those files you filtered out of your git tree with a ‘gitignore’ file…

Nix, as a build tool or package manager, was not designed with any specific version control system in mind. In fact it predates any dominance of git, because Nix’s general solution to the file ignoring problem, filterSource, was already implemented in 2007.

Over the last two to three years, various people have written functions to reuse these gitignore files. We have been using an implementation by @siers over the last couple of months and it has served us well, until we had a gitignore file that wasn’t detected because it was in a parent directory of the source directory we wanted to use.

I was nerd sniped.

Two months later, I finally got around to the implementation and I’m happy to announce that it solves some other problems as well. It reuses the tested rules by siers, doesn’t use import from derivation and can read all the files that it needs to.

Usage

You can import the gitignoreSource function from the repo like below, or use your favorite pinning method.

{ pkgs ? import <nixpkgs> {} }
let
  inherit (pkgs.stdenv) mkDerivation;
  inherit (import (builtins.fetchTarball "https://github.com/hercules-ci/gitignore/archive/master.tar.gz") { }) gitignoreSource;
in
mkDerivation {
  src = gitignoreSource ./vendored/cowsay;
  postPatch = ''
    patch -lR ${./cowsay-remove-alpaca.patch}
  '';
  # ...
}

That’s all there is to it.

It also composes with cleanSourceWith if you like to filter out some other files as well.

Comparison

Here’s a comparison with the pre-existing implementation I found.

The latest up to date comparison table is available on the repo.

Feature \ Implementation cleanSource siers siers recursive icetan Profpatsch numtide this project
Ignores .git ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
No special Nix configuration ✔️ ✔️ ✔️ ✔️ ✔️   ✔️
No import from derivation ✔️ ✔️   ✔️ ✔️ ✔️ ✔️
Uses subdirectory gitignores     ✔️     ✔️ ✔️
Uses parent gitignores           ✔️ ? ✔️
Uses user gitignores           ✔️ ✔️
Has a test suite   ✔️ ✔️ ✔️   ? ✔️
Works with restrict-eval / Hydra ✔️ ✔️   ✔️ ✔️   ✔️
Included in nixpkgs ✔️ ✔️ ✔️        
  Legend
✔️ Supported
✔️ ? Probably supported
  Not supported
? Probably not supported
- Not applicable or depends

Inclusion in Nixpkgs

I think it would be really nice to have this function in Nixpkgs, but it needs to be tested in practice first. This is where you can help out! Please give the project (GitHub) a spin and leave a thumbs up if it worked for you (issue).

Closing thoughts

I am happy to contribute to the friendly and inventive Nix community. Even though this gitignore project is just a small contribution, it wouldn’t have been possible without the ideas and work of siers, icetan, and everyone behind Nix and Nixpkgs in general.

As a company we are working hard to make good products to support the community and companies that want to use Nix. One of our goals is to keep making contributions like this, so please try our binary cache as a service, which is free for open source and just as easy to set up privately for companies. If you have an interest in our Nix CI, please subscribe.

– Robert

May 15, 2019 12:00 AM

May 14, 2019

Hercules Labs

Hercules CI #3 development update

What’s new?

Precise derivations improvements

Dependency failure tree

If a dependency failed for an attribute, you can now explore the dependency stack down to the actual build failure.

There’s also a rebuild button to retry the build for the whole stack, from the failed dependency down up to and including the build you clicked. We’ve addressed some of the styling issues visible on smaller screens.

Fixed an issue where users would end up being logged out

hercules-ci-agent 0.2

  • use gitignore instead of nix-gitignore
  • fix build on Darwin
  • limit internal concurrency to max eight OS threads for beefier machines
  • show version on --help
  • build against NixOS 19.03 as default
  • propagate agent information to agent view: Nix version, substituters, platform and Nix features

Focus for the next sprint

Cachix and thus Darwin support

The last bits missing (besides testing) are sharing derivations and artifacts between agents using cachix and the ease of Darwin agent deployment with accompanying documentation.

Stuck jobs when restarting the agent

Currently when you restart an agent that is doing work, jobs claimed by the agent will appear stuck in the queue. This sprint is planned to ship a way to remedy the issue manually via the UI. Later on it will be automatically handled by agent ping-alive.

Preview phase

Once we’re done with Darwin and Cachix support, we’ll hand out preview access to everyone who will have signed up for preview access.

You can also receive our latest updates via Twitter or read the previous development update.

May 14, 2019 12:00 AM

May 06, 2019

Matthew Bauer

Nixpkgs macOS Stdenv Updates

Over the past couple of months, I have been working on updating the macOS stdenv in Nixpkgs. This has significant impact on users of Nix/Nixpkgs on macOS. So, I want to explain what’s being updated, what the benefits are, and how we can minimize breakages.

1 macOS/Darwin stdenv changes

First, to summarize the changes that impact stdenv and the Darwin infrastructure. The PR is available at NixOS/nixpkgs PR #56744. This update has been in the works for the last few months, and is currently in the staging-next branch, waiting to be merged in NixOS/nixpkgs PR #60491. It should land on master and nixpkgs-unstable in the next few days. The main highlights are —

  • Change default LLVM toolchain from 5 to 7. LLVM 5 stdenv is still available through llvmPackages_5.stdenv attribute path.
  • Upgrade Apple SDK from 10.10 to 10.12.
  • Update libSystem symbols from 10.10 (XNU 3789.1.32) to 10.12 (XNU 3789.1.32).
  • Removed old patches to support old stdenv in Qt 5 and elsewhere.

These macOS SDK upgrades are equivalent to setting -mmacosx-version-min to 10.12 in XCode. As a result, we will break compatibility with macOS before 10.12.

2 Why do we need to set a minimum macOS version?

Without knowing internals of Nixpkgs, it might not be clear why we need to set a minimum macOS version. For instance with Linux, we are able to support any Linux kernel in Nixpkgs without any problem. The answer to this requires some understanding of how the kernel and userspace function.

Nixpkgs is able to support multiple Linux kernels because we can use multiple Libc’s at one time. For any executable, a Nix closure will include both its own Libc and the dynamic linker in its closure. This works in Linux where multiple Libc’s can be used, but not on macOS where only one Libc is available.

In short, Linux and macOS deal with compatibility between built binaries in different ways. They represent two opposite ends in how Unix-like kernels maintain compatibility with their userspace binaries.

2.1 Linux syscall compatibility

The kernel is responsible for managing core operating system functions such as start-up, memory management, device abstractions, and process isolation. For it to function, the kernel needs to interact with the rest of the operating system which is collectively referred to as “userspace”. Executables in userspace use “syscalls” to tell the kernel what to do. These syscalls are very low-level and usually not called directly by a process. Instead, an abstraction layer is provided by the standard C library, or Libc.

Linux is unique among operating systems due to the fact that the Kernel and Libc are developed independently. Linux is maintained by creator Linus Torvalds and a community of contributors. Glibc, the most popular Libc for Linux, is maintained by the GNU project. As a result, Linux has a strong separation between Syscalls and Libc.

Linux does not tie itself to any specific Libc. Even though Glibc is used in almost all distros, many alternatives are available. For instance, Musl provides a more lightweight version of Glibc, while Bionic is the Libc used in the Android operating system. In addition, multiple versions of each of these Libc’s can be used on any one kernel, even at the same time. This can become very common when using multiple Nixpkgs versions at one time.

To accomplish this, Linux provides a stable list of syscalls that it has maintained across many versions. This is specified for i386 at arch/x86/entry/syscalls/syscall_32.tbl in the kernel tree. The syscalls specified here are the interface through which the Libc communicates with the kernel. As a result, applications built in 1992 can run on a modern kernel, provided it comes with copies of all its libraries1.

2.2 macOS Libc compatibility

The macOS Libc is called libSystem. It is available on all macOS systems at /usr/lib/libSystem.B.dylib. This library is the main interface that binary compatibility is maintained in macOS. Unlike Linux, macOS maintains a stable interface in libSystem that all executables are expected to link to. This interface is guaranteed by Apple to be stable between versions.

In Nixpkgs, we maintain this compatibility through a list of symbols that are exported by libSystem. This is a simple text list and is available for viewing at NixOS/nixpkgs/pkgs/os-specific/darwin/apple-source-releases/Libsystem/system_c_symbols. The symbol list is created by listing symbols (nm) on the minimum macOS version that we support (for my PR, 10.12). We do some linking tricks to ensure that everything that we build in Nixpkgs only contains those symbols. This means that we can reproducibly build on newer versions of macOS, while maintaining compatibility with older macOS versions. Unfortunately, newer symbols introduced in later versions cannot be used even on systems that have those symbols.

A side effect of macOS design, is that fully static executables are not supported in macOS as they are on Linux. Without a stable syscall interface, there is nothing to provide compatibility between versions. As a result, Apple does not support this type of linking2.

There is no mandated reason why we need to use libSystem directly. In fact, some languages like Go have attempted to instead use the syscall interface directly. There is no reason why this couldn’t work, however, upgrades between versions will almost certainly break binaries. Go eventually abandoned this scheme in time for Go 1.12 (proposed by Nixpkgs macOS contributor copumpkin!)

2.3 Others

Some other examples may be useful. They mostly fall on one side or the other of the Syscall / Libc divide —

  • FreeBSD - breaks syscall compatibility between major releases, should use Libc for longterm binary compatibility.
  • OpenBSD - similarly, changes syscall interface, perhaps even more often than FreeBSD3.
  • NetBSD - apparently has maintained syscall compatibility since 1992. 4
  • Windows, Solaris, Fuchsia - I cannot find any information on these and how they handle binary compatibility.

2.4 LLVM triple

As a side note, this difference can be clearly seen in how we specify target systems. The LLVM triple is a 3 or 4-part string specifying what we want to build for. The parts of the triple correspond to:

<cpu>-<vendor>-<kernel>-<abi>
  • <cpu> — the CPU architecture that we are building for. Examples include x86_64, aarch64, armv7l, etc.
  • <vendor> — an arbitrary string specifying the vendor for the toolchain. In Nixpkgs, this should always be unknown.
  • <kernel> — the kernel to build for (linux).
  • <abi> — the kernel ABI to use. On Linux, this corresponds to the Libc we are using (gnu for Glibc, musl for Musl, android for Bionic).

When building for Linux, we can build for any version of Linux at one time. No version information is required. In addition, we must specify what “ABI” we want to use. In Nix, this is not very important because the Libc is provided by the closure. In fact, Nix has its own version of the LLVM triple called a Nix system tuple that omits the <abi> portion altogether! It corresponds to <cpu>-<kernel> from the LLVM triple.

In comparison, when building for BSDs, we must specify which version of the kernel we are building for. In addition, we leave off the <abi> portion, because there is only one Libc available for these platforms. They are even included in the same tree as the kernel. Examples of BSD triples include,

  • aarch64-apple-darwin16.0.0
  • x86_64-unknown-freebsd12.0
  • i386-unknown-openbsd5.8
  • armv7l-unknown-netbsd7.99

3 Compatibility table

Looking through the old versions, I’ve compiled a list of what I think are the corresponding macOS versions for each Nixpkgs release. As you can see, we try to support at least 3 previous macOS releases. This also happens to be about what Apple supports through security updates5.

Nixpkgs release macOS version
19.09 10.12, 10.13, 10.14, 10.15?
19.03 10.116, 10.12, 10.13, 10.14
18.09 10.11, 10.12, 10.13, 10.14
18.03 10.11, 10.12, 10.13, 10.14
17.09 10.10, 10.11, 10.12, 10.13
17.03 10.10, 10.11, 10.12
16.09 10.10, 10.11, 10.12
16.03 10.9?, 10.10, 10.11, 10.12

We know that some users are stuck on older versions of macOS due to reasons outside of their control. As a result, we will try to support the 19.03 branch for a little bit longer than is usually done. If your organization uses 10.11, it might be a good idea to update to a newer version along with your update to Nixpkgs 19.09.

4 Conclusion

My main goal has been to show better how Nixpkgs and macOS system interact. I got a little bit sidetracked exploring differences in binary compatibility between different operating systems. But, this should help users to better understand the differences in how macOS and Linux works in relation to Nixpkgs.

Footnotes:

1

It would be interesting to test this in practice. Finding a Libc that would work might be the hardest part. Even better if we could use Nix’s closures!

3

According to the_why_of_y on Hacker News, https://news.ycombinator.com/item?id=14011662

5

macOS updates come out about every year and Apple offers about 3 months support. More information is available at https://apple.stackexchange.com/questions/47664/what-is-apples-policy-for-supporting-security-updates-on-older-versions-of-os-x.

6

There is an issue with building on 10.11 with the new swift-corelibs derivation. As a result, you need to use prebuilt version to avoid this issue.

May 06, 2019 12:00 AM

April 30, 2019

Hercules Labs

Sprint #2 development update

What’s new in sprint #2?

Precise derivations

Agent 0.2 will communicate the structure of the derivation closure to the service, which allows us to traverse the derivation tree and dispatch each derivation to multiple agents.

Neither source or binary data used by Nix on the agent is ever sent to the service.

We will release agent 0.2 after more testing and UI improvements. as we’re still doing testing and UI improvements.

Git-rich metadata

Each job now includes a branch name, commit message and the committer:

Job rich metadata

Job page information

The job page shows information that triggered the build and timing information:

Job page

Focus for sprint #3

Cachix support

We’ve already made progress of parsing Cachix configuration, but there’s the actual pushing of derivations to be done.

Darwin / Features support

Now that precise derivations are working, they need to get aware of platforms for Darwin support. Same goes for infamous Nix “features”, which work like labels that can be used to dispatch individual derivations to specific groups of agents.

Preview phase

You’re still in time to sign up for preview access or follow us on Twitter as we will be expanding access in the coming weeks.

April 30, 2019 12:00 AM

April 16, 2019

Hercules Labs

Sprint #1 development update

Two weeks ago we launched preview access of our CI for Nix users. Thank you all for giving us the feedback through the poll and individually. We are overwhelmed with the support we got.

Focus of the preview launch was to build a fast, reliable, easy to use CI. Today you can connect your github repository in a few clicks, deploy an agent and all your commits are being tested with their status reported to GitHub.

In our latest sprint we have fixed a few issues, mainly centered around usability and clarity of what’s going on with your projects.

The following features were shipped:

The following bugs fixes were shipped:

  • When there is no agent available, enqueued jobs will show instructions to setup one

  • To prevent CSRF we’ve tightened SameSite cookie from Lax to Strict

  • CDN used to serve stale assets due to caching misconfiguration

  • Numerous fixes to the UI:

    • breadcrumbs now allow user to switch account or just navigate to it
    • no more flickering when switching pages
    • some jobs used to be stuck in Building phase
    • more minor improvements

In our upcoming sprint, #2 we will focus on:

  • Fine-grained dispatch of individual derivations (instead of just top-level derivation closures from attributes as we shipped in the preview) - what follows is testing and presenting derivations in the UI

  • Currently we only store the git revision for each job, which will be expanded to include more metadata like branch name, commit message, author, etc

  • If time allows, preliminary cachix support

You’re still in time to sign up for preview access as we will be expanding access in the following weeks.

April 16, 2019 12:00 AM

March 07, 2019

Hercules Labs

Announcing Cachix private binary caches and 0.2.0 release

In March 2018 we’ve set on a mission to streamline Nix usage in teams. Today we are shipping Nix private binary cache support to Cachix.

You can now share an unlimited number of binary caches in your group of developers, protected from public use with just a few clicks.

Authorization is based on GitHub organizations/teams (if this is a blocker for you, let us know what you need).

To get started, head over to https://cachix.org and choose a plan that suits your private binary cache needs:

Create Nix private binary cache

At the same time cachix 0.2.0 cli is out with major improvements to NixOS usage. If you run the following as root you’ll get:

$ cachix use hie-nix
Cachix configuration written to /etc/nixos/cachix.nix.
Binary cache hie-nix configuration written to /etc/nixos/cachix/hie-nix.nix.

To start using cachix add the following to your /etc/nixos/configuration.nix:

    imports = [ ./cachix.nix ];

Then run:

    $ nixos-rebuild switch

Thank you for your feedback in the poll answers. It’s clear what we should do next:

  1. Multiple signing keys (key rotation, multiple trusted users, …)

  2. Search over binary cache contents

  3. Documentation

Happy cache sharing!

March 07, 2019 12:00 AM

February 27, 2019

Sander van der Burg

Generating functional architecture documentation from Disnix service models

In my previous blog post, I have described a minimalistic architecture documentation approach for service-oriented systems based on my earlier experiences with setting up basic configuration management repositories. I used this approach to construct a documentation catalog for the platform I have been developing at Mendix.

I also explained my motivation -- it improves developer effectiveness, team consensus and the on-boarding of new team members. Moreover, it is a crucial ingredient in improving the quality of a system.

Although we are quite happy with the documentation, my biggest inconvenience is that I had to derive it entirely by hand -- I consulted various kinds of sources, but since existing documentation and information provided by people may be incomplete or inconsistent, I considered the source code and deployment configuration files the ultimate source of truth, because no matter how elegantly a diagram is drawn, it is useless if it does not match the actual implementation.

Because a manual documentation process is very costly and time consuming, a more ideal situation would be to have an automated approach that automatically derives architecture documentation from deployment specifications.

Since I am developing a deployment framework for service-oriented systems myself (Disnix), I have decided to extend it with a generator that can derive architecture diagrams and supplemental descriptions from the deployment models using the conventions I have described in my previous blog post.

Visualizing deployment architectures in Disnix


As explained in my previous blog post, the notation that I used for the diagrams was not something I invented from scratch, but something I borrowed from Disnix.

Disnix already has a feature (for quite some time) that can visualize deployment architectures referring to a description that shows how the functional parts (the services/components) are mapped to physical resources (e.g. machines/containers) in a network.

For example, after deploying a service-oriented system, such as my example web application system, by running:


$ disnix-env -s services.nix -i infrastructure.nix \
-d distribution-bundles.nix

You can visualize the corresponding deployment architecture of the system, by running:


$ disnix-visualize > out.dot

The above command-line instruction generates a directed graph in the DOT language. The resulting dot file can be converted into a displayable image (such as a PNG or SVG file) by running:


$ dot -Tpng out.dot > out.png

Resulting in a diagram of the deployment architecture that may look as follows:


The above diagram uses the following notation:

  • The light grey boxes denote machines in a network. In the above deployment scenario, we have two them.
  • The ovals denote services (more specifically: in a Disnix-context, they reflect any kind of distributable deployment unit). Services can have almost any shape, such as web services, web applications, and databases. Disnix uses a plugin system called Dysnomia to make sure that the appropriate deployment steps are carried out for a particular type of service.
  • The arrows denote inter-dependencies. When a service points to another service means that the latter is an inter-dependency of the former service. Inter-dependency relationships ensure that the dependent service gets all configuration properties so that it knows how to reach the dependency and the deployment system makes sure that inter-dependencies of a specific service are deployed first.

    In some cases, enforcing the right activation order of activation may be expensive. It is also possible to drop the ordering requirement, as denoted by the dashed arrows. This is acceptable for redirects from the portal application, but not acceptable for database connections.
  • The dark grey boxes denote containers. Containers can be any kind of runtime environment that hosts zero or more distributable deployment units. For example, the container service of a MySQL database is a MySQL DBMS, whereas the container service of a Java web application archive can be a Java Servlet container, such as Apache Tomcat.

Visualizing the functional architecture of service-oriented systems


The services of which a service-oriented systems is composed are flexible -- they can be deployed to various kinds of environments, such a test environment, a second fail-over production environment or a local machine.

Because services can be deployed to a variety of targets, it may also be desired to get an architectural view of the functional parts only.

I created a new tool called: dydisnix-visualize-services that can be used to generate functional architecture diagrams by visualizing the services in the Disnix services model:


The above diagram is a visual representation of the services model of the example web application system, using a similar notation as the deployment architecture without showing any environment characteristics:

  • Ovals denote services and arrows denote inter-dependency relationships.
  • Every service is annotated with its type, so that it becomes clear what kind of a shape a service has and what kind of deployment procedures need to be carried out.

Despite the fact that the above diagram is focused on the functional parts, it may still look quite detailed, even from a functional point of view.

Essentially, the architecture of my example web application system is a "system of sub systems" -- each sub system provides an isolated piece of functionality consisting of a database backend and web application front-end bundle. The portal sub system is the entry point and responsible for guiding the users to the sub systems implementing the functionality that they want to use.

It is also possible to annotate services in the Disnix services model with a group and description property:


{distribution, invDistribution, pkgs, system}:

let
customPkgs = import ../top-level/all-packages.nix {
inherit pkgs system;
};

groups = {
homework = "Homework";
literature = "Literature";
...
};
in
{
homeworkdb = {
name = "homeworkdb";
pkg = customPkgs.homeworkdb;
type = "mysql-database";
group = groups.homework;
description = "Database backend of the Homework subsystem";
};

homework = {
name = "homework";
pkg = customPkgs.homework;
dependsOn = {
inherit usersdb homeworkdb;
};
type = "apache-webapplication";
appName = "Homework";
group = groups.homework;
description = "Front-end of the Homework subsystem";
};

...
}

In the above services model, I have grouped every database and web application front-end bundle in a group that represents a sub system (such as Homework). By adding the --group-subservices parameter to the dydisnix-visualize-services command invocation, we can simplify the diagram to only show the sub systems and how these sub systems are inter-connected:


$ dydisnix-visualize-services -s services.nix -f png \
--group-subservices

resulting in the following functional architecture diagram:


As may be observed in the picture above, all services have been grouped. The service groups are denoted by ovals with dashed borders.

We can also query sub architecture diagrams of every group/sub system. For example, the following command generates a sub architecture diagram for the Homework group:


$ dydisnix-visualize-services -s services.nix -f png \
--group Homework --group-subservices

resulting in the following diagram:


The above diagram will only show the the services in the Homework group and their context -- i.e. non-transitive dependencies and services that have a dependency on any service in the requested group.

Services that exactly fit the group or any of its parent groups will be displayed verbatim (e.g. the homework database back-end and front-end). The other services will be categorized into in the lowest common sub group (the Users and Portal sub systems).

For more complex architectures consisting of many layers, you may probably want to generate all available architecture diagrams in one command invocation. It is also possible to run the visualization tool in batch mode. In batch mode, it will recursively generate diagrams for the top-level architecture and every possible sub group and stores them in a specified output folder:


$ dydisnix-visualize-services --batch -s services.nix -f svg \
--output-dir out

Generating supplemental documentation


Another thing I have explained in my previous blog post is that providing diagrams is useful, but they cannot clear up all confusion -- you also need to document and clarify additional details, such as the purposes of the services.

It also possible to generate a documentation page for each group showing a table of services with their descriptions and types:

The following command generates a documentation page for the Homework group:


$ dydisnix-document-services -s services.nix --group Homework

It is also possible to adjust the generation process by providing a documentation configuration file (by using the --docs parameter):


$ dydisnix-document-services -f services.nix --docs docs.nix \
--group Homework

The are a variety of settings that can be provided in a documentation configuration file:


{
groups = {
Homework = "Homework subsystem";
Literature = "Literature subsystem";
...
};

fields = [ "description" "type" ];

descriptions = {
type = "Type";
description = "Description";
};
}

The above configuration file specifies the following properties:

  • The descriptions for every group.
  • Which fields should be displayed in the overview table. It is possible to display any property of a service.
  • A description of every field in the services model.

Like the visualization tool, the documentation tool can also be used in batch mode to generate pages for all possible groups and sub groups.

Generating a documentation catalog


In addition to generating architecture diagrams and descriptions, it is also possible to combine both tools to automatically generate a complete documentation catalog for a service-oriented system, such as the web application example system:


$ dydisnix-generate-services-docs -s services.nix --docs docs.nix \
-f svg --output-dir out

By opening the entry page in the output folder, you will get an overview of the top-level architecture, with a description of the groups.


By clicking on a group hyperlink, you can inspect the sub architecture of the corresponding group, such as the 'Homework' sub system:


The above page displays the sub architecture diagram of the 'Homework' subsystem and a description of all services belonging to that group.

Another particularly interesting aspect is the 'Portal' sub system:


The portal's purpose is to redirect users to functionality provided by the other sub systems. The above architecture diagram displays all the sub systems in grouped form to illustrate that there is a dependency relationship, but without revealing all their internal details that clutters the diagram with unnecessary implementation details.

Other features


The tools support more use cases than those described in this blog post -- it is also possible, for example, to create arbitrary layers of sub groups by using the '/' character as a delimiter in the group identifier. I also used the company platform as an example case, that can be decomposed into four layers.

Availability


The tools described in this blog post are part of the latest development version of Dynamic Disnix -- a very experimental extension framework built on top of Disnix that can be used to make service-oriented systems self-adaptive by redeploying their services in case of events.

The reason why I have added these tools to Dynamic Disnix (and not the core Disnix toolset) is because the extension toolset has an infrastructure to parse and reflect over individual Disnix models.

Although I promised to make an official release of Dynamic Disnix a very long time ago, this still has not happened yet. However, the documentation feature is a compelling reason to stabilize the code and make the framework more usable.

by Sander van der Burg (noreply@blogger.com) at February 27, 2019 11:09 PM