NixOS Planet

November 22, 2017

Munich NixOS Meetup

CoP NixOS Munich

photoMunich NixOS Meetup

Dear members,
a lot of time has passed since our last community of practice meetup took place. In the meantime a lot of great stuff has happened, e.g. Release 17.09, NixCon. So let's meet once again to talk, work, pair on interesting topics in the NixOS environment.
Btw, if you don't had a chance to look at it, NixOS has adopted a great RFC approach https://github.com/ni... . Please checkout out. Maybe this is a staring point for a lot of fruitful discussions when we meet.
We @holidaycheck would like to host the meetup once again, so come and bring any people who maybe need any help with their NixOS/Nix-env.
Any topic can be proposed in the commens section upfront or in the check-in session during meetup.

Com: https://copnixos.slac...

Munich - Germany

Wednesday, November 29 at 7:30 PM

10

https://www.meetup.com/Munich-NixOS-Meetup/events/245278438/

November 22, 2017 04:53 PM

November 18, 2017

Joachim Schiele

nixcloud-webservices

motivation

on the quest to make nix the no. 1 deployment tool for webservices, we are happy to announce the release of:

  • nixcloud.webservices
  • nixcloud.reverse-proxy
  • nixcloud.email

all can be found online at:

git clone https://github.com/nixcloud/nixcloud-webservices

or visit: https://github.com/nixcloud/nixcloud-webservices

nixcloud

what is the nixcloud? paul and me (joachim) see the nixcloud as an extension to nixpkgs focusing on the deployment of webservices. we abstract installation, monitoring, DNS management and most of all 'state management'.

nixcloud.webservices

nixcloud.webservices is a extension to nixpkgs for packaging webservices (apache, nginx) or language specific implementations as go, rust, perl and so on.

nixcloud.webservices.leaps.myservice = {
  enable = true;
  proxyOptions = {
    port   = 50000;
    path   = "/foo";
    domain = "example.com";
  };
};

it comes with so many features and improvements that you are better off reading the documentation here:

https://github.com/nixcloud/nixcloud-webservices/blob/master/documentation/nixcloud.webservices.md

nixcloud.reverse-proxy

this component makes it easy to mix several webservices, like multiple webservers as apache or nginx, into one or several domains. it also abstracts TLS using ACME.

nixcloud.reverse-proxy = {
  enable = true;
  extendEtcHosts = true;
  extraMappings = [
    {  
      domain = "example.com";
      path = "/";
      http = {
        mode = "on";
        record = ''
          rewrite ^(.*)$ https://example.org permanent;
        '';
      };
      https = {
        mode = "on";
        basicAuth."joachim" = "foo";
        record = ''
          rewrite ^(.*)$ https://example.org permanent;
        '';              
      };
    }
  ];
};

https://github.com/nixcloud/nixcloud-webservices/blob/master/documentation/nixcloud.reverse-proxy.md

nixcloud.email

manual here: https://github.com/nixcloud/nixcloud-webservices/blob/master/documentation/nixcloud.email.md

the idea is to make email deployment accessible to the masses by providing an easy abstraction, especially for webhosting.

configuration example

nixcloud.email= {
  enable = true;
  domains = [ "example.com" "example.org"  ];
  ipAddress = "1.2.3.4";
  ip6Address = "afe2::2;
  hostname = "example.com";
  users = [
    # kdkdkdkdkdkd -> feed that into -> doveadm pw -s sha256-crypt
    { name = "js2"; domain = "example.org"; password = "{PLAIN}foobar1234"; aliases = [ "postmaster@r0sset.com" ]; }
    { name = "paul"; domain = "example.com"; password = "{PLAIN}supersupergeheim"; }
    { name = "catchall"; domain = "example.com"; password = "{PLAIN}foobar1234"; aliases = [ "@example.com" ]; }
    #{ name = "js2"; domain = "nix.lt"; password = "{PLAIN}supersupergeheim"; }
    #{ name = "quotatest"; domain = "nix.lt"; password = "{PLAIN}supersupergeheim"; quota = "11M"; aliases = [ "postmaster@mail.nix.lt" "postmaster@nix.lt" ]; }
  ];
};

questions?

just ask:

joachim schiele js@lastlog.de or paul seitz paul.m.seitz@gmail.com

by qknight at November 18, 2017 12:35 PM

November 07, 2017

Flying Circus

NixOS: The DOs and DON’Ts of nixpkgs overlays

One presentation at NixCon 2017 that especially drew my attention was Nicolas Pierron‘s talk about Nixpkgs overlays (video, slides). I’d like to give a quick summary here for future reference. All the credits go to Nicolas, of course.

What are overlays?

Overlays are the new standard mechanism to customize nixpkgs. They replace constructs like packageOverride and overridePackages.

How do I use overlays?

Put your overlays into ~/.config/nixpkgs/overlays. All overlays share a basic structure:

self: super: {
  …
}

Arguments:

  • self is the result of the fix point calculation. Use it to access packages which could be modified somewhere else in the overlay stack.
  • super one overlay down in the stack (and base nixpkgs for the first overlay). Use it to access the package recipes you want to customize, and for library functions.

Good examples

Add default proxy to Google Chrome

self: super:
{
   google-chrome = super.google-chrome.override {
     commandLineArgs =
       "--proxy-server='https=127.0.0.1:3128;http=127.0.0.1:3128'";
   };
}

From Alexandre Peyroux.

Fudge Nix store location

self: super:
{
  nix = super.nix.override {
    storeDir = "${<nix-dir>}/store";
     stateDir = "${<nix-dir>}/var";
   };
}

From Yann Hodique.

Add custom package

self: super:
{
  VidyoDesktop = super.callPackage ./pkgs/VidyoDesktop { };
}

From Mozilla.

Bad examples

Recursive attrset

self: super: rec {
   fakeClosure = self.writeText "fake-closure.nix" ''
     …
   '';
   fakeConfig = self.writeText "fake-config.nix" ''
    (import ${fakeClosure} {}).config.nixpkgs.config
  '';
}

Two issues:

  • Use super to access library functions.
  • Overlays should not be recursive. Use self.

Surplus arguments

{ python }:
self: super:
{
  "execnet" =
    python.overrideDerivation super."execnet" (old: {
      buildInputs = old.buildInputs ++ [ self."setuptools-scm" ];
    });
}

The issue:

  • Overlays should depend just on self and super in order to be composeable.

Awkward nixpkgs import

{ pkgs ? import <nixpkgs> {} }:
let
  projectOverlay = self: super: {
    customPythonPackages =
      (import ./requirements.nix { inherit pkgs; }).packages;
  };
in
import pkgs.path {
  overlays = [ projectOverlay ];
}

Two issues:

  • Unnecessary double-import of nixpkgs. This might break cross-system builds.
  • requirements.nix should use self not pkgs.

Improved version

shell.nix:

{ pkgsPath ? <nixpkgs> }:
import pkgsPath {
  overlays = [ import ./default.nix; ];
}

default.nix:

self: super:
{
 customPythonPackages =
   (import ./requirements.nix { inherit self; }).packages;
}

Incorrectly referenced dependencies

self: super:
let inherit (super) callPackage;
in {
  radare2 = callPackage ./pkgs/radare2 {
    inherit (super.gnome2) vte;
    lua = super.lua5;
  };
}

The issue:

  • Other packages should be taken from self not super. This way they
    can be overridden by other overlays.

Overridden attrset

self: super:
{
  lib = {
    firefoxVersion = … ;
  };
  latest = {
    firefox-nightly-bin = … ;
    firefox-beta-bin = … ;
    firefox-bin = … ;
    firefox-esr-bin = … ;
  };
}

The issue:

  • Other attributes present in lib and latest from down the overlay stack are
    erased.

Improved version

Always extend attrsets in overlays:

self: super:
{
  lib = (super.lib or {}) // {
    firefoxVersion = … ;
  };
  latest = (super.latest or {}) // {
    …
  };
}

Summary

I hope this condensed guide helps you to write better overlays. For in-depth discussion, please go watch Nicolas’ talk and read the Nixpkgs manual. Many thanks to Nicolas for putting this together!


Cover image: ⓒ 2009 studio tdes / Flickr / CC-BY-2.0


by Christian Kauhaus at November 07, 2017 09:44 AM

November 03, 2017

Sander van der Burg

Creating custom object transformations with NiJS and PNDP

In a number earlier blog posts, I have described two kinds of internal DSLs for Nix -- NiJS is a JavaScript-based internal DSL and PNDP is a PHP-based internal DSL.

These internal DSLs have a variety of application areas. Most of them are simply just experiments, but the most serious application area is code generation.

Using an internal DSL for generation has a number of advantages over string generation that is more commonly used. For example, when composing strings containing Nix expressions, we must make sure that any variable in the host language that we append to a generated expression is properly escaped to prevent code injection attacks.

Furthermore, we also have to take care of the indentation if we want to output Nix expression code that should be readable. Finally, string manipulation itself is not a very intuitive activity as it makes it very hard to read what the generated code would look like.

Translating host language objects to the Nix expression language


A very important feature of both internal DSLs is that they can literally translate some language constructs from the host language (JavaScript or PHP) to the Nix expression because they have (nearly) an identical meaning. For example, the following JavaScript code fragment:

var nijs = require('nijs');

var expr = {
hello: "Hello",
name: {
firstName: "Sander",
lastName: "van der Burg"
},
numbers: [ 1, 2, 3, 4, 5 ]
};

var output = nijs.jsToNix(expr, true);
console.log(output);

will output the following Nix expression:

{
hello = "Hello",
name = {
firstName = "Sander";
lastName = "van der Burg";
};
numbers = [
1
2
3
4
5
];
}

In the above example, strings will be translated to strings (and quotes will be escaped if necessary), objects to attribute sets, and the array of numbers to a list of numbers. Furthermore, the generated code is also pretty printed so that attribute set and list members have 2 spaces of indentation.

Similarly, in PHP we can compose the following code fragment to get an identical Nix output:

use PNDP\NixGenerator;

$expr = array(
"hello" => "Hello",
"name" => array(
"firstName" => "Sander",
"lastName => "van der Burg"
),
"numbers" => array(1, 2, 3, 4, 5)
);

$output = NixGenerator::phpToNix($expr, true);
echo($output);

The PHP generator uses a number of clever tricks to determine whether an array is associative or sequential -- the former gets translated into a Nix attribute set while the latter gets translated into a list.

There are objects in the Nix expression language for which no equivalent exists in the host language. For example, Nix also allows you to define objects of a 'URL' and 'file' type. Neither JavaScript nor PHP have a direct equivalent. Moreover, it may be desired to generate other kinds of language constructs, such as function declarations and function invocations.

To still generate these kinds of objects, you must compose an abstract syntax tree from objects that inherit from the NixObject prototype or class. For example, we can define a function invocation to fetchurl {} in Nixpkgs as follows in JavaScript:

var expr = new nijs.NixFunInvocation({
funExpr: new nijs.NixExpression("fetchurl"),
paramExpr: {
url: new nijs.NixURL("mirror://gnu/hello/hello-2.10.tar.gz"),
sha256: "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"
}
});

and in PHP as follows:

use PNDP\AST\NixExpression;
use PNDP\AST\NixFunInvocation;
use PNDP\AST\NixURL;

$expr = new NixFunInvocation(new NixExpression("fetchurl"), array(
"url" => new NixURL("mirror://gnu/hello/hello-2.10.tar.gz"),
"sha256" => "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"
));

Both of the objects in the above code fragments translate to the following Nix expression:

fetchurl {
url = mirror://gnu/hello/hello-2.10.tar.gz;
sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
}

Transforming custom object structures into Nix expressions


The earlier described use cases are basically one-on-one translations from the host language (JavaScript or PHP) to the guest language (Nix). In some cases, literal translations do not make sense -- for example, it may be possible that we already have an application with an existing data model from which we want to derive deployments that should be carried out with Nix.

In the latest versions of NiJS and PNDP, it is also possible to specify how to transform custom object structures into a Nix expression. This can be done by inheriting from the NixASTNode class or prototype and overriding the toNixAST() method.

For example, we may have a system already providing a representation of a file that should be downloaded from an external source:

function HelloSourceModel() {
this.src = "mirror://gnu/hello/hello-2.10.tar.gz";
this.sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
}

The above module defines a constructor function composing an object that refers to the GNU Hello package provided by a GNU mirror site.

A direct translation of an object constructed by the above function to the Nix expression language does not provide anything meaningful -- it can, for example, not be used to let Nix fetch the package from the mirror site.

We can inherit from NixASTNode and implement our own custom toNixAST() function to provide a more meaningful Nix translation:

var nijs = require('nijs');
var inherit = require('nijs/lib/ast/util/inherit.js').inherit;

/* HelloSourceModel inherits from NixASTNode */
inherit(nijs.NixASTNode, HelloSourceModel);

/**
* @see NixASTNode#toNixAST
*/
HelloSourceModel.prototype.toNixAST = function() {
return this.args.fetchurl()({
url: new nijs.NixURL(this.src),
sha256: this.sha256
});
};

The toNixAST() function shown above composes an abstract syntax tree (AST) for a function invocation to fetchurl {} in the Nix expression language with the url and sha256 properties a parameters.

An object that inherits from the NixASTNode prototype also indirectly inherits from NixObject. This means that we can directly attach such an object to any other AST object. The generator uses the underlying toNixAST() function to automatically convert it to its AST representation:

var helloSource = new HelloSourceModel();
var output = nijs.jsToNix(helloSource, true);
console.log(output);

In the above code fragment, we directly pass the construct HelloSourceModel object instance to the generator. The output will be the following Nix expression:

fetchurl {
url = mirror://gnu/hello/hello-2.10.tar.gz;
sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
}

In some cases, it may not be possible to inherit from NixASTNode, for example, when the object already inherits from another prototype or class that is beyond the user's control.

It is also possible to use the NixASTNode constructor function as an adapter. For example, we can take any object with a toNixAST() function:

var helloSourceWrapper = {
toNixAST: function() {
return new nijs.NixFunInvocation({
funExpr: new nijs.NixExpression("fetchurl"),
paramExpr: {
url: new nijs.NixURL(this.src),
sha256: this.sha256
}
});
}
};

By wrapping the helloSourceWrapper object in the NixASTNode constructor, we can convert it to an object that is an instance of NixASTNode:

new nijs.NixASTNode(helloSourceWrapper)

In PHP, we can change any class into a NixASTNode by implementing the NixASTConvertable interface:

use PNDP\AST\NixASTConvertable;
use PNDP\AST\NixURL;

class HelloSourceModel implements NixASTConvertable
{
/**
* @see NixASTConvertable::toNixAST()
*/
public function toNixAST()
{
return $this->args->fetchurl(array(
"url" => new NixURL($this->src),
"sha256" => $this->sha256
));
}
}

By passing an object that implements the NixASTConvertable interface to the NixASTNode constructor, it can be converted:

new NixASTNode(new HelloSourceModel())

Motivating use case: the node2nix and composer2nix generators


My main motivation to use custom transformations is to improve the quality of the node2nix and composer2nix generators -- the former converts NPM package configurations to Nix expressions and the latter converts PHP composer package configurations to Nix expressions.

Although NiJS and PNDP provide a number of powerful properties to improve the code generation steps of these tools, e.g. I no longer have to think much about escaping strings or pretty printing, there are still many organizational coding issues left. For example, the code that parses the configurations, fetches the external sources, and generates the code are mixed. As a consequence, the code is very hard to read, update, maintain and to ensure its correctness.

The new transformation facilities allow me to separate concerns much better. For example, both generators now have a data model that reflects the NPM and composer problem domain. For example, I could compose the following (simplified) class diagram for node2nix's problem domain:


A crucial part of node2nix's generator is the package class shown on the top left on the diagram. A package requires zero or more packages as dependencies and may provide zero or more packages in the node_modules/ folder residing in the package's base directory.

For readers not familiar with NPM's dependency management: every package can install its dependencies privately in a node_modules/ folder residing in the same base directory. The CommonJS module ensures that every file is considered to be a unique module that should not interfere with other modules. Sharing is accomplished by putting a dependency in a node_modules/ folder of an enclosing parent package.

NPM 2.x always installs a package dependency privately unless a parent package exists that can provide a conforming version. NPM 3.x (and later) will also move a package into the node_modules/ folder hierarchy as high as possible to prevent too many layers of nested node_modules/ folders (this is particularly a problem on Windows). The class structure in the above diagram reflects this kind of dependency organisation.

In addition to a package dependency graph, we also need to obtain package metadata and compute their output hashes. NPM packages originate from various kinds of sources, such as the NPM registry, Git repositories, HTTP sites and local directories on the filesystem.

To optimize the process and support sharing of common sources among packages, we can use a source cache that memorizes all unique source referencess.

The Package::resolveDependencies() method sets the generation process in motion -- it will construct the dependency graph replicating NPM's dependency resolution algorithm as faithfully as possible, and resolves all the dependencies' (and transitive dependencies) metadata.

After resolving all dependencies and their metadata, we must generate the output Nix expressions. One Nix expression is copied (the build infrastructure) and two are generated -- a composition expression and a package or collection expression.

We can also compose a class diagram for the generation infrastructure:


In the above class diagram, every generated expression is represented a class inheriting from NixASTNode. We can also reuse some classes from the domain model as constituents for the generated expressions, by also inheriting from NixASTNode and overriding the toNixAST() method:

  • The source objects can be translated into sub expressions that invoke fetchurl {} and fetchgit {}.
  • The sources cache can be translated into an attribute set exposing all sources that are used as dependencies for packages.
  • A package instance can be converted into a function invocation to nodeenv.buildNodePackage {} that, in addition to configuring build properties, binds the required dependencies to the sources in the sources cache attribute set.

By decomposing the expression into objects and combining the objects' AST representations, we can nicely modularize the generation process.

For composer2nix, we can also compose a class diagram for its domain -- the generation process:


The above class diagram has many similarities, but also some major differences compared to node2nix. composer provides so-called lock files that pinpoint the exact versions of all dependencies and transitive dependencies. As a result, we do not need to replicate composer's dependency resolution algorithm.

Instead, the generation process is driven by the ComposerConfig class that encapsulates the properties of the composer.json and composer.lock files of a package. From a composer configuration, the generator constructs a package object that refers to the package we intend to deploy and populates a source cache with source objects that come from various sources, such as Git, Mercurial and Subversion repositories, Zip files, and directories residing on the local filesystem.

For the generation process, we can adopt a similar strategy that exposes the generated Nix expressions as classes and uses some classes of the domain model as constituents for the generation process:


Discussion


In this blog post, I have described a new feature for the NiJS and PNDP frameworks, making it possible to implement custom transformations. Some of its benefits are that it allows an existing object model to be reused and concerns in an application can be separated much more conveniently.

These facilities are not only useful for the improvement of the architecture of the node2nix and composer2nix generators -- at the company I work for (Conference Compass), we developed our own domain-specific configuration management tool.

Despite the fact that it uses several tools from the Nix project to carry out deployments, it uses a domain model that is not Nix-specific at all. Instead, it uses terminology and an organization that reflects company processes and systems.

For example, we use a backend-for-frontend organization that provides a backend for each mobile application that we ship. We call these backends configurators. Optionally, every configurator can import data from various external sources that we call channels. The tool's data model reflects this kind of organization, and generates Nix expressions that contain all relevant implementation details, if necessary.

Finally, the fact that I modified node2nix to have a much cleaner architecture has another reason beyond quality improvement. Currently, NPM version 5.x (that comes with Node.js 8.x) is still unsupported. To make it work with Nix, we require a slightly different generation process and a completely different builder environment. The new architecture allows me to reuse the common parts much more conveniently. More details about the new NPM support will follow (hopefully) soon.

Availability


I have released new versions of NiJS and PNDP that have the custom transformation facilities included.

Furthermore, I have decided to release new versions for node2nix and composer2nix that use the new generation facilities in their architecture. The improved architecture revealed a very uncommon but nasty bug with bundled dependencies in node2nix, that is now solved.

by Sander van der Burg (noreply@blogger.com) at November 03, 2017 10:43 PM

October 04, 2017

Flying Circus

Announcing fc-userscan

NixOS manages dependencies in a very strict way—sometimes too strict? Here at Flying Circus, many users prefer to compile custom applications in home directories. They link them against libraries they have installed before by nix-env. This works well… until something is updated! On the next change anywhere down the dependency chain, libraries get new hashes in the Nix store, the garbage collector removes old versions, and user applications break until recompiled.

In this blog post, I would like to introduce fc-userscan. This little tool scans (home) directories recursively for Nix store references and registers them as per-user roots with the garbage collector. This way, dependencies will be protected even if they cease to be referenced from “official” Nix roots like the current-system profile or a user’s local Nix profile. After registering formerly unmanaged references with fc-userscan, one can fearlessly run updates and garbage collection.

This problem would not be there if everyone would just write Nix expressions for everything. This is, for various reasons, not realistic. Some users find it hard to accommodate with Nix concepts (“I just want to compile my software and don’t care about your arcane Linux distro!”). Others use deploy tools like zc.buildout which are explicitly designed for in-place updates and don’t really distinguish compile time from run time. With fc-userscan, you can get both: Nix expressions for superb reproducibility, and an acceptable degree of reliance for impurely compiled applications.

fc-userscan is published as open source software under a 3-clause BSD license. Get the Rust source code or the latest pre-compiled binaries for x86_64. Pull requests are welcome.

Example: How to get stale Nix store references

Imagine we have Python installed in our Nix user profile. When creating a Python virtual environment, we get unmanaged references into the Nix store:

$ nix-env -iA nixpkgs.python35
$ ~/.nix-profile/bin/pyvenv venv
$ ls -og venv/bin/python3.5
lrwxrwxrwx 1 71 Sep 29 11:19 venv/bin/python3.5 -> /nix/store/8lh4pxhhi4cx00pp1zxpz9pqyy44kjm6-python3-3.5.4/bin/python3.5

The garbage collector does not know anything about this reference. This is OK as long as nothing gets deleted from the Nix store. But eventually channel updates arrive. After an seemingly innocent run of nixos-rebuild or nix-env –upgrade, chances are high that the Python package’s hash has changed. The next run of nix-collect-garbage will trash the Python binary still referenced from our virtualenv.

Although we use Python for illustration, the problem is universal. You can easily run into similar trouble with C programs, for example.

Protecting unmanaged references

We run fc-userscan to detect and register Nix store references:

$ fc-userscan -v venv
fc-userscan: Scouting venv
2 references in /nix/var/nix/gcroots/profiles/per-user/ck/home/ck/venv
Processed 741 files (8 MB read) in 0.011 s
fc-userscan: Finished venv

Now the exact Python interpreter version is registered as a garbage collection root:

$ ls -og /nix/var/nix/gcroots/profiles/per-user/ck/home/ck/venv
lrwxrwxrwx 1 57 Sep 29 11:46 8lh4pxhhi4cx00pp1zxpz9pqyy44kjm6 -> /nix/store/8lh4pxhhi4cx00pp1zxpz9pqyy44kjm6-python3-3.5.4
[...]

The Python interpreter and all its dependencies are now protected. The file layout in the per-user directories is organized in such a way that is possible to re-scan arbitrary directories and always get correct results.

Feature overview

Include/exclude lists

To reduce system load, fc-userscan can be instructed to skip files that are unlikely to contain store references. Specify include/exclude globs from

  1. a default ignore file ~/.userscan-ignore if existent (in gitignore(5) format)
  2. a custom ignore file with -­-exclude-from (in gitignore(5) format)
  3. the command line with -­-exclude, -­-include.

A related option is -­-quickcheck which causes fc-userscan to stop scanning a file when there is no Nix store reference in the first n bytes of that file.

Cache

fc-userscan is able to preserve scan results between runs to avoid re-scanning unchanged files. Simply add -­-cache FILE. It uses the ctime inode attribute to determine if a file has been changed or not. The cache is stored as compressed messagepack file so that it does not take much space even on large installations.

Unzip on the fly

Some runnable application artefacts are stored as ZIP files and may contain Nix store references. The most prominent instances are compressed Python eggs and JAR files. fc-userscan has support to transparently decompress files that match glob patterns passed with -­-unzip. Note that scanning inside compressed ZIP archives is significantly slower than scanning regular files.

List mode

It is possible to scan directory hierarchies and just print all found references to stdout instead of registering them with -­-list. The output format is script friendly.

These options and many more are explained in the fc-userscan(1) man page which ships with the release.

Interaction with nix-collect-garbage

How does fc-userscan interact with nix-collect-garbage now? We at Flying Circus run fc-userscan for all users and start nix-collect-garbage directly afterwards. fc-userscan exits unsuccessfully if there was a problem (e.g., unreadable files). We recommend not to run nix-collect-garbage if fc-userscan found a problem. Otherwise, not all relevant files might have been checked for Nix store references and nix-collect-garbage might delete too much.

Here is an example script which scans all users’ home directories:

failed=0
while read user home; do
  sudo -u $user -H -- \
    fc-userscan -v 2 \
    --exclude-from /etc/userscan.exclude \
    --cache $home/.cache/fc-userscan.cache \
    --unzip '*.egg'
    $home || failed=1
done < <(getent passwd | awk -F: '$4 >= 100 { print $1 " " $6 }')

if (( failed )); then
  echo "ERROR: fc-userscan failed (see above)"
  exit 1
else
  nix-collect-garbage --delete-older-than 3d
fi

Conclusion

While in theory it would be better to drive all software installations on a NixOS system via derivations, fc-userscan brings us flexibility to protect impurely installed applications.


Cover image © 2007 by benfrantzdale CC-BY-SA 2.0


by Christian Kauhaus at October 04, 2017 12:32 PM

October 02, 2017

Sander van der Burg

Deploying PHP composer packages with the Nix package manager

In two earlier blog posts, I have described various pieces of my custom web framework that I used to actively develop many years ago. The framework is quite modular -- every concern, such as layout management, data management, the editor, and the gallery, are separated into packages that can be deployed independently, so that web applications only have to include what they actually need.

Although modularity is quite useful for a variety of reasons, the framework did not start out as being modular in the beginning -- when I just started developing web applications in PHP, I did not reuse anything at all. Slowly, I discovered similarities between my projects and started sharing snippets of common functionality between them. Gradually, I learned that keeping these common aspects up to date became a burden. As a result, I developed a "common framework" that I reused among all my PHP projects.

Having a common framework for my web application projects reduced the amount of required maintenance, but introduced a new drawback -- its size kept growing and growing. As a result, many simple web applications that only required a small subset of the framework's functionality still had to embed the entire framework, making them unnecessarily big.

Today, a bit of extra PHP code is not so much of a problem, but around the time I was still actively developing web applications, many shared web hosting providers only offered a small amount of storage capacity, typically just a few megabytes.

To cope with the growing size of the framework, I decided to modularize the code by separating the framework's concerns into packages that can be deployed independently. I "invented" my own conventions to integrate the framework packages into web applications:

  • In the base directory of the web application project, I create a lib/ directory that contains symlinks to the framework packages.
  • In every PHP script that displays a page (typically only index.php), I configure the include path to refer to the packages' content in the lib/ folder, such as:


    set_include_path("./lib/sblayout:./lib/sbdata:./lib/sbcrud");

  • Each PHP module is responsible for loading the desired classes or utility functions from the framework packages. As a result, I ended up writing a substantial amount of require() statements, such as:


    require_once("data/model/Form.class.php");
    require_once("data/model/field/HiddenField.class.php");
    require_once("data/model/field/TextField.class.php");
    require_once("data/model/field/DateField.class.php");
    require_once("data/model/field/TextAreaField.class.php");
    require_once("data/model/field/URLField.class.php");
    require_once("data/model/field/FileField.class.php");

After my (approximately) 8 years of absence from the PHP domain, I discovered that a tool has been developed to support convenient construction of modular PHP applications: composer. Composer is heavily inspired by the NPM package manager, that is the defacto package delivery mechanism for Node.js applications.

In the last couple of months (it progresses quite slowly as it is a non-urgent side project), I have decided to get rid of my custom modularity conventions in my framework packages, and to adopt composer instead.

Furthermore, composer is a useful deployment tool, but its scope is limited to PHP applications only. As frequent readers may probably already know, I use Nix-based solutions to deploy entire software systems (that are also composed of non-PHP packages) from a single declarative specification.

To be able to include PHP composer packages in a Nix deployment process, I have developed a generator named: composer2nix that can be used to generate Nix deployment expressions from composer configuration files.

In this blog post, I will explain the concepts of composer2nix and show how it can be used.

Using composer


Using composer is generally quite straight forward. In the most common usage scenario, there is typically a PHP project (often a web application) that requires a number of dependencies. By changing the current working folder to the project directory, and running:


$ composer install

Composer will obtain all required dependencies and stores them in the vendor/ sub directory.

The vendor/ folder follows a very specific organisation:


$ find vendor/ -maxdepth 2 -type d
vendor/bin
vendor/composer
vendor/phpdocumentor
vendor/phpdocumentor/fileset
vendor/phpdocumentor/graphviz
vendor/phpdocumentor/reflection-docblock
vendor/phpdocumentor/reflection
vendor/phpdocumentor/phpdocumentor
vendor/svanderburg
vendor/svanderburg/pndp
...

The vendor/ folder structure (mostly) consists two levels: the outer directory defines the namespace of the packages and the inner directory the package names.

There are a couple of folders deviating from this convention -- most notably, the vendor/composer directory, that is used by composer to track package installations:


$ ls vendor/composer
autoload_classmap.php
autoload_files.php
autoload_namespaces.php
autoload_psr4.php
autoload_real.php
autoload_static.php
ClassLoader.php
installed.json
LICENSE

In addition to obtaining packages and storing them in the vendor/ folder, composer also generates autoload scripts (as shown above) that can be used to automatically make code units (typically classes) provided by the packages available for use in the project. Adding the following statement to one of your project's PHP scripts:


require_once("vendor/autoload.php");

suffices to load the functionality exposed by the packages that composer installs.

Composer can be used to install both runtime and development dependencies. Many development dependencies (such as phpunit or phpdocumentor) provide command-line utilities to carry out tasks. Composer packages can also declare which executables they provide. Composer automatically generates symlinks for all provided executables in the: vendor/bin folder:


$ ls -l vendor/bin/
lrwxrwxrwx 1 sander users 29 Sep 26 11:49 jsonlint -> ../seld/jsonlint/bin/jsonlint
lrwxrwxrwx 1 sander users 41 Sep 26 11:49 phpdoc -> ../phpdocumentor/phpdocumentor/bin/phpdoc
lrwxrwxrwx 1 sander users 45 Sep 26 11:49 phpdoc.php -> ../phpdocumentor/phpdocumentor/bin/phpdoc.php
lrwxrwxrwx 1 sander users 34 Sep 26 11:49 pndp-build -> ../svanderburg/pndp/bin/pndp-build
lrwxrwxrwx 1 sander users 46 Sep 26 11:49 validate-json -> ../justinrainbow/json-schema/bin/validate-json

For example, you can run the following command-line instruction from the base directory of a project to generate API documentation:


$ vendor/bin/phpdocumentor -d src -t out

In some cases (the composer documentation often discourages this) you may want to install end-user packages globally. They can be installed into the global composer configuration directory by running:


$ composer global require phpunit/phpunit

After installing a package globally (and adding: $HOME/.config/composer/vendor/bin directory to the PATH environment variable), we should be able to run:


$ phpunit --help

The composer configuration


The deployment operations that composer carries out are driven by a configuration file named: composer.json. An example of such a configuration file could be:


{
"name": "svanderburg/composer2nix",
"description": "Generate Nix expressions to build PHP composer packages",
"type": "library",
"license": "MIT",
"authors": [
{
"name": "Sander van der Burg",
"email": "svanderburg@gmail.com",
"homepage": "http://sandervanderburg.nl"
}
],

"require": {
"svanderburg/pndp": "0.0.1"
},
"require-dev": {
"phpdocumentor/phpdocumentor": "2.9.x"
},

"autoload": {
"psr-4": { "Composer2Nix\\": "src/Composer2Nix" }
},

"bin": [ "bin/composer2nix" ]
}

The above configuration file declares the following configuration properties:

  • A number of meta attributes, such as the package name, description, license and authors.
  • The package type. The type: library indicates that this project is a library that can be used in another project.
  • The project's runtime (require) and development (require-dev) dependencies. In a dependency object, the keys refer to the package names and the values to version specifications that can be either:
    • A semver compatible version specifier that can be an exact version (e.g. 0.0.1), wildcard (e.g. 1.0.x), or version range (e.g. >= 1.0.0).
    • A version alias that directly (or indirectly) resolves to a branch in the VCS repository of the dependency. For example, the dev-master version specifier refers to the current master branch of the Git repository of the package.
  • The autoloader configuration. In the above example, we configure the autoloader to load all classes belonging to the Composer2Nix namespace, from the src/Composer2Nix sub directory.

By default, composer obtains all packages from the Packagist repository. However, it is also possible to consult other kinds of repositories, such as external HTTP sites or VCS repositories of various kinds (including Git, Mercurial and Subversion).

External repositories can be specified by adding a 'repositories' object to the composer configuration:


{
"name": "svanderburg/composer2nix",
"description": "Generate Nix expressions to build PHP composer packages",
"type": "library",
"license": "MIT",
"authors": [
{
"name": "Sander van der Burg",
"email": "svanderburg@gmail.com",
"homepage": "http://sandervanderburg.nl"
}
],
"repositories": [
{
"type": "vcs",
"url": "https://github.com/svanderburg/pndp"
}
],

"require": {
"svanderburg/pndp": "dev-master"
},
"require-dev": {
"phpdocumentor/phpdocumentor": "2.9.x"
},

"autoload": {
"psr-4": { "Composer2Nix\\": "src/Composer2Nix" }
},

"bin": [ "bin/composer2nix" ]
}

In the above example, we have defined PNDP's GitHub repository as an external repository and changed the version specifier of svanderburg/pndp to use the latest Git master branch.

Composer uses a version resolution strategy that will parse composer configuration files and branch names in all repositories to figure out where a version can be obtained from and takes the first option that matches the dependency specification. Packagist is consulted last, making it possible for the user to override dependencies.

Pinpointing dependency versions


The version specifiers of dependencies in a composer.json configuration file are nominal and have some drawbacks when it comes to reproducibility -- for example, the version specifier: >= 1.0.1 may resolve to version 1.0.2 today and to 1.0.3 tomorrow, making it very difficult to exactly reproduce a deployment elsewhere at a later point in time.

Although direct dependencies can be easily controlled by the user, it is quite difficult to control the version resolutions of the transitive dependencies. To cope with this problem, composer will always generate lock files (composer.lock) that pinpoint the exact dependency versions (including all transitive dependencies) the first time when it gets invoked (or when composer update is called):


{
"_readme": [
"This file locks the dependencies of your project to a known state",
"Read more about it at https://getcomposer.org/doc/01-basic-usage.md#composer-lock-the-lock-file",
"This file is @generated automatically"
],
"content-hash": "ca5ed9191c272685068c66b76ed1bae8",
"packages": [
{
"name": "svanderburg/pndp",
"version": "v0.0.1",
"source": {
"type": "git",
"url": "https://github.com/svanderburg/pndp.git",
"reference": "99b0904e0f2efb35b8f012892912e0d171e9c2da"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/svanderburg/pndp/zipball/99b0904e0f2efb35b8f012892912e0d171e9c2da",
"reference": "99b0904e0f2efb35b8f012892912e0d171e9c2da",
"shasum": ""
},
"bin": [
"bin/pndp-build"
],
...
}
...
]
}

By bundling the composer.lock file with the package, it becomes possible to reproduce a deployment elsewhere with the exact same package versions.

The Nix package manager


Nix is a package manager whose main purpose is to build all kinds of software packages from source code, such as GNU Autotools, CMake, Perl's MakeMaker, Apache Ant, and Python projects.

Nix's main purpose is not be a build tool (it can actually also be used for building projects, but this application area is still highly experimental). Instead, Nix manages dependencies and complements existing build tools by providing dedicated build environments to make deployments reliable and reproducible, such as clearing all environment variables, making files read-only after the package has been built, restricting network access and resetting the files' timestamps to 1.

Most importantly, in these dedicated environments Nix ensures that only specified dependencies can be found. This may probably sound inconvenient at first, but this property exists for a good reason: if a package unknowingly depends on another package then it may work on the machine where it has been built, but may fail on another machine because this unknown dependency is missing. By building a package in a pure environment in which all dependencies are known, we eliminate this problem.

To provide stricter purity guarantees, Nix isolates packages by storing them in a so-called "Nix store" (that typically resides in: /nix/store) in which every directory entry corresponds to a package. Every path in the Nix store is prefixed by hash code, such as:


/nix/store/2gi1ghzlmb1fjpqqfb4hyh543kzhhgpi-firefox-52.0.1

The hash is derived from all build-time dependencies to build the package.

Because every package is stored in its own path and variants of packages never share the same name because of the hash prefix, it becomes harder for builds to accidentally succeed because of undeclared dependencies. Dependencies can only be found if the environment has been configured in such a way that the Nix store paths to the packages are known, for example, by configuring environment variables, such as: export PATH=/nix/store/5vyssyqvbirdihqrpqhbkq138ax64bjy-gnumake-4.2.1/bin.

The Nix expression language and build environment abstractions have all kinds of facilities to make the configuration of dependencies convenient.

Integrating composer deployments into Nix builder environments


Invoking composer in a Nix builder environment introduces an additional challenge -- composer is not only a tool that does build management (e.g. it can execute script directives that can carry out arbitrary build steps), but also dependency management. The latter property conflicts with the Nix package manager.

In a Nix builder environment, network access is typically restricted, because it affects reproducibility (although it still possible to hack around this restriction) -- when downloading a file from an external site it is not known in advance what you will get. An unknown artifact influences the outcome of a package build in unpredictable ways.

Network access in Nix build environments is only permitted in so-called fixed output derivations. For a fixed output derivation, the output hash must be known in advance so that Nix can verify whether we have obtained the artifact we want.

The solution to cope with a conflicting dependency manager is by substituting it -- we must let Nix obtain the dependencies and force the tool to only execute its build management tasks.

We can populate the vendor/ folder ourselves. As explained earlier, the composer.lock file stores the exact versions of pinpointed dependencies including all transitive dependencies. For example, when a project declares svanderburg/pndp version 0.0.1 as a dependency, it may translate to the following entry in the composer.lock file:


"packages": [
{
"name": "svanderburg/pndp",
"version": "v0.0.1",
"source": {
"type": "git",
"url": "https://github.com/svanderburg/pndp.git",
"reference": "99b0904e0f2efb35b8f012892912e0d171e9c2da"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/svanderburg/pndp/zipball/99b0904e0f2efb35b8f012892912e0d171e9c2da",
"reference": "99b0904e0f2efb35b8f012892912e0d171e9c2da",
"shasum": ""
},
...
}
...
]

As can be seen in the code fragment above, the dependency translates to two kinds of pinpointed source objects -- a source reference to a specific revision in a Git repository and a dist reference to a zipball containing a snapshot of the given Git revision.

The reason why every dependency translates to two kinds of objects is that composer supports two kinds of installation modes: source (to obtain a dependency directly from a VCS) and dist (to obtain a dependency from a zipball).

We can translate the 'dist' reference into the following Nix function invocation:


"svanderburg/pndp" = {
targetDir = "";
src = composerEnv.buildZipPackage {
name = "svanderburg-pndp-99b0904e0f2efb35b8f012892912e0d171e9c2da";
src = fetchurl {
url = https://api.github.com/repos/svanderburg/pndp/zipball/99b0904e0f2efb35b8f012892912e0d171e9c2da;
sha256 = "19l7i7adp76bjf32x9a2ykm0r5cgcmi4wf4cm4127miy3yhs0n4y";
};
};
};

and the 'source' reference to the following Nix function invocation:


"svanderburg/pndp" = {
targetDir = "";
src = fetchgit {
name = "svanderburg-pndp-99b0904e0f2efb35b8f012892912e0d171e9c2da";
url = "https://github.com/svanderburg/pndp.git";
rev = "99b0904e0f2efb35b8f012892912e0d171e9c2da";
sha256 = "15i311dc0123v3ppa69f49ssnlyzizaafzxxr50crdfrm8g6i4kh";
};
};

(As a sidenote: we need the targetDir property to provide compatibility with the deprecated PSR-0 autoloading standard. Old autoload packages can be stored in a sub folder of a package residing in the vendor/ structure.)

To generate the above function invocations, we need more than just the properties provided by the composer.lock file. Since download functions in Nix are fixed output derivations, we must compute the output hashes of the downloads by invoking a Nix prefetch script, such as nix-prefetch-url or nix-prefetch-git. The composer2nix generator will automatically invoke the appropriate prefetch script to augment the generated expressions with output hashes.

To ensure maximum compatibility with composer's behaviour, the dependencies obtained by Nix must be copied into to the vendor/ folder. In theory, symlinking would be more space efficient, but experiments have shown that some packages (such as phpunit) may attempt to load the project's autoload script, e.g. by invoking:


require_once(realpath("../../autoload.php"));

The above require invocation does not work if the dependency is a symlink -- the require path resolves to a path in the Nix store (e.g. /nix/store/...). The parent's parent path corresponds to /nix where no autoload script is stored. (As a sidenote: I have decided to still provide symlinking as an option for deployment scenarios where this is not an issue).

After some experimentation, I discovered that composer uses the following file to track which packages have been installed: vendor/composer/installed.json. The contents appears to be quite similar to the composer.lock file:


[
{
"name": "svanderburg/pndp",
"version": "v0.0.1",
"version_normalized": "0.0.1.0",
"source": {
"type": "git",
"url": "https://github.com/svanderburg/pndp.git",
"reference": "99b0904e0f2efb35b8f012892912e0d171e9c2da"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/svanderburg/pndp/zipball/99b0904e0f2efb35b8f012892912e0d171e9c2da",
"reference": "99b0904e0f2efb35b8f012892912e0d171e9c2da",
"shasum": ""
},
...
},
...
]

Reconstructing the above file can be done by merging the contents of the packages and packages-dev objects in the composer.lock file.

Another missing piece in the puzzle is the autoload scripts. We can force composer to dump the autoload script, by running:


$ composer dump-autoload --optimize

The above command generates an optimized autoloader script. A non-optimized autoload script dynamically inspects the contents of the package folders to load modules. This is convenient in the development stage of a project, in which the files continuously change, but in production environments this introduces quite a bit of load time overhead.

Since packages in the Nix store can never change after they have been built, it makes no sense to generate a non-optimized autoloader script.

Finally, the last remaining practical issue, is PHP packages providing command-line utilities. Most executables have the following shebang line:


#!/usr/bin/env php

To ensure that these CLI tools work in Nix builder environments, the above shebang must be subsituted by the PHP executable that resides in the Nix store.

After carrying out the above described steps, running the following command:


$ composer install --optimize-autoloader

is simply just a formality -- it will not download or change anything.

Use cases


composer2nix has a variety of use cases. The most obvious one is to use it to package a web application project with Nix instead of composer. Running the following command generates Nix expressions from the composer configuration files:


$ composer2nix

By running the following command, we can use Nix to obtain the dependencies and generate a package with a vendor/ folder:


$ nix-build
$ ls result/
index.php vendor/

In addition to web applications, we can also deploy command-line utility projects implemented in PHP. For these kinds of projects it make more sense generate a bin/ sub folder in which the executables can be found.

For example, for the composer2nix project, we can generate a CLI-specific expression by adding the --executable parameter:


$ composer2nix --executable

We can install the composer2nix executable in our Nix profile by running:


$ nix-env -f default.nix -i

and then invoke composer2nix as follows:


$ composer2nix --help

We can also deploy third party command-line utilities directly from the Packagist repository:


$ composer2nix -p phpunit/phpunit
$ nix-env -f default.nix -iA phpunit-phpunit
$ phpunit --version

The most powerful application is not the integration with Nix itself, but the integration with other Nix projects. For example, we can define a NixOS configuration running an Apache HTTP server instance with PHP and our example web application:


{pkgs, config, ...}:

let
myexampleapp = import /home/sander/myexampleapp {
inherit pkgs;
};
in
{
services.httpd = {
enable = true;
adminAddr = "admin@localhost";
extraModules = [
{ name = "php7"; path = "${pkgs.php}/modules/libphp7.so"; }
];
documentRoot = myexampleapp;
};

...
}

We can deploy the above NixOS configuration as follows:


$ nixos-rebuild switch

By running only one simple command-line instruction, we have a running system with the Apache webserver serving our web application.

Discussion


In addition to composer2nix, I have also been responsible for developing node2nix, a tool that generates Nix expressions from NPM package configurations. Because composer is heavily inspired by NPM, we see many similarities in the architecture of both generators. For example, both generate the same kinds of expressions (a builder environment, a packages expression and a composition expression), have a similar separation of concerns, and both use an internal DSL for generating Nix expressions (NiJS and PNDP).

There are also a number of conceptual differences -- dependencies in NPM can be private to a package or shared among multiple packages. In composer, all dependencies in a project are shared.

The reason why NPM's dependency management is more powerful is because Node.js uses the CommonJS module system. CommonJS considers each file to be a unique module. This, for example, makes it possible for one module to load a version of a package from a certain filesystem location and another version of the same package from another filesystem location within the same project.

By contrast, in PHP, isolation is accomplished by the namespace declarations in each file. Namespaces can not be dynamically altered so that multiple versions can safely coexist in one project. Furthermore, the vendor/ directory structure makes it possible to store only one variant of a package.

Despite the fact that composer's dependency management is less powerful makes constructing a generator much more straightforward compared to NPM.

Another feature that composer supports for quite some time, and NPM until very recently is pinpointing/locking dependencies. When generating Nix expressions from NPM package configurations, we must replicate NPM's dependency resolving algorithm. In composer, we can simply take whatever the composer.lock file provides. The lock file saves us from replicating the dependency lookup process making the generation process considerably easier.

Acknowledgments


My implementation is not the first attempt that tries to integrate composer with Nix. After a few days of developing, I discovered another attempt that seems to be in a very early development stage. I did not try or use this version.

Availability


composer2nix can be obtained from Packagist and my GitHub page.

Despite the fact that I did quite a bit of research, composer2nix should still be considered a prototype. One of its known limitations is that it does not support fossil repositories yet.

by Sander van der Burg (noreply@blogger.com) at October 02, 2017 10:15 PM

September 27, 2017

Munich NixOS Meetup

NixOS 17.09 Release Party

photoMunich NixOS Meetup

NixOS 17.09 "Hummingbird" will be the next release scheduled to be released on September 28, 2017. Time to party!

Meet other fellow Nix/NixOS users & developers and the 17.09 release managers for some drinks.

We're grateful that HolidayCheck will be hosting the party!

Munich - Germany

Friday, September 29 at 7:00 PM

15

https://www.meetup.com/Munich-NixOS-Meetup/events/243579914/

September 27, 2017 02:36 PM

September 22, 2017

Georges Dubus

Share scripts that have dependencies with Nix

Share scripts that have dependencies with Nix

Writing and sharing scripts is essential to productivity in development teams. But if you want the script to work everywhere, you can only depend on the tools that are everywhere. That usually means not using the newer features of a language, and not using libraries. In this article, I explain how I use Nix to write scripts that can run everywhere, regardless on what they depend on.

In my work as a software engineer, I often write small scripts to help with administrative actions or provide some nice view of what we are working on. From time to time, one of these scripts is useful enough that I want to share it with my colleagues.

But sharing the scripts comes with a challenge. I write my scripts in Python because it is a great language for that kind of things. But the best parts are outside of the standard library. As a result, to share my scripts, I also have to write a documentation explaining how to get the right version of python, create a python virtual environment, activate it, install the dependencies, and run the script. That's not worth the hassle, and my colleagues won't bother following the complex set of instructions for a script.

Recently, I've started using Nix to make self-contained scripts, and it's been working pretty well.

A script that can download its dependencies

For example, some of the things I like using in python scripts are:

  • the new string formatting syntax in Python 3.6,
  • the requests package for anything that touches HTTP
  • docopt for argument parsing.

My scripts will usually look like:

#! /usr/bin/env nix-shell
#! nix-shell -i python -p python36 python36Packages.requests2 python36Packages.docopt -I nixpkgs=https://github.com/NixOS/nixpkgs-channels/archive/nixos-17.03.tar.gz
# You'll need nix to automatically download the dependencies: `curl https://nixos.org/nix/install | sh`
"""
Some script

Usage:
  some_script.py <user> <variant> [--api=<api>]
  some_script.py -h | --help

Arguments:
  <user>        format: 42
  <variant>     format: 2

Options:
  -h --help     Show this screen.
  --api=<api>   The api instance to talk to [default: the-api.internal]
"""
import requests
from docopt import docopt

def doit(user, variant, api):
    # The actual thing I want to do
    pass

if __name__ == '__main__':
    arguments = docopt(__doc__)
    doit(arguments['<user>'], arguments['<variant>'], arguments['--api'])

Now, anybody can run the script with ./some_script.py, which will download the dependencies and run the script with them. They'll need to install nix if they don't have it yet (with curl https://nixos.org/nix/install | sh).

What actually happens there?

The first line of the script is a shebang. It tells the system to run the script with nix-shell, one of the tools provided by Nix.

The second line specifies how to invoke nix-shell:

  • -i python: this argument means we want nix-shell to run the script with the python command (-i stands for interpreter).
  • -p python36 python36Packages.requests2 python36Packages.docopt: this argument is the list of the nix packages that my script is going to use. Here, I want python 3.6, and the python packages requests and docopt. I usually use the Nix packages page to find the name of the dependencies I want.
  • -I nixpkgs=https://github.com/NixOS/nixpkgs-channels/archive/nixos-17.03.tar.gz: the version of nix I want to use for this script. This ensures people running the script in the future will get compatible versions of the dependencies.

When I run the script, nix-shell starts by downloading https://github.com/NixOS/nixpkgs-channels/archive/nixos-17.03.tar.gz if it has not done so recently. This is the definition of all the nix packages. Then, nix-shell looks into this file to know how to download the packages I want and their dependencies. Finally, it runs my script in an environment where the dependencies are available.

Try it now

This technique allows coworkers to share scripts written in different technologies without the cost of teaching everybody how to install dependencies and run scripts in each language.

Try it yourself right now with the example script from earlier. Make sure you have Nix (or install it with curl https://nixos.org/nix/install | sh), get the script, and run it without caring about the dependencies!

by Georges Dubus at September 22, 2017 01:00 PM

September 11, 2017

Sander van der Burg

PNDP: An internal DSL for Nix in PHP

It has been a while since I wrote a Nix-related blog post. In many of my earlier Nix blog posts, I have elaborated about various Nix applications and their benefits.

However, when you are developing a product or service, you typically do not only want to use configuration management tools, such as Nix -- you may also want to build a platform that is tailored towards your needs, so that common operations can be executed structurally and conveniently.

When it is desired to integrate custom solutions with Nix-related tools, you basically have one recurring challenge -- you must generate deployment specifications in the Nix expression language.

The most obvious solution is to use string manipulation to generate the expressions we want, but this has a number of disadvantages. Foremost, composing strings is not a very intuitive activity -- it is not always obvious to see what the end result would be by looking at the code.

Furthermore, it is difficult to ensure that a generated expression is correct and safe. For example, if a string value is not properly escaped, it may be possible to inject arbitrary deployment code putting the security of the deployed system at risk.

For these reasons, I have developed NiJS: an internal DSL for JavaScript, a couple of years ago to make integration with JavaScript-based applications more convenient. Most notably, NiJS is used by node2nix to generate Nix expressions from NPM package deployment specifications.

I have been doing PHP development in the last couple of weeks and realized that I needed a similar solution for this language. In this blog post, I will describe PNDP, an internal DSL for Nix in PHP, and show how it can be used.

Composing Nix packages in PHP


The Nix packages repository follows a specific convention for organizing packages -- every package is a Nix expression file containing a function definition describing how to build a package from source code and its build-time dependencies.

A top-level composition expression file provides all the function invocations that build variants of packages (typically only one per package) by providing the desired versions of the build-time dependencies as function parameters.

Every package definition typically invokes stdenv.mkDerivation {} (or abstractions built around it) that composes a dedicated build environment in which only the specified dependencies can be found and other kinds of precautions are taken to improve build reproducibility. In this builder environment, we can execute many kinds of build steps, such as running GNU Make, CMake, or Apache Ant.

In our internal DSL in PHP we can replicate these conventions using PHP language constructs. We can compose a proxy to the stdenv.mkDerivation {} invocation in PHP by writing the following class:


namespace Pkgs;
use PNDP\AST\NixFunInvocation;
use PNDP\AST\NixExpression;

class Stdenv
{
public function mkDerivation($args)
{
return new NixFunInvocation(new NixExpression("pkgs.stdenv.mkDerivation"), $args);
}
}

In the above code fragment, we define a class named: Stdenv exposing a method named mkDerivation. The method composes an abstract syntax tree for a function invocation to stdenv.mkDerivation {} using an arbitrary PHP object of any type as a parameter.

With the proxy shown above, we can create our own in packages in PHP by providing a function definition that specifies how a package can be built from source code and its build-time dependencies:


namespace Pkgs;
use PNDP\AST\NixURL;

class Hello
{
public static function composePackage($args)
{
return $args->stdenv->mkDerivation(array(
"name" => "hello-2.10",

"src" => $args->fetchurl(array(
"url" => new NixURL("mirror://gnu/hello/hello-2.10.tar.gz"),
"sha256" => "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"
)),

"doCheck" => true,

"meta" => array(
"description" => "A program that produces a familiar, friendly greeting",
"homepage" => new NixURL("http://www.gnu.org/software/hello/manual"),
"license" => "GPLv3+"
)
));
}
}

The above code fragment defines a class named 'Hello' exposing one static method named: composePackage(). The composePackage method invokes the stdenv.mkDerivation {} proxy (shown earlier) to build GNU Hello from source code.

In addition to constructing a package, the above code fragment also follows the PHP conventions for modularization -- in PHP it is a common practice to modularize code chunks into classes that reside in their own namespace. For example, by following these conventions, we can also automatically load our package classes by using an autoloading implementation that follows the PSR-4 recommendation.

We can create compositions of packages as follows:


class Pkgs
{
public $stdenv;

public function __construct()
{
$this->stdenv = new Pkgs\Stdenv();
}

public function fetchurl($args)
{
return Pkgs\Fetchurl::composePackage($this, $args);
}

public function hello()
{
return Pkgs\Hello::composePackage($this);
}
}

As with the previous example, the composition example is a class. In this case, it exposes variants of packages by calling the functions with their required function arguments. In the above example, there is only one variant of the GNU Hello package. As a result, it suffices to just propagate the object itself as build parameters.

Contrary to the Nix expression language, we must expose each package composition as a method -- the Nix expression language is a lazy language that only invokes functions when their results are needed, PHP is an eager language that will evaluate them at construction time.

An implication of using eager evaluation is that opening the composition module, triggers all packages to be built. By wrapping the compositions into methods, we can make sure that only the requested packages are evaluated when needed.

Another practical implication of creating methods for each package composition is that it can become quite tedious if we have many of them. PHP offers a magic method named: __call() that gets invoked when we invoke a method that does not exists. We can use this magic method to automatically compose a package based on the method name:


public function __call($name, $arguments)
{
// Compose the classname from the function name
$className = ucfirst($name);
// Compose the name of the method to compose the package
$methodName = 'Pkgs\\'.$className.'::composePackage';
// Prepend $this so that it becomes the first function parameter
array_unshift($arguments, $this);
// Dynamically the invoke the class' composition method with $this as first parameter and the remaining parameters
return call_user_func_array($methodName, $arguments);
}

The above method takes the (non-existent) method name, converts it into the corresponding class name (by using the camel case naming convention), invokes the package's composition method using the composition object itself as a first parameter, and any other method parameters as successive parameters.

Converting PHP language constructs into Nix language constructs


Everything that PNDP does boils down to the phpToNix() function that automatically converts most PHP language constructs into semantically equivalent or similar Nix language constructs. For example, the following PHP language constructs are converted to Nix as follows:

  • A variable of type boolean, integer or double are converted verbatim.
  • A string will be converted into a string in the Nix expression language, and conflicting characters, such as the backslash and double quote, will be escaped.
  • In PHP, arrays can be sequential (when all elements have numeric keys that appear in numeric order) or associative in the remainder of the cases. The generator tries to detect what kind of array we have. It recursively converts sequential arrays into Nix lists of Nix language elements, and associative arrays into Nix attribute sets.
  • An object that is an instance of a class, will be converted into a Nix attribute set exposing its public properties.
  • A NULL reference gets converted into a Nix null value.
  • Variables that have an unknown type or are a resource will throw an exception.

As with NiJS (and JavaScript), the PHP host language does not provide equivalents for all Nix language constructs, such as values of the URL type, or encoding Nix function definitions.

You can still generate these objects by composing an abstract syntax from objects that are instances of the NixObject class. For example, when composing a NixURL object, we can generate a value of the URL type in the Nix expression language.

Arrays are a bit confusing in PHP, because you do not always know in advance whether it would yield a list or attribute set. To make these conversions explicit and prevent generation errors, they can be wrapped inside a NixList or NixAttrSet object.

Building packages programmatically


The PNDPBuild::callNixBuild() function can be used to build a generated Nix expression, such as the GNU Hello example shown earlier:


/* Evaluate the package */
$expr = PNDPBuild::evaluatePackage("Pkgs.php", "hello", false);

/* Call nix-build */
PNDPBuild::callNixBuild($expr, array());

In the code fragment above, we open the composition class file, named: Pkgs.php and we evaluate the hello() method to generate the Nix expression. Finally, we call the callNixBuild() function, in which we evaluate the generated expression by the Nix package manager. When the build succeeds, the resulting Nix store path is printed on the standard output.

Building packages from the command-line


As the previous code example is so common, there is also a command-line utility that can execute the same task. The following instruction builds the GNU Hello package from the composition class (Pkgs.php):


$ pndp-build -f Pkgs.php -A hello

It may also be useful to see what kind of Nix expression is generated for debugging or testing purposes. The --eval-only option prints the generated Nix expression on the standard output:


$ pndp-build -f Pkgs.js -A hello --eval-only

We can also nicely format the generated expression to improve readability:


$ pndp-build -f Pkgs.js -A hello --eval-only --format

Discussion


In this blog post, I have described PNDP: an internal DSL for Nix in PHP.

PNDP is not the first internal DSL I have developed for Nix. A couple of years ago, I also wrote NiJS: an internal DSL in JavaScript. PNDP shares a lot of concepts and implementation details with NiJS.

Contrary to NiJS, the functionality of PNDP is much more limited -- I have developed PNDP mainly for code generation purposes. In NiJS, I have also been exploring the abilities of the JavaScript language, such as exposing JavaScript functions in the Nix expression language, and the possibilities of an internal DSL, such as creating an interpreter that makes it a primitive standalone package manager. In PNDP, all this extra functionality is missing, since I have no practical need for them.

In a future blog post, I will describe an application that uses PNDP as a generator.

Availability


PNDP can obtained from Packagist as well as my GitHub page. It can be used under the terms and conditions of the MIT license.

by Sander van der Burg (noreply@blogger.com) at September 11, 2017 09:21 PM

August 23, 2017

Ollie Charles

Providing an API for extensible-effects and monad transformers

I was recently working on a small little project - a client API for the ListenBrainz project. Most of the details aren’t particularly interesting - it’s just a HTTP client library to a REST-like API with JSON. For the implementation, I let Servant and aeson do most of the heavy lifting, but I got stuck when considering what final API to give to my users.

Obviously, interacting with ListenBrainz requires some sort of IO so whatever API I will be offering has to live within some sort of monad. Currently, there are three major options:

  1. Supply an API targetting a concrete monad stack.

    Under this option, our API would have types such as

    submitListens :: ... -> M ()
    getListens :: ... -> M Listens

    where M is some particular monad (or monad transformer).

  2. Supply an API using type classes

    This is the mtl approach. Rather than choosing which monad my users have to work in, my API can be polymorphic over monads that support accessing the ListenBrainz API. This means my API is more like:

    submitListens :: MonadListenBrainz m => ... -> m ()
    getListens :: MonadListenBrainz m => ... -> m Listens
  3. Use an extensible effects framework.

    Extensible effects are a fairly new entry, that are something of a mix of the above options. We target a family of concrete monads - Eff - but the extensible effects framework lets our effect (querying ListenBrainz) seamlessly compose with other effects. Using freer-effects, our API would be:

    submitListens :: Member ListenBrainzAPICall effects => ... -> Eff effects ()
    getListens :: Member ListenBrainzAPICall effects => ... -> Eff effects Listens

So, which do we choose? Evaluating the options, I have some concerns.

For option one, we impose pain on all our users who want to use a different monad stack. It’s unlikely that you’re application is going to be written soley to query ListenBrainz, which means client code becomes littered with lift. You may write that off as syntatic, but there is another problem - we have committed to an interpretation strategy. Rather than describing API calls, my library now skips directly to prescribing how to run API calls. However, it’s entirely possible that you want to intercept these calls - maybe introducing a caching layer or additional logging. Your only option is to duplicate my API into your own project and wrap each function call and then change your program to use your API rather than mine. Essentially, the program itself is no longer a first class value that you can transform.

Extensible effects gives us a solution to both of the above. The use of the Member type class automatically reshuffles effects so that multiple effects can be combined without syntatic overhead, and we only commit to an interpretation strategy when we actually run the program. Eff is essentially a free monad, which captures the syntax tree of effects, rather than the result of their execution.

Sounds good, but extensible effects come with (at least) two problems that make me hesistant: they are experimental and esoteric, and it’s unclear that they are performant. By using only extensible effects, I am forcing an extensible effects framework on my users, and I’d rather not dictate that. Of course, extensible effects can be composed with traditional monad transformers, but I’ve still imposed an unnecessary burden on my users.

So, what do we do? Well, as Old El Paso has taught us: why don’t we have both?

It’s trivial to actually support both a monad transformer stack and extensible effects by using an mtl type class. As I argue in Monad transformers, free monads, mtl, laws and a new approach, I think the best pattern for an mtl class is to be a monad homomorphism from a program description, and often a free monad is a fine choice to lift:

class Monad m => MonadListenBrainz m where
  liftListenBrainz :: Free f a -> m a

But what about f? As observed earlier, extensible effects are basically free monads, so we can actually share the same implementation. For freer-effects, we might describe the ListenBrainz API with a GADT such as:

data ListenBrainzAPICall returns where
  GetListens :: ... -> ListenBrainzAPICall Listens
  SubmitListens :: ... -> ListenBrainzAPICall ()

However, this isn’t a functor - it’s just a normal data type. In order for Free f a to actually be a monad, we need f to be a functor. We could rewrite ListenBrainzAPICall into a functor, but it’s even easier to just fabricate a functor for free - and that’s exactly what Coyoneda will do. Thus our mtl type class becomes:

class Monad m => MonadListenBrainz m where
  liftListenBrainz :: Free (Coyoneda ListenBrainzAPICall) a -> m a 

We can now provide an implementation in terms of a monad transformer:

instance Monad m => MonadListenBrainz (ListenBrainzT m)
  liftListenBrainz f =
    iterM (join . lowerCoyoneda . hoistCoyoneda go)

    where
      go :: ListenBrainzAPICall a -> ListenBrainzT m a

or extensible effects:

instance Member ListenBrainzAPICall effs => MonadListenBrainz (Eff effs) where
  liftListenBrainz f = iterM (join . lowerCoyoneda . hoistCoyoneda send) f 

or maybe directly to a free monad for later inspection:

instance MonadListenBrainz (Free (Coyoneda ListenBrainzAPICall)) where
  liftListenBrainz = id

For the actual implementation of performing the API call, I work with a concrete monad transformer stack:

performAPICall :: Manager -> ListenBrainzAPICall a -> IO (Either ServantError a)

which both my extensible effects “run” function calls, or the go function in the iterM call for ListenBrainzT’s MonadListenBrainz instance.

In conclusion, I’m able to offer my users a choice of either:

  • a traditional monad transformer approach, which doesn’t commit to a particular intepretation strategy by using an mtl type class
  • extensible effects

All without extra syntatic burden, a complicated type class, or duplicating the implementation.

You can see the final implemantion of listenbrainz-client here.

Bonus - what about the ReaderT pattern?

The ReaderT design pattern has been mentioned recently, so where does this fit in? There are two options if we wanted to follow this pattern:

  • We require a HTTP Manager in our environment, and commit to using this. This has all the problems of providing a concrete monad transformer stack - we are committing to an interpretation.
  • We require a family of functions that explain how to perform each API call. This kind of like a van Laarhoven free monad, or really just explicit dictionary passing. I don’t see this really gaining much on abstracting with type classes.

I don’t feel like the ReaderT design pattern offers anything that isn’t already dealt with above.

by Oliver Charles at August 23, 2017 12:00 AM

July 29, 2017

Munich NixOS Meetup

NixOS Community of Practice

photoMunich NixOS Meetup

With the NixOS community of practice (CoP) we want to establish a regular get-together with people who are passionate about any Nix / NixOS related topic. There will be no talks and there is no agenda, instead we discuss, learn or work in pairs or groups on specific topics.


There will be food and drinks sponsored by HolidayCheck.

Munich - Germany

Thursday, August 3 at 5:00 PM

8

https://www.meetup.com/Munich-NixOS-Meetup/events/242120700/

July 29, 2017 04:26 PM

July 03, 2017

Typing nix

Records (again)

This week has been dedicated to records: parsing, simplification and (a bit of) typing.

Summary of the week

Parsing of records

Nothing really fancy here: I've extended the parser to support records, record patterns and record type annotations.

The record and record patterns are exactly the same as in plain Nix − except that they may contain type annotations. The record type annotations looks similar to records: { field_1; ...; field_n; } or { field_1; ...; field_n; ... }. The fields may have the form field_name = type or field_name =? type. The first one means that the field named field_name has type type and the second one that the field named field_name is optional and has type type if present.

Note that contrary to records, the field names can't be dynamic (this wouldn't really make sense), and the syntactic sugar x.y.z = type isn't available. The trailing semicolon is also optional.

For example, the record

{
  x = 1;
  y = {
    z = true;
    k = x: x + 1;
  };
}

has the type

{
  x = Int;
  y = {
    z = Bool;
    k = Int → Int;
  }
}

and the function

{ x /*: Int */, y /*: Bool */ ? true, ... }: if y then x else x+1

has the type

{ x = Int; y =? Bool; ... } → Int

Simplification

Nix has a special syntax for defining nested records:

{
  x.y = 1;
  x.z = 2;
}

Although this syntax is really practical, we can't directly work on such a representation, and we must transform a record such as the previous one into something of the form { x = { y = 1; z = 2; }; }.

This is a rather simple task in the general case, but happends to be impossible as soon as some field names are dynamic: How may one know whether { ${e1}.x = 1; ${e2}.y = 2; } is a record of the form { foo = { x = 1; y = 2; }; } or of the form { foo = { x = 1; }; bar = { y = 2; }; }?

So we have to be careful while treating them. Currently, I simply assume for the sake of simplicity that no conflict will occur (i.e. that the previous example will correspond to { foo = { x = 1; }; bar = { y = 2; }; }, but this is obviously wrong in the general case, so I'll have to add more subtlety in the future.

Typing of record patterns

The two previous points took me quite a long time (worse that what I expected to be honest), so I just began the actual typing work. I however got enough time to type the record patterns (which isn't really useful while I can't type record literals, but still).

More on this next week, because it makes more sense to present the whole record typing at once.

Next steps

Time flies, and I already have only two months left (minus some holidays that I'll have to take for my own sanity) until the end of my internship (my defense is planned on September, 1st). I plan to continue my work on this − and hope other people will join me once it reaches an advanced enough state − but I won't be able to be full-time on it, so it will likely advance much slower (hence I want to do as much as possible while I can spend all my days on it).

So here's a quick plan of what remains to do:

  • On the typing side of things:

    • Finish the typing of records

    • Write a default typing environment with all the built-in values.

      Some of them (like functionArgs for example) will be available as operators with custom typing rules because it isn't really possible to give them a meaningful type.

    • Get a real gradual type

      There's currently a type that is called "gradual" in the tool, but it is in fact nothing but a new type constant (akin to Int or Bool) without any special behaviour.

    • Fix the subtyping

      The subtyping relation that is used isn't exactly the one one would expect (because it has been designed for a strict language and Nix is lazy, and this introduces some differences in the subtyping).

      Unless some clever hack is found to reuse the existing algorithm this is gonna be really tricky and I may not be able to do it before September.

    • Rewrite the pretty-printing of types

      The pretty-printing of the types is currently handled by CDuce (the language whose type-system I based this one on), but as its types are slightly different from ours, the pretty-printing isn't always exactly accurate. So if it isn't too hard, I'll rewrite a custom pretty-printer.

  • On a more practical side:

    • Document everything

      Although all the documentation should be present in my work notes, this blog or my future thesis, a proper manual and tutorial has to be written for this to be usable by someone else than me.

    • Run more extensive tests on the parser to ensure its compatibility with plain Nix

    • Enhance the interface

      Apart from an OCaml library, the interface for the typechecker is a quickly written executable which I use for testing and which will require some work in order to be usable

    • Choose the default warnings and errors

      The errors reported by the tool are configurable, but a couple of defaults sets (the "strict" and the "permissive" modes) will cover 99% of the usages and avoids the need for the user to understand every detail of the type system.

    • Enhance the error reporting

      The error messages are rather rough and need to be polished

July 03, 2017 12:00 AM

June 26, 2017

Typing nix

Back to records

After some more engineering on the typer, I'll now start dealing with records

Summary of the week

This week has been mostly spent on engineering and preparing future work.

Improving the language coverage

The tool now parses and types some more languages constructs (like the assert or the include). I still have a little more work to do on this , like parsing the different flavours of strings/paths and the <foo> syntax, but this should be only some parsing work, and the typechecker itself should not be touched.

Warnings and errors

Before this week, the behaviour of the typechecker was to raise an exception whenever a type error occur, and return the type of the current expression otherwise. While this is sufficient for typechecking, it was not fully satisfactory, because in a near future I plan to also issue some − non fatal − warnings, which exceptions couldn't provide me.

So I refactored the typechecking engine to also return a list of warnings. A nice side-effect of this is that it was also trivial to make errors non-fatal − in case of a type-error, the typer just gives the type ? to the expression and adds the error to the list. This allows the typechecker to show more errors in one run.

Typer pragmas

In order to define locally which warnings are enabled, I also added some "pragmas" to the syntax (inside comments like for the type annotations in order to keep compatibility with standard Nix, of course). Those will be used to tell the typer to activate or deactivate certain warnings/errors for a given expression − or to switch between strict and permissive mode. Those modes will by the way most probably be defined as some presets of activated warnings and errors.

Records

Records are the next big step for this work.

I already briefly presented their typing in a previous post. The theory has globally stayed unchanged as this seems to be the better trade-off between usability and expressiveness. There will however be some differences between the strict and the permissive mode. Here are some notable cases that will covered by warnings.

Overlapping fields

If e is an arbitrary expression of type String, and e' of type τ then we know that we can type { e = e'; } with the type { _ = Undef ∧ τ; }, which means that each field of the record is either of type τ or undefined.

Now, let's assume that e1 and e2 both have type String (and e1' and e2' have type τ1 and τ2 respectively). We would like to type { e1 = e1'; e2 = e2'; } with { _ = Undef ∧ τ1 ∧ τ2 }, which would mean that each field of the record can have the type τ1 or τ2 or be undefined.

However, this isn't exactly true, because if both e1 and e2 evaluate to the same value, then the record is undefined. This can be ignored, of course, as if the record is undefined it can be given any type we want (and in particular { _ = Undef ∧ τ1 ∧ τ2 }), but there will probably be a warning for this, as it may lead to unpredictable failures.

Access to a maybe-undefined field

If e has type { _ = τ; }, then we can't ensure that accessing to any of its fields will succeed. Thus it will be forbidden in strict mode, unless guarded by an or clause. In a similar fashion, an expression such as e1.${e2} where we can't ensure that e2 evaluates to a label of e1 will be forbidden.

However, this a rather common pattern, so it will probably be allowed in permissive mode.

The with statement

The with construct is one of the worst evils in Nix. Records are a hard topic because they can have dynamic fields which are impossible to reason about in the general case. And with brings this difficulty to the top-level. This means that you can't even know which variables are in scope. And this is really bad. So it is possible that this construct will be simply forbidden in strict mode.

In permissive mode, a "best effort" typing will be done, depending on whether the imported record is statically known or not. It should also be possible (but probably later) to add some warnings about variable shadowing In case you don't know, with behaves quite weird when it comes to variable shadowing: If a variable is already in scope, it will not be redefined by the with statement. For example, let x = 1; in with { x = true; }; x evaluates to 1 and not true. This has been decided to prevent accidental variable capture, but it also causes accidental variable "non-capture" in the sense that you may be surprised if you didn't remember that the variable was already defined earlier. .

Misc

I also finalized last week a blog post for Tweag I/O about this work. It turned out to be quite popular, making the first page on Hacker News (even being #1 for a few minutes!) and hopefully made some good advertising for Nix in general.

June 26, 2017 12:00 AM

June 14, 2017

Typing nix

Things won are done, joy's soul lies in the doing

I couldn't find the time last week to make something presentable out of the typechecker as I promised, but almost all the features are already here − except records which I'll probably start implementing in a few weeks.

Implementation

State of the typechecker

I finished most of the engine for the language without records, so I'll now work on adding everything that is needed around, and handling some constructs that could be ignored from a theoretical point of view − mostly because it could be considered as just some syntactic sugar.

Here are the constructs that need to be considered:

Assert
The assert construct should be typed in a rather simple way, as assert e1; e2 is equivalent to if e1 then e2 else raise "Assertion failed" Well, it is not strictly equivalent because the exceptions raised by assert and the one raised by raise aren't exactly the same − see this comment −, but four our purposes, it is the same. .

Import
The simplest way to handle the import statement is to simply replace it by a dump of the imported file. This is probably what I will do in a first time. In a second time, it may be useful for performance reasons to type the imported file separately and memoïze the result (this is what nix does with the evaluation iirc).

Different modes of check

During the design of the type-system, there has been a constant tension between having a safe type-system and being able to typecheck as much existing code as possible.

Eventually, it became clear that both weren't compatible, and that in order to keep the best of both worlds, we needed to have (at least) two different versions (a "permissive" one and a "strict" one) of the type-system. So that's what we decided to do, with an annotation to switch from one to the other. Nothing has been implemented in this direction, and the boundary between strict and permissive hasn't been decided yet − especially, it will mostly concern records. But an important point is that the gradual type will be considered differently.

On the theoretical side

There hasn't been much improvement here, as I am still stuck with the proof of correctness, which is definitely harder than expected (but I must admit, I didn't work a lot on it). A PhD student of my advisor is also working on a really similar proof, and hopefully he will find a solution − as I don't really have much more time for this because I need to work on the implementation.

Next week

I'll start drafting headlines for my dissertation; this will allow me to see which points need some more work.

In parallel, I plan to make the type annotations for lists − this will probably deserve its own post as there is some really elegant stuff that I borrow from CDuce and that are worth presenting, as well as to treat the assert and import keywords.

I'm also rewriting the parser − this is already almost over, but there is still a few polishing work ongoing, as the old one was a quick and dirty crafted one, that I wrote in order to be quickly able to work on the typechecker itself, but which behaves quite badly − in particular regarding error messages.

June 14, 2017 12:00 AM

June 02, 2017

Typing nix

What is past is prologue

This week has been shared between a lot of easy fun putting together all the pieces that I worked on during the last weeks, and a lot of frustration trying to prove the soundness of the system.

Implementation

After several weeks spent trying to understand the API of CDuce − the language our type-system takes it inspiration from and whose code we planned to reuse as much as possible to ease the Implementation −, I eventually understood it enough to start using it for writing the inference engine.

This has been really enjoyable, as all I had to do was plugging together a few functions that where already dealing with all the boring low-level stuff, and everything has been almost instantly working. I could quickly write a prototype of typer for the core lambda-calculus of the language. It is still a little bit too rough on the edges to be presented, but I should (hopefully) be able to present some nice examples for next week.

On the theoretical side

I talked last week about some difficulties with the proof of the soundness of the type-system.

Those difficulties still hold, although some progress has been done. There are a couple of paths that we are currently exploring for a this:

  • We may change a bit the semantic of the language we work on (which is currently an approximation of the lazyness of nix) with something closer actual lazyness. This should probably solve our problems, but at the cost of redefining the language with a heavier and less practical formalism.

  • We may use a non-syntactic proof using something like what is described in this paper which a colleague from tweag suggested to me. While mathematically-heavy, this kind of proof has the advantage of being able to operate on our simplified semantic, which means we can keep our simple formalism everywhere else. It also is more modular than syntactic proofs, which means that we may prove some extensions of the core system without needing to rewrite the full proof.

Next week

Next week will be more or less like this one: I'll continue the implementation using all the nice features offered by CDuce. In the same time, I'll keep on working on the proof − I hope that this will be solved before next Friday.

Provided I find the time − which is rather unlikely, but who knows? − I'll also try to make an online demonstration of the engine to show the progress of the implementation.

June 02, 2017 12:00 AM