NixOS Planet

March 30, 2017

Munich NixOS Meetup

NixOS 17.03 Release Party

photoMunich NixOS Meetup

NixOS 17.03 "Gorilla" will be the next release scheduled to be released on March 30, 2017. Time to party on March 31, 2017!

The largest changes are:

• Nixpkgs is now extensible through overlays. See

• The setuid wrapper functionality now supports setting    capabilities.

• server uses branch 1.19. Due to ABI incompatibilities,    ati_unfree keeps forcing 1.17    and amdgpu-pro starts forcing 1.18.

• PHP now defaults to PHP 7.1

• Packages in nixpkgs can be marked as insecure through listed  vulnerabilities.

• This release is based on Glibc 2.25, GCC 5.4.0, systemd    232 and Nix 1.11.7. The default Linux kernel is 4.9.

The art is from http://www.hasanlai.c... released under CC-BY-SA-4.0.

München - Germany

Friday, March 31 at 7:00 PM


March 30, 2017 11:07 AM

March 20, 2017

Munich NixOS Meetup

NixOS 17.03 Release Sprint

photoMunich NixOS Meetup

The next stable NixOS release 17.03 'Gorilla' is going to happen on the 31st of March. The goal of this sprint is to fix critical issues before the release. The release manager will also attend and is available for guidance and feedback.

• Blocking issues:

• All 17.03 issues:

The sprint will be held at the Mayflower office in Munich on Saturday and Sunday starting at 11:00. Drinks will be provided.

The art is from http://www.hasanlai.c... released under CC-BY-SA-4.0.

München - Germany

Saturday, March 25 at 11:00 AM


March 20, 2017 08:49 PM

March 14, 2017

Sander van der Burg

Reconstructing Disnix deployment configurations

In two earlier blog posts, I have described Dynamic Disnix, an experimental framework enabling self-adaptive redeployment on top of Disnix. The purpose of this framework is to redeploy a service-oriented system whenever the conditions of the environment change, so that the system can still meet its functional and non-functional requirements.

An important category of events that change the environment are machines that crash and disappear from the network -- when a disappearing machine used to host a crucial service, a system can no longer meet its functional requirements. Fortunately, Dynamic Disnix is capable of automatically responding to such events by deploying the missing components elsewhere.

Although Dynamic Disnix supports the recovery of missing services, there is one particular kind of failure I did not take into account. In addition to potentially crashing target machines that host the services of which a service-oriented systems consist, the coordinator machine that initiates the deployment process and stores the deployment state could also disappear. When the deployment state gets lost, it is no longer possible to reliably update the system.

In this blog post, I will describe a new addition to the Disnix toolset that can be used to cope with these kinds of failures by reconstructing a coordinator machine's deployment configuration from the meta data stored on the target machines.

The Disnix upgrade workflow

As explained in earlier blog posts, Disnix requires three kinds of deployment models to carry out a deployment process: a services model capturing the components of which a system consists, an infrastructure model describing the available target machines and their properties, and a distribution model mapping services in the services model to target machines in the infrastructure model. By writing instances of these three models and running the following command-line instruction:

$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix

Disnix will carry out all activities necessary to deploy the system: building the services and its intra-dependencies from source code, distributing the services and its intra-dependencies, and activating all services in the right order.

When changing any of the models and running the same command-line instruction again, Disnix attempts to upgrade the system by only rebuilding the aspects that have changed, and only deactivating the obsolete services and activating new services.

Disnix (as well as other Nix-related tools) attempt to optimize a redeployment process by only executing the steps that are required to reach a new deployment state. In Disnix, the building and distribution steps are optimized due to the fact that every package is stored in isolation the Nix store in which each package has a unique filename with a hash prefix, such as:


As explained in a number of earlier blog posts, the hash prefix (acv1y1zf7w0i6jx02kfa6gxyn2kfwj3l...) is derived from all inputs used to build the package including its source code, build script, and libraries that it links to. That, for example, means that if we upgrade a system and nothing to the any of inputs of Firefox changes, we get an identical hash and if such a package build already exists, we do not have to build or transfer the package from an external site again.

The building step in Disnix produces a so-called low-level manifest file that is used by tools executing the remaining deployment activities:

<?xml version="1.0"?>
<manifest version="1">

The above manifest file contains the following kinds of information:

  • The distribution element section maps Nix profiles (containing references to all packages implementing the services deployed to the machine) to target machines in the network. This information is used by the distribution step to transfer packages from the coordinator machine to a target machine.
  • The activation element section contains elements specifying which service to activate on which machine in the network including other properties relevant to the activation, such as the type plugin that needs to be invoked that takes care of the activation process. This information is used by the activation step.
  • The targets section contains properties of the machines in the network and is used by all tools that carry out remote deployment steps.
  • There is also an optional snapshots section (not shown in the code fragment above) that contains the properties of services whose state need to be snapshotted, transferred and restored in case their location changes.

When a Disnix (re)deployment process successfully completes, Disnix stores the above manifest as a Disnix coordinator Nix profile on the coorindator machine for future reference with the purpose to optimize the successive upgrade step -- when redeploying a system Disnix will compare the generated manifest with the previously deployed generated instance and only deactivate services that have become obsolete and activating services that are new, making upgrades more efficient than fresh installations.

Unfortunately, when the coordinator machine storing the manifests gets lost, then also the deployment manifest gets lost. As a result, a system can no longer be reliably upgraded -- without deactivating obsolete services, newly deployed services may conflict with services that are already running on the target machines preventing the system from working properly.

Reconstructible manifests

Recently, I have modified Disnix in such a way that the deployment manifests on the coordinator machine can be reconstructed. Each Nix profile that Disnix distributes to a target machine includes a so-called profile manifest file, e.g. /nix/store/aiawhpk5irpjqj25kh6ah6pqfvaifm53-test1/manifest. Previously, this file only contained the Nix store paths to the deployed services and was primarily used by the disnix-query tool to display the installed set of services per machines.

In the latest Disnix, I have changed the format of the profile manifest file to contain all required meta data so that the the activation mappings can be reconstructed on the coordinator machine:

[{ target = "test2"; container = "process"; _key = "4827dfcde5497466b5d218edcd3326327a4174f2b23fd3c9956e664e2386a080"; } { target = "test2"; container = "process"; _key = "b629e50900fe8637c4d3ddf8e37fc5420f2f08a9ecd476648274da63f9e1ebcc"; } { target = "test1"; container = "process"; _key = "d85ba27c57ba626fa63be2520fee356570626674c5635435d9768cf7da943aa3"; }]

The above code fragment shows a portion of the profile manifest. It has a line-oriented structure in which every 7 lines represent the properties of a deployed service. The first line denotes the name of the service, second line the Nix store path, third line the Dysnomia container, fourth line the Dysnomia type, fifth line the hash code derived of all properties, sixth line whether the attached state must be managed by Disnix and the seventh line an encoding of the inter-dependencies.

The other portions of the deployment manifest can be reconstructed as follows: the distribution section can be derived by querying the Nix store paths of the installed profiles on the target machines, the snapshots section by checking which services have been marked as stateful and the targets section can be directly derived from a provided infrastructure model.

With the augmented data in the profile manifests on the target machines, I have developed a tool named disnix-reconstruct that can reconstruct a deployment manifest from all the meta data the manifests on the target machines provide.

I can now, for example, delete all the deployment manifest generations on the coordinator machine:

$ rm /nix/var/nix/profiles/per-user/sander/disnix-coordinator/*

and reconstruct the latest deployment manifest, by running:

$ disnix-reconstruct infrastructure.nix

The above command resolves the full paths to the Nix profiles on the target machines, then downloads their intra-dependency closures to the coordinator machine, reconstructs the deployment manifest from the profile manifests and finally installs the generated deployment manifest.

If the above command succeeds, then we can reliably upgrade a system again with the usual command-line instruction:

$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix

Extending the self-adaptive deployment framework

In addition to reconstructing deployment manifests that have gone missing, disnix-reconstruct offers another benefit -- the self-adaptive redeployment framework described in the two earlier blog posts is capable of responding to various kinds of events, including redeploying services to other machines when a machine crashes and disappears from the network.

However, when a machine disappears from the network and reappears at a later point in time, Disnix no longer knows about its configuration. When such a machine reappears in the network, this could have disastrous results.

Fortunately, by adding disnix-reconstruct to the framework we can solve this issue:

As shown in the above diagram, whenever a change in the infrastructure is detected, we reconstruct the deployment manifest so that Disnix knows which services are deployed to it. Then when the system is being redeployed, the services on the reappearing machines can also be upgraded or undeployed completely, if needed.

The automatic reconstruction feature can be used by providing the --reconstruct parameter to the self adapt tool:

$ dydisnix-self-adapt -s services.nix -i infrastructure.nix -q qos.nix \


In this blog post, I have described the latest addition to Disnix: disnix-reconstruct that can be used to reconstruct the deployment manifest on the coordinator machine from meta data stored on the target machines. With this addition, we can still update systems if the coordinator machine gets lost.

Furthermore, we can use this addition in the self-adaptive deployment framework to deal with reappearing machines that already have services deployed to them.

Finally, besides developing disnix-reconstruct, I have reached another stable point. As a result, I have decided to release Disnix 0.7. Consult the Disnix homepage for more information.

by Sander van der Burg ( at March 14, 2017 11:02 PM

March 02, 2017

Munich NixOS Meetup

guix/guixSD: Talk & Discussion

photoMunich NixOS Meetup

John Darrington will give a talk about guix and guixSD, a package manager and a Linux distribution which are based on similar concepts as nix/NixOS.

The talk will be in English.

Augsburg - Germany

Wednesday, March 29 at 7:00 PM


March 02, 2017 08:16 AM

December 31, 2016

Domen Kozar

Reflecting on 2016

Haven't blogged in 2016, but a lot has happened.

A quick summary of highlighted events:

2016 was a functional programming year as I've planned by end of 2015.

I greatly miss Python community and in that spirit, I've attended EuroPython 2016 and helped organize DragonSprint in Ljubljana. I don't think there's a place for me in OOP anymore, but I'll surely attend community events as nostalgia will kick in.

2017 seems extremely exciting, plans will unveil as I go, starting with some exciting news in January for Nix community.

by Domen Kožar at December 31, 2016 06:00 PM

December 27, 2016

Joachim Schiele


nixos assembly

if you want to join our nixos sprint, you can find our assembly here:

hope to see you!


by qknight at December 27, 2016 11:35 AM

December 20, 2016

Rok Garbas

Reproducible builds summit in Berlin

Last week a Reproducible Builds Summit was held in Berlin and since it was almost at my doorstep I had to take part. Along with Eelco Dolstra, I represented the NixOS voice in the debates that were happening at the summit.

Group picture of participant of Reproducible Builds summit in Berlin

When is a build reproducible?

There were quite few discussion sessions trying to define, when some piece of software is reproducible. While it is important to have a definition - in words, I was hoping that there would be a simple tool that could tell me this.

Such tool might never exist. What I realized during the summit is that reproducibility is not something which is true or false, but something that we is true until somebody disproves it. Reproducibility is a goal which we are always working towards, just like security.

As with zero vulnerability days, there should probably be zero reproducibility days.

Making sure that our software is reproducibility would follow similar practices we have already for security: 3rd party auditing, CVE-like database of reproducibility bugs, ...

Knowing if something is reproducible is not a simple yes/no question, but it is a process you need to follow. Yaay I am not too old to learn something :)

Different kinds of reproducibility

For the purpose of this blog post I would like to point out that there are - at least - two kinds of reproducibility.

When we talked about reproducibility at the summit we were of course referring to a bit-by-bit reproducibility or as I will continue to call it during this blog post binary reproducibility.

And then there is reproducibility which I like to call build reproducibility which only ensures the reproducibility of build environments (eg. versions of tools in build environments are the same).

Purpose of Reproducible Builds effort is of course to be binary reproducible, but to be able to build something bit by bit identical you need to use the same version of tools, which makes build reproducibility a pre-step of binary reproducibility.

Build reproducibility is a prerequirementof binary reproducibility.

Why this distinction matter I will explain in a bit, but for now just acknowledge this naming.

We are all biased

A leader in Reproducible Builds efforts is Debian community. You can see that Debian community is working hard on this and many Debian developers were present at the summit.

Getting the involvement outside of the Debian community is high on the list, since everybody realizes that only with common efforts we will be able to achive reproducibility nirvana.

But regardless of all the good intentions, I noticed two biases that I would like to point.

  • Many of us look down on language specific package managers (eg. pip, cabal, ...) as being less worthy usually talking about them as: "Who in their sane mind would use latest version of packages?". It sounds like a usual developer vs. sysadmin conversation. I hope at next summit we could also have representatives from some of the language specific package managers joined the discussions.
  • I got the impression that the sole reason of reproducible builds is that you would be more secure. That implies that everybody cares about security. Which would be great, but in a world with tight deadlines and startups security is usually the first thing that gets crossed out of the list. We need to make a more compelling reason then just security. I am aware that security is important to many, but we must also understand that it is not top priority for everybody.

Probably I missed some, because of my own biases, but the only way to overcome such thing is to try to expose each other biases.

Why would you care about reproducible builds?

My personal quest for this summit was to find better ways to market reproducible builds. I found that many who tried or want to introduce reproducible builds at work, fail because of few of reasons:

  • you need to opt in and switch from tools that you are used to.
  • many tools you are already using were not built with reproducible issues in mind.
  • reproducibility many times sounds like: all or nothing.
  • the prevalent (marketed) benefit of reproducible builds is better security. not everybody requires that level of security.
  • high cost (usually in developers hours) is usually required.

Reproducibility as productivity tool

What if we turn the marketing of reproducible builds around. What if the main (marketed) reason for the reproducible builds would be to improve developer productivity?

In previous paragraph I already explained that reproducibility is not a simple yes/no questions, but it is a process which one must follow. Also the path to reproducible builds is not a simple turn on/off switch. There are many steps on the way to binary reproducibility that already improve current development practices while getting us closer to binary reproducibility.

One example of this could be: reproducible development environments. If we have tools that could recreate any environment, couldn't we also use the for development purposes?

And we don't have to stop here. Isn't the most expensive part of fixing bugs, reproducing them? Couldn't build reproducibility also help us with reproducing certain type of bugs?

Maybe by now it became I little clearer, that there are many benefits along the way to binary reproducibility, that might be more convincing for some companies.

Cross platform build tool

Another discussion I watched from across the room was, BuildInfo specification. As part of reproducibility building Debian packages, also an BuildInfo file is (will be) produced which has all needed instructions, sources and final checksums, that allows somebody else to verify (reproduce) the resulting binary.

I was not alone thinking that this verification process should/could be distribution agnostic. Even a group was formed to discuss this, but sadly I was busy in other discussions to take part in this.

But then I realized that BuildInfo effort is actually changing a binary distribution like Debian into a source -> binary like distribution. Why produce BuildInfo file after the build process and why not start with it and only record the checksums of binaries after the build is done.

Is there such build tool that works across distributions would allow us to have BuildInfo specification (except the checksums) before the build process? Of course there is: Nix.

What many do not know about Nix is that Nix is first and foremost a build tool. It only happens that there is a database of packages already described how to be built and a side-effect we get is that Nix can also be a package manager. But initially it is a build tool. Nix can build .deb or .rpm packages or any other format you want.

What I would like to see is that whoever is looking into this direction that gives Nix a try and at least learn from it, because Nix and NixOS community is doing build reproducibility already for the last 10 years.


Few things I want you to take from this blog post are:

  • Go to the next Reproducible Builds summit. It was great. I hope to be there too.
  • Reproducibility is a process and not a state.
  • There are many useful steps before you reach reproducible nirvana. It might make sense to market those as well.
  • Nix is a cross distribution build tool. Use it. I know I will :)

by Rok Garbas at December 20, 2016 11:00 PM

December 14, 2016

Joachim Schiele



our latest encounter with rust was cargo which we replaced in a way that it is not used for deployment in nixos anymore.

more can be found at:


here is a tiny example using the tar-crate with the source from

tar-example source code

tar-example = stdenv.mkDerivation rec {
  name = "tar-example";
  src = ./example/src3;
  buildInputs = with allCrates; [ tar filetime libc xattr ]; 
  buildPhase = ''
    ${symlinkCalc buildInputs}

    # this creates the files needed for the test
    echo "hu" > file1.txt
    echo "bar" > file2.txt
    echo "batz" > file3.txt 

    ${rustcNightly}/bin/rustc $src/ --crate-type "bin" --emit=dep-info,link --crate-name main -L dependency=mylibs --extern tar=${allCrates.tar}/libtar.rlib
    if [ -f foo.tar ]; then
      echo -e "---------\nSUCCESS: tar-example was executed successfully!   \n--------"
      echo "FAIL: not working!"
    mkdir $out

after executing it

nix-build  -A tar-example
these derivations will be built:
building path(s) ‘/nix/store/hgravz7r38n3ic8msn94qy761zzi6jyw-tarunpacking sources
patching sources
no configure script, doing nothing
tar -  --extern libc=/nix/store/nc2jvn8rzbrbbqdwfwc7clzl99za9w2r-libc/liblibc.rlib --extern filetime=/nix/store/a5rr0mvyqnvq3mawhacwb49i101lyp4v-filetime/libfiletime.rlib
namefix tar
name tar
About to use rustc to compile some lib - tar
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/hgravz7r38n3ic8msn94qy761zzi6jyw-tar
patching script interpreter paths in /nix/store/hgravz7r38n3ic8msn94qy761zzi6jyw-tar
building path(s) ‘/nix/store/m0kslv072cphsk11n4696lzncc6rprc1-tar-exampleunpacking sources
unpacking source archive /nix/store/3q2yq22lh5shr0w4fxhcw8h1s61p6q9y-src3
source root is src3
patching sources
no configure script, doing nothing
warning: unused import: `std::io::prelude::*;`, #[warn(unused_imports)] on by default
 --> /nix/store/3q2yq22lh5shr0w4fxhcw8h1s61p6q9y-src3/
3 | use std::io::prelude::*;
  |     ^^^^^^^^^^^^^^^^^^^^

SUCCESS: tar-example was executed successfully!   
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/m0kslv072cphsk11n4696lzncc6rprc1-tar-example
patching script interpreter paths in /nix/store/m0kslv072cphsk11n4696lzncc6rprc1-tar-example


the project is still in its early stages but evolves into a reimplementation of cargo. however, it is pretty easy to be used and works for many crates already.

by qknight at December 14, 2016 07:35 PM

November 15, 2016

Joachim Schiele



paul and me visisted the augsburger openlab again!



  • package
    • ✔ build with ant
  • ✔ Initialize package tests


  • Quassel + qt4 doesn't support postgresql as database backend
    • ✔ Add an option to the quassel service to allow the qt5 version
  • "nixify" postfix configuration


paul, michael & qknight

  • ✔ started nextcloud packaging
  • ✔ leaps: packaged with tests:
  • ✔ made work on nixos,
  • fixed email system so that qknight can use thunderbird with STARTTLS and submission

    submissionOptions = {
      "smtpd_tls_security_level" = "encrypt";
      "smtpd_sasl_auth_enable" = "yes";
      "smtpd_client_restrictions" = "permit_sasl_authenticated,reject";
      "smtpd_sasl_type" = "dovecot";
      "smtpd_sasl_path" = "private/auth";
  • LXC: Unprivileged container with NixOS as guest and as host:
    • ✔ LXC container ist started as root that spawnes the LXC as user 100000 which is unprivileged on the host
    • ✔ shared read only store with the host
    • ✔ container can be build and updated on the host with nix-env


this sprint was awesome. we got so many things to work!

by qknight at November 15, 2016 12:10 PM

November 07, 2016

Anders Papitto

Scripting pulseaudio, bluetooth, jack

Posted on November 7, 2016

I’ve just leveled up my audio configuration, and there’s precious little information out there on how to script against pulseaudio and bluetooth and jack, so I’ll document it a bit.

Future versions of all snippets will live in my nixos configuration repo.

Here are some things that I have working - I can run jack alongside pulseaudio. youtube videos can play at the same time as audio software. - I can play audio through bluetooth headphones. I can even do this simultaneously with running jack - though, only pulseaudio outputs can be sent to the bluetooth headphones.

And that’s basically it. But it’s tricky to set up for the first time. So, let’s look at the implementation.

system configuration

First of all, I’ve scrapped the default /etc/pulse/, and written my own. It’s largely the same - I went through the default and copied in each line, except that I skipped all the modules that have to do with restoring streams and devices, etc. It’s quite annoying when pulseaudio has persistent state and tries to do “smart” things - I would rather my scripts be in full control. Note the presence of some jackdbus and bluetooth modules.

In that file, I also have some pam limits configured, as well as some kernel modules and /etc/jackdrc. Those precede my most recent pass of configuration, so I’m not sure if they’re actually necessary.

Assuming that you can get jack and/or bluetooth to work the first time, the interesting bits are the scripts that make them not a pain to deal with. I have two main scripts - switch-to-jack, switch-to-bluetooth. I also have a mostly-unused switch-to-stereo for completeness.

switch-to-jack looks like this

set -x
until pacmd list-sinks | egrep -q 'jack_out'
    jack_control start
pactl set-sink-volume jack_out 50%
pacmd set-default-sink jack_out
for index in $(pacmd list-sink-inputs | grep index | awk '{ print $2 }')
    pacmd move-sink-input $index jack_out

It starts jack with jack_control. This will cause jack to take exclusive control of the sound card. pulseaudio will notice this and add a sink and source for jack - however because all the ‘smart’ modules were removed, it won’t redirect any active streams to jack. They will just freeze, because they no longer have access to the sound card.

Then, I set the volume to something low enough to be safe - disabling the ‘smarts’ means that the jack device always gets added at 100% volume, which generally is too loud. After doing that, it’s safe to tell future audio to default to playing over jack, and then to move all existing streams there.

switch-to-bluetooth is similar


set -x

DEVICE=$(bluetoothctl <<< devices | egrep '^Device' | awk '{ print $2 }')
until bluetoothctl <<< show | grep -q 'Powered: yes'
    bluetoothctl <<< 'power on'
until pacmd list-sinks | egrep -q 'name:.*bluez_sink'
    bluetoothctl <<< "connect $DEVICE"

TARGET_CARD=$(pacmd list-cards | grep 'name:' | egrep -o 'bluez.*[^>]')
TARGET_SINK=$(pacmd list-sinks | grep 'name:' | egrep -o 'bluez.*[^>]')

until pacmd list-cards | egrep -q 'active profile: <a2dp_sink>'
    pacmd set-card-profile $TARGET_CARD a2dp_sink

pactl set-sink-volume $TARGET_SINK 30%
pacmd set-default-sink $TARGET_SINK
for index in $(pacmd list-sink-inputs | grep index | awk '{ print $2 }')
    pacmd move-sink-input $index $TARGET_SINK

A couple differences are

  • we make sure bluetooth is on
  • we have to connect to the specific device. If I had more than one set of bluetooth headphones this logic might be more complicated.
  • I have to always tell pulseaudio to use a2dp instead of hsp/hfp, because again I disabled the ‘remembering’ modules in pulseaudio.

Also note that this is the ‘steady-state’ implementation. The first time you want to connect a particular bluetooth device, you have to go through a little dance that looks something like this:

$ bluetoothctl
[NEW] Controller 60:57:18:9B:AB:71 BlueZ 5.40 [default]

[bluetooth]# agent on
[bluetooth]# Agent registered

[bluetooth]# discoverable on
[bluetooth]# Changing discoverable on succeeded
[CHG] Controller 60:57:18:9B:AB:71 Discoverable: yes

[bluetooth]# scan on
[bluetooth]# Discovery started
[CHG] Controller 60:57:18:9B:AB:71 Discovering: yes
[NEW] Device E8:07:BF:00:14:14 Mixcder ShareMe 7

[bluetooth]# [CHG] Device E8:07:BF:00:14:14 RSSI: -52

[bluetooth]# scan off
[bluetooth]# [CHG] Device E8:07:BF:00:14:14 RSSI is nil
Discovery stopped
[CHG] Controller 60:57:18:9B:AB:71 Discovering: no

[bluetooth]# devices
Device E8:07:BF:00:14:14 Mixcder ShareMe 7
[bluetooth]# pair E8:07:BF:00:14:14
Attempting to pair with E8:07:BF:00:14:14
[CHG] Device E8:07:BF:00:14:14 Connected: yes

[Mixcder ShareMe 7]# [CHG] Device E8:07:BF:00:14:14 Modalias: bluetooth:v0094p5081d0101
[CHG] Device E8:07:BF:00:14:14 UUIDs: 00001108-0000-1000-8000-00805f9b34fb
[CHG] Device E8:07:BF:00:14:14 UUIDs: 0000110b-0000-1000-8000-00805f9b34fb
[CHG] Device E8:07:BF:00:14:14 UUIDs: 0000110c-0000-1000-8000-00805f9b34fb
[CHG] Device E8:07:BF:00:14:14 UUIDs: 0000110e-0000-1000-8000-00805f9b34fb
[CHG] Device E8:07:BF:00:14:14 UUIDs: 0000111e-0000-1000-8000-00805f9b34fb
[CHG] Device E8:07:BF:00:14:14 UUIDs: 00001200-0000-1000-8000-00805f9b34fb
[CHG] Device E8:07:BF:00:14:14 Paired: yes
Pairing successful

[CHG] Device E8:07:BF:00:14:14 Connected: no

[bluetooth]# connect E8:07:BF:00:14:14
Attempting to connect to E8:07:BF:00:14:14
[CHG] Device E8:07:BF:00:14:14 Connected: yes
Connection successful

[Mixcder ShareMe 7]# exit
[Mixcder ShareMe 7]# Agent unregistered
[DEL] Controller 60:57:18:9B:AB:71 BlueZ 5.40 [default]

Abandoning ship

Laptop suspend causes bluetooth to disconnect. With the pulseaudio ‘smarts’ disabled, as far as I’m aware the bluetooth-connected streams will just die. To work around this, I switch everything to jack right before shutting down with a systemd service.

Other tips

Given the command line tooling, it’s not really necessary to use any guis. However, I found it convenient to use pavucontrol while setting all this up just to keep an eye on what was active and which streams were going where.

Note that I set volume with pactl, and most other things use pacmd. I’m not sure what the exact differences between these tools is, but pacmd doesn’t support setting a percent-based volume.

November 07, 2016 12:00 AM

November 06, 2016

Joachim Schiele

pulseaudio tcp streaming


this is a simple setup for streaming pulseaudio streams over the network.


hardware.pulseaudio = {
  enable = true;
  tcp.enable = true;
  tcp.anonymousClients.allowedIpRanges = [ "" ];
  zeroconf.publish.enable = true;



to use the new setup simply play some music and in pavucontrol you can select a different output device for the listed stream.

by qknight at November 06, 2016 12:35 AM

November 05, 2016

Joachim Schiele



a few months back i've replaced the odroid XU4 with this APU 2c4 board.

installing nixos

first have a look into the apu2 manual.

since there is no VGA/DVI output but only a RS232 serial interface we need to use that:

  1. serial cable

    out of simplicity i soldered one myself, the pins are:

    pin 2 to pin 3
    pin 3 to pin 2
    pin 5 to pin 5 (GND)

    i've been using this with a USB 2 RS232 converter

    # lsusb
    Bus 003 Device 003: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port
  2. connecting via serial console:

    picocom /dev/ttyUSB0 -b 115200
  3. nixos boot cd

    download the nixos-minimal-16.03.714.69420c5-x86_64-linux.iso and use unetbootin to deploy it to an USB stick. afterwards mount the first partition of the USB-stick and append this to the syslinux.cfg file's kernel command line:


    info: using the serial console you can see the GRUB output, see the kernel's output after boot and finally get a shell.

  4. booting from the USB stick

    the apu 2c4 features corebios and the process is straight forward, just hit F10 and select the USB stick

  5. nixos installation

    basically follow the nixos manual

    info: but don't forget to include this line in configuration.nix:

    boot.kernelParams = [ "console=ttyS0,115200n8" ];


# Edit this configuration file to define what should be installed on
# your system.  Help is available in the configuration.nix(5) man page
# and in the NixOS manual (accessible by running ‘nixos-help’).

{ config, pkgs, ... }:

  pw = import ./passwords.nix;
# setfacl -R -m u:joachim:rwx /backup

  imports =
    [ # Include the results of the hardware scan.

  # Use the GRUB 2 boot loader.
  boot.loader.grub.enable = true;
  boot.loader.grub.version = 2;
  # Define on which hard drive you want to install Grub.
  boot.loader.grub.device = "/dev/sda";

  boot.kernelParams = [ "console=ttyS0,115200n8" ];

  networking = {
    hostName = "apu-nixi"; # Define your hostname.
    bridges.br0.interfaces = [ "enp1s0" "wlp4s0" ];
    firewall = {
      enable = true;
      allowPing = true;
      allowedTCPPorts = [ 22 ];
      #allowedUDPPorts = [ 5353 ];


  # networking.wireless.enable = true;  # Enables wireless support via wpa_supplicant.

  #Select internationalisation properties.
  i18n = {
    consoleFont = "Lat2-Terminus16";
    consoleKeyMap = "us";
    defaultLocale = "en_US.UTF-8";

  security.sudo.enable = true;

  programs.zsh.enable = true;
  users.defaultUserShell = "/run/current-system/sw/bin/zsh";

  services = {
    nscd.enable = true;
    ntp.enable = true;
    klogd.enable = true;
    nixosManual.enable = false; # slows down nixos-rebuilds, also requires nixpkgs.config.allowUnfree here..?
    xserver.enable = false;

    cron = {
      enable = true;
      mailto = "";
      systemCronJobs = [
        "0 0,8,16 * * * joachim cd /backup/; ./"
        #*     *     *   *    *            command to be executed
        #-     -     -   -    -
        #|     |     |   |    |
        #|     |     |   |    +----- day of week (0 - 6) (Sunday=0)
        #|     |     |   +------- month (1 - 12)
        #|     |     +--------- day of month (1 - 31) 
        #|     +----------- hour (0 - 23)
        #+------------- min (0 - 59)

  # Set your time zone.
  # time.timeZone = "Europe/Amsterdam";

  # List packages installed in system profile. To search by name, run:
  environment.systemPackages = with pkgs; [

  time.timeZone = "Europe/Berlin";

  # Enable the OpenSSH daemon.
  services.openssh = {
    enable = true;
    permitRootLogin = "without-password";
  }; = [ "sys-subsystem-net-devices-wlp4s0.device" ];

  services.hostapd = {
    enable = true;
    wpaPassphrase = pw.wpaPassphrase;
    interface = "wlp4s0";

  # Define a user account. Don't forget to set a password with ‘passwd’.
  users.extraUsers.joachim = {
    isNormalUser = true;
    uid = 1000;

  # The NixOS release to be compatible with for stateful data such as databases.
  system.stateVersion = "16.09";

WD passport USB 3.0 bug

with a WD passport USB 3.0 disk i can't boot the system since i hit this bug.

SeaBIOS (version ?-20160311_005214-0c3a223c2ee6)
XHCI init on dev 00:10.0: regs @ 0xfea22000, 4 ports, 32 slots, 32 byte contexts
XHCI    extcap 0x1 @ fea22500
XHCI    protocol USB  3.00, 2 ports (offset 1), def 0
XHCI    protocol USB  2.00, 2 ports (offset 3), def 10
XHCI    extcap 0xa @ fea22540
Found 2 serial ports
ATA controller 1 at 4010/4020/0 (irq 0 dev 88)
EHCI init on dev 00:13.0 (regs=0xfea25420)
ATA controller 2 at 4018/4024/0 (irq 0 dev 88)
Searching bootorder for: /pci@i0cf8/*@14,7
Searching bootorder for: /rom@img/memtest
Searching bootorder for: /rom@img/setup
ata0-0: KINGSTON SMS200S360G ATA-8 Hard-Disk (57241 MiBytes)
Searching bootorder for: /pci@i0cf8/*@11/drive@0/disk@0
XHCI port #3: 0x002202a0, powered, pls 5, speed 0 [ - ]
XHCI port #1: 0x00021203, powered, enabled, pls 0, speed 4 [Super]
Searching bootorder for: /pci@i0cf8/usb@10/storage@1/*@0/*@0,0
Searching bootorder for: /pci@i0cf8/usb@10/usb-*@1
USB MSC vendor='WD' product='My Passport 0827' rev='1012' type=0 removable=0
call16 with invalid stack
PCEngines apu2
coreboot build 20160311


how to recover a bricked BIOS (after flashing)? on APU1 it was SPI, and there's a header, so like wires + a ch341a should do it.


the APU i'm using also has a Mini-PCIe wireless card built and you can choose from these two cards:

the access point works nicely with my android devices as well as my linux laptops.

buy the APU

if you want to buy an APU, buy the APU bundle.


the APU is runnig NixOS and is very stable and fast while using little energy. would use/buy again!

by qknight at November 05, 2016 02:35 PM

November 04, 2016

Edward Tjörnhammar

Plant Based UML Wiki

Turn Gitit into a UML renderer

November 04, 2016 12:00 AM

October 26, 2016

Sander van der Burg

Push and pull deployment of Nix packages

In earlier blog posts, I have covered various general concepts of the Nix package manager. For example, I have written three blog posts explaining Nix from a system administrator, programming language, and a sales perspective.

Furthermore, I have written a number of blog posts demonstrating how Nix can be used, such as managing a set of private packages, packaging binary-only software, and Nix's declarative nature -- the Nix package manager (as well as other tools in the Nix project), are driven by declarative specifications describing the structure of the system to deploy (e.g. the packages and its dependencies). From such a specification, Nix carries out all required deployment activities, including building the packages from its sources, distributing packages from the producer to consumer site, installation and uninstallation.

In addition to the execution of these activities, Nix provides a number of powerful non-functional properties, such as strong guarantees that dependency specifications are complete, that package builds can be reproduced, and that upgrades are non-destructive, atomic, and can always be rolled back.

Although many of these blog posts cover the activities that the Nix package manager typically carries out (e.g. the three explanation recipes cover building and the blog post explaining the user environment concepts discusses installing, uninstalling and garbage collection), I have not elaborated much about the distribution mechanisms.

A while ago, I noticed that people were using one of my earlier blog post as a reference. Despite being able to set up a collection of private Nix expressions, there were still some open questions left, such as fully setting up a private package repository, including the distribution of package builds.

In this blog post, I will explain Nix's distribution concepts and show how they can be applied to private package builds. As with my earlier blog post on managing private packages, these steps should be relatively easy to repeat.

Source and transparent binary deployments

As explained in my earlier blog post on packaging private software, Nix is in principle a source-based package manager. Nix packages are driven by Nix expressions describing how to build packages from source code and all its build-time dependencies, for example:

{ stdenv, fetchurl
, pkgconfig, perl, glib, gpm, slang, zip, unzip, file, gettext
, libX11, libICE, e2fsprogs

stdenv.mkDerivation {
name = "mc-4.8.12";

src = fetchurl {
url =;
sha256 = "15lkwcis0labshq9k8c2fqdwv8az2c87qpdqwp5p31s8gb1gqm0h";

buildInputs = [ pkgconfig perl glib gpm slang zip unzip file gettext
libX11 libICE e2fsprogs ];

meta = {
description = "File Manager and User Shell for the GNU Project";
homepage =;
license = "GPLv2+";
maintainers = [ stdenv.lib.maintainers.sander ];

The above expression (mc.nix) describes how to build Midnight Commander from source code and its dependencies, such as pkgconfig, perl and glib. Because the above expression does not specify any build procedure, the Nix builder environment reverts to the standard GNU Autotools build procedure, that typically consists of the following build steps: ./configure; make; make install.

Besides describing how to build a package, we must also compose a package by providing it the right versions or variants of the dependencies that it requires. Composition is typically done in a second expression referring to the former:

{ nixpkgs ? <nixpkgs>
, system ? builtins.currentSystem

pkgs = import nixpkgs { inherit system; };

callPackage = pkgs.lib.callPackageWith (pkgs // pkgs.xlibs // self);

self = rec {
mc = callPackage ./mc.nix { };

# Other package imports

In the above expression (default.nix), the mc attribute imports our former expression and provides the build-time dependencies as function parameters. The dependencies originate from the Nixpkgs collection.

To build the Nix expression shown above, it typically suffices to run:

$ nix-build -A mc

The result of a Nix build is a Nix store path in which the build result is stored.

As may be noticed, the prefix of the package name (jal99995sk6rixym4gfwcagmdiqrwv9a) is a SHA256 hash code that has been derived from all involved build dependencies, such as the source tarball, build-time dependencies and build scripts. Changing any of these dependencies (such as the version of the Midnight Commander) triggers a rebuild and yields a different hash code. Because hash codes ensure that the Nix store paths to packages will be unique, we can safely store multiple versions and variants of the same packages next to each other.

In addition to executing builds, Nix also takes as many precautions to ensure purity. For example, package builds are carried out in isolated environments in which only the specified dependencies can be found. Moreover, Nix uses all kinds of techniques to make builds more deterministic, such as resetting the timestamps of all files to 1 UNIX time, making build outputs read-only etc.

The combination of unique hash codes and pure builds results in a property called transparent binary deployments -- a package with an identical hash prefix results in a (nearly) bit-identical build regardless on what machine the build has been performed. If we want to deploy a package with a certain hash prefix that already exists on a trustable remote machine, then we can also transfer the package instead of building it again.

Distribution models

The Nix package manager (as well as its sub projects) support two kinds of distribution models -- push and pull deployment.

Push deployment

Push deployment is IMO conceptually the simplest, but at the same time, infrequently used on package management level and not very well-known. The idea of push deployment is that you take an existing package build on your machine (the producer) and transfer it elsewhere, including all its required dependencies.

With Nix this can be easily accomplished with the nix-copy-closure command, for example:

$ nix-copy-closure --to \

The above command serializes the Midnight Commander store path including all its dependencies, transfers them to the provided target machine through SSH, and then de-serializes and imports the store paths into the remote Nix store.

An implication of push deployment is that the producer requires authority over the consumer machine. Moreover, nix-copy-closure can transfer store paths from one machine to another, but does not execute any additional deployment steps, such as the "installation" of packages (in Nix packages become available to end-users by composing a Nix user environment that is in the user's PATH).

Pull deployment

With pull deployment the consumer machine is control instead of the producer machine. As a result, the producer does not require any authority over another machine.

As with push deployment, we can also use the nix-copy-closure command for pull deployment:

$ nix-copy-closure --from \

The above command invocation is similar to the previous example, but instead copies a closure of the Midnight Commander from the producer machine.

Aside from nix-copy-closure, Nix offers another pull deployment mechanism that is more powerful and more commonly used, namely: channels. This is what people typically use when installing "end-user" packages with Nix.

The idea of channels is that they are remote HTTP/HTTPS URLs where you can subscribe yourself to. They provide a set of Nix expressions and a binary cache of substitutes:

$ nix-channel --add

The above command adds the NixOS unstable channel to the list of channels. Running the following command:

$ nix-channel --update

Fetches or updates the collection of Nix expressions from the channels allowing us to install any package that it provides. By default, the NIX_PATH environment variable is configured in such a way that they refer to the expressions obtained from channels:

$ echo $NIX_PATH

With a preconfigured channel, we can install any package we want including prebuilt substitutes, by running:

$ nix-env -i mc

The above command installs the Midnight Commander from the set of Nix expressions from the channel and automatically fetches the substitutes of all involved dependencies. After the installation succeeds, we can start it by running:

$ mc

The push deployment mechanisms of nix-copy-closure and pull deployment mechanisms of the channels can also be combined. When running the following command:

$ nix-copy-closure --to \
/nix/store/jal99995sk6rixym4gfwcagmdiqrwv9a-mc-4.8.12 \

The consumer machine first attempts to pull the substitutes of the dependency closure from the binary caches first, and finally the producer pushes the missing packages to the consumer machine. This approach is particularly useful if the connection between the producer and the consumer is slow, but the connection between the consumer and the binary cache is fast.

Setting up a private binary cache

With the concepts of push and pull deployment explained, you may probably wonder how these mechanisms can be used to your own private set of Nix packages?

Fortunately, with nix-copy-closure no additional work is required as it works on any store path, regardless of how it is produced. However, when it is desired to set up your own private binary cache, some additional work is required.

As channels/binary caches use HTTP as a transport protocol, we need to set up a web server with a document root folder serving static files. In NixOS, we can easily configure an Apache HTTP server instance by adding the following lines to a NixOS configuration: /etc/nixos/configuration.nix:

services.httpd = {
enable = true;
adminAddr = "";
hostName = "producer";
documentRoot = "/var/www";

With the nix-push command we can generate a binary cache for our private package set (that includes the Midnight Commander and all its dependencies):

$ nix-push --dest /var/www $(nix-build default.nix)

As may be observed by inspecting the document root folder, we now have a set of compressed NAR files representing the serialized forms of all packages involved and narinfo files capturing a package's metadata:

$ ls /var/www

In addition to the binary cache, we also need to make the corresponding Nix expressions available. This can be done by simply creating a tarball out of the private Nix expressions and publishing the corresponding file through the web server:

tar cfvz /var/www/custompkgs.tar.gz custompkgs

On the customer machine, we need to configure the binary cache, by adding the following property to /etc/nix.conf:

binary-caches =

In NixOS, this property can be set by adding the following property to /etc/nixos/configuration.nix:

nix.binaryCaches = [ "" ];
nix.requireSignedBinaryCaches = false;

Additionally, we have to configure the NIX_PATH environment variable to refer to our tarball of Nix expressions:

$ export NIX_PATH=custompkgs=$NIX_PATH

Now, when we run the following command-line instruction:

$ nix-env -f '<custompkgs>' -iA mc
downloading ‘’... [0/0 KiB, 0.0 KiB/s]
unpacking ‘’...
installing ‘mc-4.8.12’

We can install our custom Midnight Commander package by pulling the package from our own custom HTTP server without explicitly obtaining the set of Nix expressions or building it from source code.


In this blog post, I have explained Nix's push and pull deployment concepts and shown how we can use them for a set of private packages, including setting up a private binary cache. The basic idea of binary cache distribution is quite simple: create a tarball of your private Nix expressions, construct a binary cache with nix-push and publish the files with an HTTP server.

In real-life production scenarios, there are typically more aspects you need to take into consideration beyond the details mentioned in this blog post.

For example, to make binary caches safe and trustable, it is also recommended to use HTTPS instead of plain HTTP connections. Moreover, you may want to sign the substitutes with a cryptographic key. The manual page of nix-push provides more details on how to set this up.

Some inconvenient aspects of the binary cache generation approach shown in this blog post (in addition to the fact that we need to set up an HTTP server), is that the approach is static -- whenever, we have a new version of a package built, we need to regenerate the binary cache and the package set.

To alleviate these inconveniences, there is also a utility called nix-serve that spawns a standalone web server generating substitutes on the fly.

Moreover, the newest version of Nix also provides a so-called binary cache Nix store. When Nix performs operations on the Nix store, it basically talks to a module with a standardized interface. When using the binary cache store module (instead of the standalone or remote Nix store plugin), Nix automatically generates NAR files for any package that gets imported into the Nix store, for example after the successful completion a build. Besides an ordinary binary cache store plugin, there is also plugin capable of uploading substitutes directly to an Amazon AWS S3 bucket.

Apart from the Nix package manager, also Nix-related projects use Nix's distribution facilities in some extent. Hydra, the Nix-based continuous integration server, also supports pull deployment as it can dynamically generate channels from jobsets. Users can subscribe to these channels to install the bleeding-edge builds of a project.

NixOps, a tool that deploys networks of NixOS configurations and automatically instantiates VMs in the cloud, as well as Disnix, a tool that deploys service-oriented systems (distributed systems that can be decomposed in a "distributable units", a.k.a. services) both use push deployment -- from a coordinator machine (that has authority over a collection of target machines) packages are distributed.

Concluding remarks

After writing this blog post and some thinking, I have become quite curious to see what a pull deployment variant of Disnix (and maybe NixOps) would look like.

Although Disnix and NixOps suit all my needs at the moment, I can imagine that when we apply the same concepts in large organizations with multiple distributed teams, it can no longer considered to be practical to work with a centralized deployment approach that requires authority over all build artefacts and the entire production environment.

by Sander van der Burg ( at October 26, 2016 09:38 PM

Joachim Schiele



inspired by the shackspace's grafana usage for moisture/temperature monitoring i wanted to use grafana myself. since i'm also active in the fablab neckar-alb and we are having a nice project, called cycle-logistics tübingen, to monitor this seemd to be a good opportunity to apply this toolchain.

we are interested in the voting behaviour:

this blog posting documents all steps needed to rebuild the setup so you can leverage this toolchain for your own projects!

here is a screenshot of how it looks:


below you can find a detailed listing and discussion of the single programs used. the source code can be found on github, except the nixos specific parts which is listed below exclusively.


selenium is used to visit, parse the DOM tree and to export the data as data.json.

to execute one needs an python environment with two additional libraries. nix-shell along with collect_data-environment.nix is used to create that environment on the fly.

#! /usr/bin/env nix-shell
#! nix-shell collect_data-environment.nix --command 'python3'

from selenium import webdriver
from selenium import selenium
from selenium.webdriver.common.keys import Keys

from import By
from import WebDriverWait # available since 2.4.0
from import expected_conditions as EC # available since 2.26.0
from pyvirtualdisplay import Display

import sys
import os

display = Display(visible=0, size=(800, 600))

from distutils.version import LooseVersion, StrictVersion
if LooseVersion(webdriver.__version__) < LooseVersion("2.51"):
    sys.exit("error: version of selenium ("
        + str(LooseVersion(webdriver.__version__))
        + ") is too old, needs 2.51 at least")

ff = webdriver.Firefox()

st = ""

v = ff.execute_script("""
var t = document.getElementById('profile').childNodes; 
var ret = []
for (var i = 0; i < t.length; i++) {
  if ('id' in t[i]) {
    if(t[i].id.includes('profil-')) {
      var myID = t[i].id.replace("profil-","");
      var myVotes = t[i].getElementsByClassName('profile-txt-stimmen')[0].innerHTML;
      var myTitle = t[i].getElementsByClassName('archive-untertitel')[0].innerHTML;
      var myVerein = t[i].getElementsByClassName('archive-titel')[0].innerHTML;
      //console.log(myID,myVerein, myTitle, myVotes)
      var r = new Object();
      r.Id = parseInt(myID);
      r.Votes = parseInt(myVotes);
      r.Verein = myVerein; 
      r.Title = myTitle;
var date = new Date();
var exp = {};
exp.Date = date;
exp.Data = ret;
var j = JSON.stringify(exp,  null, "\t");
return j;

print (v)



with import <nixpkgs> { };

  pkgs1 = import (pkgs.fetchFromGitHub {
    owner = "qknight";
    repo = "nixpkgs";
    rev = "a1dd8b2a5b035b758f23584dbf212dfbf3bff67d";
    sha256 = "1zn9znsjg6hw99mshs0yjpcnh9cf2h0y5fw37hj6pfzvvxfrfp9j";
  }) {};


pkgs1.python3Packages.buildPythonPackage rec {
  name = "crawler";
  version = "0.0.1";

  buildInputs = [ pkgs.firefox xorg.xorgserver ];

  propagatedBuildInputs = with pkgs1.python3Packages; [
    virtual-display selenium


info: in the above environment, two different versions of nixpkgs are mixed, which is a nix speciality.

virtual-display and selenium from an older nixpkgs base called pkgs1 while firefox is the one coming with the nixos operating system called pkgs.

golang import

the go based importer is very simple and basically follows the example from the influxdb client code base.

if you want to build the inject_into_fluxdb binary you can simply use nix-shell and inside that shell simply type go build. you have to put that binary into the right place, which is /var/lib/crawler/, manually since this was only a prototype.

warning use nixos-rebuild switch with the nixos specific changes from below first so that the nixos system will create the user/group and directory (crawler/crawler and /var/lib/crawler). and when you deploy stuff into that directory, make sure you use chown crawler:crawler . -R in that directory.


package main

import (
//   "os"

type rec struct {
  Id     int
  Votes  int
  Verein string
  Title  string

type json_message struct {
  Date string
  Data []rec

const (
  MyDB     = "square_holes"
  username = "bubba"
  password = "bumblebeetuna"

func main() {

  f, err2 := ioutil.ReadFile("data.json")
  if err2 != nil {
    log.Fatalln("Error: ", err2)

  var l json_message
  err2 = json.Unmarshal(f, &l)
  if err2 != nil {
    log.Fatalln("Error: ", err2)

  // Make client
  c, err := client.NewHTTPClient(client.HTTPConfig{
    Addr:     "http://localhost:8086",
    Username: username,
    Password: password,

  if err != nil {
    log.Fatalln("Error: ", err)

  // Create a new point batch
  bp, err := client.NewBatchPoints(client.BatchPointsConfig{
    Database:  MyDB,
    Precision: "s",

  if err != nil {
    log.Fatalln("Error: ", err)
  layout := "2006-01-02T15:04:05.000Z"
  t, err3 := time.Parse(layout, l.Date)

  if err3 != nil {
  for _, r := range l.Data {
    pt, err := client.NewPoint("frei", map[string]string{"Id": strconv.Itoa(r.Id)}, structs.Map(r), t.Local())
    if err != nil {
      log.Fatalln("Error: ", err)

  // Write the batch


go2nix in version 1.1.1 has been used to generate the default.nix and deps.nix automatically. this is also the reason for the weird directory naming inside the GIT repo.

warning there are two different implementation of a go to nix dependency converter and both are called go2nix. i was using the one from kamilchm the other never worked for me.

# This file was generated by go2nix.
    goPackagePath = "";
    fetch = {
      type = "git";
      url = "";
      rev = "dc3312cb1a4513a366c4c9e622ad55c32df12ed3";
      sha256 = "0wgm6shjf6pzapqphs576dv7rnajgv580rlp0n08zbg6fxf544cd";
    goPackagePath = "";
    fetch = {
      type = "git";
      url = "";
      rev = "6fa145943a9723f9660586450f4cdcf72a801816";
      sha256 = "14ggx1als2hz0227xlps8klhn5s478kczqx6i6l66pxidmqz1d61";


info: go2nix generates a default.nix which is basically a dropin when used in nixpkgs but i wanted to use it with nix-shell so a few lines needed changes. just be awere of that!

{ pkgs ? import <nixpkgs>{} } :
  stdenv = pkgs.stdenv;
  buildGoPackage = pkgs.buildGoPackage;
  fetchgit = pkgs.fetchgit;
  fetchhg = pkgs.fetchhg;
  fetchbzr = pkgs.fetchbzr; 
  fetchsvn = pkgs.fetchsvn;

buildGoPackage rec {
  name = "crawler-${version}";
  version = "20161024-${stdenv.lib.strings.substring 0 7 rev}";
  rev = "6159f49025fd5500e5c2cf8ceeca4295e72c1de5";

  goPackagePath = "fooooooooooo";

  src = ./.;

  goDeps = ./deps.nix;

  meta = {



info: in the configuration.nix excerpt apache, which is called httpd in nixos`, is used as a reverse proxy. you don't have to follow that example but it is a nice setup once one gets it working.

  imports =
    [ # Include the results of the hardware scan.
  services.grafana = {
    security = {
    users = {
      allowSignUp = false;
      allowOrgCreate = true;
    analytics.reporting.enable = false;


  services.influxdb = {
    enable = true;
  services.httpd = {
    enable = true;
    enablePHP = true;
    logPerVirtualHost = true;
    hostName = "";
    extraModules = [
      { name = "php7"; path = "${pkgs.php}/modules/"; }
      { name = "deflate"; path = "${pkgs.apacheHttpd}/modules/"; }
      { name = "proxy_wstunnel"; path = "${pkgs.apacheHttpd}/modules/"; }
 virtualHosts =
      # (https)
        hostName = "";
        serverAliases = [ "" "" ];

        documentRoot = "/www/";
        enableSSL = true;
        sslServerCert = "/ssl/acme/";
        sslServerKey = "/ssl/acme/";
       extraConfig = ''
          Alias /.well-known/acme-challenge /var/www/challenges/
          <Directory "/var/www/challenges/">
            Options -Indexes
            AllowOverride None
            Order allow,deny
            Allow from all
            Require all granted
          RedirectMatch ^/$ /main/
          #Alias /main /www/
          <Directory "/www/">
            Options -Indexes
            AllowOverride None
            Require all granted

          SetOutputFilter DEFLATE

          <Directory "/www/">
            Options -Indexes
            AllowOverride None
            Order allow,deny
            Allow from all

          # prevent a forward proxy! 
          ProxyRequests off

          # User-Agent / browser identification is used from the original client
          ProxyVia Off
          ProxyPreserveHost On

          RewriteEngine On
          RewriteRule ^/grafana$ /grafana/ [R]

          <Proxy *>
          Order deny,allow
          Allow from all

          ProxyPass /grafana/ retry=0
          ProxyPassReverse /grafana/


simply put crawler.nix into /etc/nixos and reference it from configuration.nix using the imports directive.

{ config, pkgs, lib, ... } @ args:

#with lib;

  cfg =;
  stateDir = "/var/lib/crawler/";
in { 
  config = {
    users = {
      users.crawler= {
        #note this is a hack since this is not commited to the nixpkgs
        uid             = 2147483647;
        description     = "crawler server user";
        group           = "crawler";
        home            = stateDir;
        createHome      = true;

      groups.crawler= {
        #note this is a hack since this is not commited to the nixpkgs
        gid = 2147483648;
    }; {
      script = ''
          source /etc/profile
          export HOME=${stateDir}
          ${stateDir}/ > ${stateDir}/data.json
          cd ${stateDir}
      serviceConfig = {
        Nice = 19;
        IOSchedulingClass = "idle";
        PrivateTmp = "yes";
        NoNewPrivileges = "yes";
        ReadWriteDirectories = stateDir;
        WorkingDirectory = stateDir;

    systemd.timers.crawler = { 
      description = "crawler service";
      partOf      = [ "crawler.service" ];
      wantedBy    = [ "" ];
      timerConfig = {
        OnCalendar = "*:0/30";
        Persistent = true;

info: note the timerConfig.OnCalendar setting which starts the crawling every 30 minutes.


[root@nixcloud:/var/lib/crawler]# ls -lathr
total 5.6M
drwxr-xr-x 21 root    root    4.0K Oct 24 18:09 ..
drwx------  3 crawler crawler 4.0K Oct 24 23:24 .cache
drwx------  3 crawler crawler 4.0K Oct 24 23:24 .dbus
drwxr-xr-x  2 crawler crawler 4.0K Oct 24 23:24 Desktop
drwx------  4 crawler crawler 4.0K Oct 24 23:24 .mozilla
drwxr-xr-x  3 crawler crawler 4.0K Oct 25 13:37
-rw-r--r--  1 crawler crawler  490 Oct 25 13:37 collect_data-environment.nix
-rwxr-xr-x  1 crawler crawler 1.8K Oct 25 18:15
drwx------  8 crawler crawler 4.0K Oct 25 18:15 .
drwxr-xr-x  8 crawler crawler 4.0K Oct 25 18:15 .git
-rwxr-xr-x  1 crawler crawler 5.6M Oct 25 18:16 inject_into_fluxdb
-rw-r--r--  1 crawler crawler 5.1K Oct 27 12:30 data.json


this is an example of the data.json which is generated by selenium and with jq, a very nice tool to process json in a shell, one can experiment with the values.

  "Date": "2016-10-27T11:00:55.123Z",
  "Data": [
      "Id": 338,
      "Votes": 2252,
      "Verein": "Ziegenprojekt am Jusi und Florian",
      "Title": "Schwäbischer Albverein Kohlberg/Kappishäuseren"
      "Id": 215,
      "Votes": 2220,
      "Verein": "„Karl, der Käfer, wurde nicht gefragt …“ – ein Baumprojekt",
      "Title": "Waldkindergarten Schurwaldspatzen e.V."
      "Id": 194,
      "Votes": 34,
      "Verein": "Plankton: Das wilde Treiben im Baggersee!",
      "Title": "Tübinger Mikroskopische Gesellschaft e.V. (Tümpelgruppe)"

jq usage example

cat data.json | jq '.Data[0]'
  "Id": 338,
  "Votes": 2252,
  "Verein": "Ziegenprojekt am Jusi und Florian",
  "Title": "Schwäbischer Albverein Kohlberg/Kappishäuseren"

grafana setup

first you need to add an influxdb data source:

based on that you need to configure the graph to use the influxdb source:


hope you enjoyed reading this and if you have further questions, drop an email to:



by qknight at October 26, 2016 04:35 PM