NixOS Planet

February 19, 2017

Munich NixOS Meetup

guix/guixSD: Talk & Discussion

photoMunich NixOS Meetup

John Darrington will give a talk about guix and guixSD, a package manager and a Linux distribution which are based on similar concepts as nix/NixOS.

The talk will be in English.

Augsburg - Germany

Wednesday, March 22 at 7:00 PM

7

https://www.meetup.com/Munich-NixOS-Meetup/events/237831744/

February 19, 2017 07:51 PM

December 31, 2016

Domen Kozar

Reflecting on 2016

Haven't blogged in 2016, but a lot has happened.

A quick summary of highlighted events:

2016 was a functional programming year as I've planned by end of 2015.

I greatly miss Python community and in that spirit, I've attended EuroPython 2016 and helped organize DragonSprint in Ljubljana. I don't think there's a place for me in OOP anymore, but I'll surely attend community events as nostalgia will kick in.

2017 seems extremely exciting, plans will unveil as I go, starting with some exciting news in January for Nix community.

by Domen Kožar at December 31, 2016 06:00 PM

December 27, 2016

Joachim Schiele

33C3-nixos

nixos assembly

if you want to join our nixos sprint, you can find our assembly here:

hope to see you!

qknight

by qknight at December 27, 2016 11:35 AM

December 20, 2016

Rok Garbas

Reproducible builds summit in Berlin

Last week a Reproducible Builds Summit was held in Berlin and since it was almost at my doorstep I had to take part. Along with Eelco Dolstra, I represented the NixOS voice in the debates that were happening at the summit.

Group picture of participant of Reproducible Builds summit in Berlin

When is a build reproducible?

There were quite few discussion sessions trying to define, when some piece of software is reproducible. While it is important to have a definition - in words, I was hoping that there would be a simple tool that could tell me this.

Such tool might never exist. What I realized during the summit is that reproducibility is not something which is true or false, but something that we is true until somebody disproves it. Reproducibility is a goal which we are always working towards, just like security.

As with zero vulnerability days, there should probably be zero reproducibility days.

Making sure that our software is reproducibility would follow similar practices we have already for security: 3rd party auditing, CVE-like database of reproducibility bugs, ...

Knowing if something is reproducible is not a simple yes/no question, but it is a process you need to follow. Yaay I am not too old to learn something :)

Different kinds of reproducibility

For the purpose of this blog post I would like to point out that there are - at least - two kinds of reproducibility.

When we talked about reproducibility at the summit we were of course referring to a bit-by-bit reproducibility or as I will continue to call it during this blog post binary reproducibility.

And then there is reproducibility which I like to call build reproducibility which only ensures the reproducibility of build environments (eg. versions of tools in build environments are the same).

Purpose of Reproducible Builds effort is of course to be binary reproducible, but to be able to build something bit by bit identical you need to use the same version of tools, which makes build reproducibility a pre-step of binary reproducibility.

Build reproducibility is a prerequirementof binary reproducibility.

Why this distinction matter I will explain in a bit, but for now just acknowledge this naming.

We are all biased

A leader in Reproducible Builds efforts is Debian community. You can see that Debian community is working hard on this and many Debian developers were present at the summit.

Getting the involvement outside of the Debian community is high on the list, since everybody realizes that only with common efforts we will be able to achive reproducibility nirvana.

But regardless of all the good intentions, I noticed two biases that I would like to point.

  • Many of us look down on language specific package managers (eg. pip, cabal, ...) as being less worthy usually talking about them as: "Who in their sane mind would use latest version of packages?". It sounds like a usual developer vs. sysadmin conversation. I hope at next summit we could also have representatives from some of the language specific package managers joined the discussions.
  • I got the impression that the sole reason of reproducible builds is that you would be more secure. That implies that everybody cares about security. Which would be great, but in a world with tight deadlines and startups security is usually the first thing that gets crossed out of the list. We need to make a more compelling reason then just security. I am aware that security is important to many, but we must also understand that it is not top priority for everybody.

Probably I missed some, because of my own biases, but the only way to overcome such thing is to try to expose each other biases.

Why would you care about reproducible builds?

My personal quest for this summit was to find better ways to market reproducible builds. I found that many who tried or want to introduce reproducible builds at work, fail because of few of reasons:

  • you need to opt in and switch from tools that you are used to.
  • many tools you are already using were not built with reproducible issues in mind.
  • reproducibility many times sounds like: all or nothing.
  • the prevalent (marketed) benefit of reproducible builds is better security. not everybody requires that level of security.
  • high cost (usually in developers hours) is usually required.

Reproducibility as productivity tool

What if we turn the marketing of reproducible builds around. What if the main (marketed) reason for the reproducible builds would be to improve developer productivity?

In previous paragraph I already explained that reproducibility is not a simple yes/no questions, but it is a process which one must follow. Also the path to reproducible builds is not a simple turn on/off switch. There are many steps on the way to binary reproducibility that already improve current development practices while getting us closer to binary reproducibility.

One example of this could be: reproducible development environments. If we have tools that could recreate any environment, couldn't we also use the for development purposes?

And we don't have to stop here. Isn't the most expensive part of fixing bugs, reproducing them? Couldn't build reproducibility also help us with reproducing certain type of bugs?

Maybe by now it became I little clearer, that there are many benefits along the way to binary reproducibility, that might be more convincing for some companies.

Cross platform build tool

Another discussion I watched from across the room was, BuildInfo specification. As part of reproducibility building Debian packages, also an BuildInfo file is (will be) produced which has all needed instructions, sources and final checksums, that allows somebody else to verify (reproduce) the resulting binary.

I was not alone thinking that this verification process should/could be distribution agnostic. Even a group was formed to discuss this, but sadly I was busy in other discussions to take part in this.

But then I realized that BuildInfo effort is actually changing a binary distribution like Debian into a source -> binary like distribution. Why produce BuildInfo file after the build process and why not start with it and only record the checksums of binaries after the build is done.

Is there such build tool that works across distributions would allow us to have BuildInfo specification (except the checksums) before the build process? Of course there is: Nix.

What many do not know about Nix is that Nix is first and foremost a build tool. It only happens that there is a database of packages already described how to be built and a side-effect we get is that Nix can also be a package manager. But initially it is a build tool. Nix can build .deb or .rpm packages or any other format you want.

What I would like to see is that whoever is looking into this direction that gives Nix a try and at least learn from it, because Nix and NixOS community is doing build reproducibility already for the last 10 years.

Conclusion

Few things I want you to take from this blog post are:

  • Go to the next Reproducible Builds summit. It was great. I hope to be there too.
  • Reproducibility is a process and not a state.
  • There are many useful steps before you reach reproducible nirvana. It might make sense to market those as well.
  • Nix is a cross distribution build tool. Use it. I know I will :)

by Rok Garbas at December 20, 2016 11:00 PM

December 14, 2016

Joachim Schiele

rust-cargo

motivation

our latest encounter with rust was cargo which we replaced in a way that it is not used for deployment in nixos anymore.

more can be found at: https://github.com/nixcloud/nixcrates

tar-example

here is a tiny example using the tar-crate with the source from https://github.com/alexcrichton/tar-rs#writing-an-archive

tar-example source code

tar-example = stdenv.mkDerivation rec {
  name = "tar-example";
  src = ./example/src3;
  buildInputs = with allCrates; [ tar filetime libc xattr ]; 
  buildPhase = ''
    ${symlinkCalc buildInputs}

    # this creates the files needed for the test
    echo "hu" > file1.txt
    echo "bar" > file2.txt
    echo "batz" > file3.txt 

    ${rustcNightly}/bin/rustc $src/main.rs --crate-type "bin" --emit=dep-info,link --crate-name main -L dependency=mylibs --extern tar=${allCrates.tar}/libtar.rlib
    ./main
    if [ -f foo.tar ]; then
      echo -e "---------\nSUCCESS: tar-example was executed successfully!   \n--------"
    else
      echo "FAIL: not working!"
    fi
  '';
  installPhase=''
    mkdir $out
  '';
};

after executing it

nix-build  -A tar-example
these derivations will be built:
  /nix/store/wmfbc5x59pnhmvh4fdql7gnbymsli33w-tar.drv
  /nix/store/sr1cgvkapz044wgx1rw6261hl7d9682y-tar-example.drv
building path(s) ‘/nix/store/hgravz7r38n3ic8msn94qy761zzi6jyw-tarunpacking sources
tar-0.4.9/.gitignore
tar-0.4.9/.travis.yml
tar-0.4.9/Cargo.toml
tar-0.4.9/LICENSE-APACHE
tar-0.4.9/LICENSE-MIT
tar-0.4.9/README.md
tar-0.4.9/appveyor.yml
tar-0.4.9/examples/extract_file.rs
tar-0.4.9/examples/list.rs
tar-0.4.9/examples/raw_list.rs
tar-0.4.9/examples/write.rs
tar-0.4.9/src/archive.rs
tar-0.4.9/src/builder.rs
tar-0.4.9/src/entry.rs
tar-0.4.9/src/entry_type.rs
tar-0.4.9/src/error.rs
tar-0.4.9/src/header.rs
tar-0.4.9/src/lib.rs
tar-0.4.9/src/pax.rs
tar-0.4.9/tests/all.rs
tar-0.4.9/tests/archives/directory.tar
tar-0.4.9/tests/archives/duplicate_dirs.tar
tar-0.4.9/tests/archives/empty_filename.tar
tar-0.4.9/tests/archives/file_times.tar
tar-0.4.9/tests/archives/link.tar
tar-0.4.9/tests/archives/pax.tar
tar-0.4.9/tests/archives/reading_files.tar
tar-0.4.9/tests/archives/simple.tar
tar-0.4.9/tests/archives/spaces.tar
tar-0.4.9/tests/archives/sparse.tar
tar-0.4.9/tests/archives/xattrs.tar
tar-0.4.9/tests/entry.rs
tar-0.4.9/tests/header/mod.rs
patching sources
configuring
no configure script, doing nothing
building
tar -  --extern libc=/nix/store/nc2jvn8rzbrbbqdwfwc7clzl99za9w2r-libc/liblibc.rlib --extern filetime=/nix/store/a5rr0mvyqnvq3mawhacwb49i101lyp4v-filetime/libfiletime.rlib
namefix tar
name tar
About to use rustc to compile some lib - tar
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/hgravz7r38n3ic8msn94qy761zzi6jyw-tar
patching script interpreter paths in /nix/store/hgravz7r38n3ic8msn94qy761zzi6jyw-tar
building path(s) ‘/nix/store/m0kslv072cphsk11n4696lzncc6rprc1-tar-exampleunpacking sources
unpacking source archive /nix/store/3q2yq22lh5shr0w4fxhcw8h1s61p6q9y-src3
source root is src3
patching sources
configuring
no configure script, doing nothing
building
warning: unused import: `std::io::prelude::*;`, #[warn(unused_imports)] on by default
 --> /nix/store/3q2yq22lh5shr0w4fxhcw8h1s61p6q9y-src3/main.rs:3:5
  |
3 | use std::io::prelude::*;
  |     ^^^^^^^^^^^^^^^^^^^^

---------
SUCCESS: tar-example was executed successfully!   
--------
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/m0kslv072cphsk11n4696lzncc6rprc1-tar-example
patching script interpreter paths in /nix/store/m0kslv072cphsk11n4696lzncc6rprc1-tar-example
/nix/store/m0kslv072cphsk11n4696lzncc6rprc1-tar-example

summary

the project is still in its early stages but evolves into a reimplementation of cargo. however, it is pretty easy to be used and works for many crates already.

by qknight at December 14, 2016 07:35 PM

November 15, 2016

Joachim Schiele

nixos-augsburg-sprint-2016-11

nixos-sprint

paul and me visisted the augsburger openlab again!

projects

profpatsch

  • package draw.io
    • ✔ build with ant
  • ✔ Initialize package tests

uwap

  • Quassel + qt4 doesn't support postgresql as database backend
    • ✔ Add an option to the quassel service to allow the qt5 version
  • "nixify" postfix configuration

christine

paul, michael & qknight

  • ✔ started nextcloud packaging
  • ✔ leaps: packaged with tests: https://github.com/NixOS/nixpkgs/commit/47d81ed3473f33cfb48f2be079f50cdfac60f1e7
  • ✔ made https://github.com/QuiteRSS/quiterss work on nixos, https://github.com/NixOS/nixpkgs/pull/20245
  • fixed nixcloud.io email system so that qknight can use thunderbird with STARTTLS and submission

    submissionOptions = {
      "smtpd_tls_security_level" = "encrypt";
      "smtpd_sasl_auth_enable" = "yes";
      "smtpd_client_restrictions" = "permit_sasl_authenticated,reject";
      "smtpd_sasl_type" = "dovecot";
      "smtpd_sasl_path" = "private/auth";
    };
  • LXC: Unprivileged container with NixOS as guest and as host:
    • ✔ LXC container ist started as root that spawnes the LXC as user 100000 which is unprivileged on the host
    • ✔ shared read only store with the host
    • ✔ container can be build and updated on the host with nix-env

summary

this sprint was awesome. we got so many things to work!

by qknight at November 15, 2016 12:10 PM

November 07, 2016

Anders Papitto

Scripting pulseaudio, bluetooth, jack

Posted on November 7, 2016

I’ve just leveled up my audio configuration, and there’s precious little information out there on how to script against pulseaudio and bluetooth and jack, so I’ll document it a bit.

Future versions of all snippets will live in my nixos configuration repo.

Here are some things that I have working - I can run jack alongside pulseaudio. youtube videos can play at the same time as audio software. - I can play audio through bluetooth headphones. I can even do this simultaneously with running jack - though, only pulseaudio outputs can be sent to the bluetooth headphones.

And that’s basically it. But it’s tricky to set up for the first time. So, let’s look at the implementation.

system configuration

First of all, I’ve scrapped the default /etc/pulse/default.pa, and written my own. It’s largely the same - I went through the default and copied in each line, except that I skipped all the modules that have to do with restoring streams and devices, etc. It’s quite annoying when pulseaudio has persistent state and tries to do “smart” things - I would rather my scripts be in full control. Note the presence of some jackdbus and bluetooth modules.

In that file, I also have some pam limits configured, as well as some kernel modules and /etc/jackdrc. Those precede my most recent pass of configuration, so I’m not sure if they’re actually necessary.

Assuming that you can get jack and/or bluetooth to work the first time, the interesting bits are the scripts that make them not a pain to deal with. I have two main scripts - switch-to-jack, switch-to-bluetooth. I also have a mostly-unused switch-to-stereo for completeness.

switch-to-jack looks like this

#!/bin/bash
set -x
until pacmd list-sinks | egrep -q 'jack_out'
do
    jack_control start
done
pactl set-sink-volume jack_out 50%
pacmd set-default-sink jack_out
for index in $(pacmd list-sink-inputs | grep index | awk '{ print $2 }')
do
    pacmd move-sink-input $index jack_out
done

It starts jack with jack_control. This will cause jack to take exclusive control of the sound card. pulseaudio will notice this and add a sink and source for jack - however because all the ‘smart’ modules were removed, it won’t redirect any active streams to jack. They will just freeze, because they no longer have access to the sound card.

Then, I set the volume to something low enough to be safe - disabling the ‘smarts’ means that the jack device always gets added at 100% volume, which generally is too loud. After doing that, it’s safe to tell future audio to default to playing over jack, and then to move all existing streams there.

switch-to-bluetooth is similar

#!/bin/bash

set -x

DEVICE=$(bluetoothctl <<< devices | egrep '^Device' | awk '{ print $2 }')
until bluetoothctl <<< show | grep -q 'Powered: yes'
do
    bluetoothctl <<< 'power on'
done
until pacmd list-sinks | egrep -q 'name:.*bluez_sink'
do
    bluetoothctl <<< "connect $DEVICE"
done

TARGET_CARD=$(pacmd list-cards | grep 'name:' | egrep -o 'bluez.*[^>]')
TARGET_SINK=$(pacmd list-sinks | grep 'name:' | egrep -o 'bluez.*[^>]')

until pacmd list-cards | egrep -q 'active profile: <a2dp_sink>'
do
    pacmd set-card-profile $TARGET_CARD a2dp_sink
done


pactl set-sink-volume $TARGET_SINK 30%
pacmd set-default-sink $TARGET_SINK
for index in $(pacmd list-sink-inputs | grep index | awk '{ print $2 }')
do
    pacmd move-sink-input $index $TARGET_SINK
done

A couple differences are

  • we make sure bluetooth is on
  • we have to connect to the specific device. If I had more than one set of bluetooth headphones this logic might be more complicated.
  • I have to always tell pulseaudio to use a2dp instead of hsp/hfp, because again I disabled the ‘remembering’ modules in pulseaudio.

Also note that this is the ‘steady-state’ implementation. The first time you want to connect a particular bluetooth device, you have to go through a little dance that looks something like this:

$ bluetoothctl
[NEW] Controller 60:57:18:9B:AB:71 BlueZ 5.40 [default]

[bluetooth]# agent on
[bluetooth]# Agent registered

[bluetooth]# discoverable on
[bluetooth]# Changing discoverable on succeeded
[CHG] Controller 60:57:18:9B:AB:71 Discoverable: yes

[bluetooth]# scan on
[bluetooth]# Discovery started
[CHG] Controller 60:57:18:9B:AB:71 Discovering: yes
[NEW] Device E8:07:BF:00:14:14 Mixcder ShareMe 7

[bluetooth]# [CHG] Device E8:07:BF:00:14:14 RSSI: -52

[bluetooth]# scan off
[bluetooth]# [CHG] Device E8:07:BF:00:14:14 RSSI is nil
Discovery stopped
[CHG] Controller 60:57:18:9B:AB:71 Discovering: no

[bluetooth]# devices
Device E8:07:BF:00:14:14 Mixcder ShareMe 7
[bluetooth]# pair E8:07:BF:00:14:14
Attempting to pair with E8:07:BF:00:14:14
[CHG] Device E8:07:BF:00:14:14 Connected: yes

[Mixcder ShareMe 7]# [CHG] Device E8:07:BF:00:14:14 Modalias: bluetooth:v0094p5081d0101
[CHG] Device E8:07:BF:00:14:14 UUIDs: 00001108-0000-1000-8000-00805f9b34fb
[CHG] Device E8:07:BF:00:14:14 UUIDs: 0000110b-0000-1000-8000-00805f9b34fb
[CHG] Device E8:07:BF:00:14:14 UUIDs: 0000110c-0000-1000-8000-00805f9b34fb
[CHG] Device E8:07:BF:00:14:14 UUIDs: 0000110e-0000-1000-8000-00805f9b34fb
[CHG] Device E8:07:BF:00:14:14 UUIDs: 0000111e-0000-1000-8000-00805f9b34fb
[CHG] Device E8:07:BF:00:14:14 UUIDs: 00001200-0000-1000-8000-00805f9b34fb
[CHG] Device E8:07:BF:00:14:14 Paired: yes
Pairing successful

[CHG] Device E8:07:BF:00:14:14 Connected: no

[bluetooth]# connect E8:07:BF:00:14:14
Attempting to connect to E8:07:BF:00:14:14
[CHG] Device E8:07:BF:00:14:14 Connected: yes
Connection successful

[Mixcder ShareMe 7]# exit
[Mixcder ShareMe 7]# Agent unregistered
[DEL] Controller 60:57:18:9B:AB:71 BlueZ 5.40 [default]

Abandoning ship

Laptop suspend causes bluetooth to disconnect. With the pulseaudio ‘smarts’ disabled, as far as I’m aware the bluetooth-connected streams will just die. To work around this, I switch everything to jack right before shutting down with a systemd service.

Other tips

Given the command line tooling, it’s not really necessary to use any guis. However, I found it convenient to use pavucontrol while setting all this up just to keep an eye on what was active and which streams were going where.

Note that I set volume with pactl, and most other things use pacmd. I’m not sure what the exact differences between these tools is, but pacmd doesn’t support setting a percent-based volume.

November 07, 2016 12:00 AM

November 06, 2016

Joachim Schiele

pulseaudio tcp streaming

motivation

this is a simple setup for streaming pulseaudio streams over the network.

server

https://github.com/openlab-aux/vuizvui/blob/master/machines/labnet/labtops.nix#L26

hardware.pulseaudio = {
  enable = true;
  tcp.enable = true;
  tcp.anonymousClients.allowedIpRanges = [ "172.16.0.0/16" ];
  zeroconf.publish.enable = true;
};

client

pulseaudio.zeroconf.discovery.enable=true;

to use the new setup simply play some music and in pavucontrol you can select a different output device for the listed stream.

by qknight at November 06, 2016 12:35 AM

November 05, 2016

Joachim Schiele

apu

motivation

a few months back i've replaced the odroid XU4 with this APU 2c4 board.

installing nixos

first have a look into the apu2 manual.

since there is no VGA/DVI output but only a RS232 serial interface we need to use that:

  1. serial cable

    out of simplicity i soldered one myself, the pins are:

    pin 2 to pin 3
    pin 3 to pin 2
    pin 5 to pin 5 (GND)

    i've been using this with a USB 2 RS232 converter

    # lsusb
    Bus 003 Device 003: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port
  2. connecting via serial console:

    picocom /dev/ttyUSB0 -b 115200
  3. nixos boot cd

    download the nixos-minimal-16.03.714.69420c5-x86_64-linux.iso and use unetbootin to deploy it to an USB stick. afterwards mount the first partition of the USB-stick and append this to the syslinux.cfg file's kernel command line:

    console=ttyS0,115200n8

    info: using the serial console you can see the GRUB output, see the kernel's output after boot and finally get a shell.

  4. booting from the USB stick

    the apu 2c4 features corebios and the process is straight forward, just hit F10 and select the USB stick

  5. nixos installation

    basically follow the nixos manual

    info: but don't forget to include this line in configuration.nix:

    boot.kernelParams = [ "console=ttyS0,115200n8" ];

/etc/nixos/configuration.nix

# Edit this configuration file to define what should be installed on
# your system.  Help is available in the configuration.nix(5) man page
# and in the NixOS manual (accessible by running ‘nixos-help’).

{ config, pkgs, ... }:

let
  pw = import ./passwords.nix;
in
# setfacl -R -m u:joachim:rwx /backup

{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];

  # Use the GRUB 2 boot loader.
  boot.loader.grub.enable = true;
  boot.loader.grub.version = 2;
  # Define on which hard drive you want to install Grub.
  boot.loader.grub.device = "/dev/sda";

  boot.kernelParams = [ "console=ttyS0,115200n8" ];

  networking = {
    hostName = "apu-nixi"; # Define your hostname.
    bridges.br0.interfaces = [ "enp1s0" "wlp4s0" ];
    firewall = {
      enable = true;
      allowPing = true;
      allowedTCPPorts = [ 22 ];
      #allowedUDPPorts = [ 5353 ];
    };

  };

  # networking.wireless.enable = true;  # Enables wireless support via wpa_supplicant.

  #Select internationalisation properties.
  i18n = {
    consoleFont = "Lat2-Terminus16";
    consoleKeyMap = "us";
    defaultLocale = "en_US.UTF-8";
  };

  security.sudo.enable = true;

  programs.zsh.enable = true;
  users.defaultUserShell = "/run/current-system/sw/bin/zsh";

  services = {
    nscd.enable = true;
    ntp.enable = true;
    klogd.enable = true;
    nixosManual.enable = false; # slows down nixos-rebuilds, also requires nixpkgs.config.allowUnfree here..?
    xserver.enable = false;

    cron = {
      enable = true;
      mailto = "js@lastlog.de";
      systemCronJobs = [
        "0 0,8,16 * * * joachim cd /backup/; ./run_backup.sh"
        #*     *     *   *    *            command to be executed
        #-     -     -   -    -
        #|     |     |   |    |
        #|     |     |   |    +----- day of week (0 - 6) (Sunday=0)
        #|     |     |   +------- month (1 - 12)
        #|     |     +--------- day of month (1 - 31) 
        #|     +----------- hour (0 - 23)
        #+------------- min (0 - 59)
      ]; 
    };  
  };  

  # Set your time zone.
  # time.timeZone = "Europe/Amsterdam";

  # List packages installed in system profile. To search by name, run:
  environment.systemPackages = with pkgs; [
    borgbackup
    bridge-utils
    pciutils
    openssl
    ethtool
    #borg
    iotop
    cryptsetup
    parted
    pv
    tmux
    htop
    git
    dfc
    vim
    wget
    linuxPackages.cpupower
    powertop
    usbutils
    ethtool
    smartmontools
    nix-repl
    manpages
    ntfs3g
    lsof
    iptraf
    mc
    hdparm
    sdparm
    file
    dcfldd
    dhex
    inotifyTools
    nmap
    tcpdump
    silver-searcher
    #emacs
  ];

  time.timeZone = "Europe/Berlin";

  # Enable the OpenSSH daemon.
  services.openssh = {
    enable = true;
    permitRootLogin = "without-password";
  };

  systemd.services.hostapd.after = [ "sys-subsystem-net-devices-wlp4s0.device" ];

  services.hostapd = {
    enable = true;
    wpaPassphrase = pw.wpaPassphrase;
    interface = "wlp4s0";
    ssid="flux";
  };


  # Define a user account. Don't forget to set a password with ‘passwd’.
  users.extraUsers.joachim = {
    isNormalUser = true;
    uid = 1000;
  };

  # The NixOS release to be compatible with for stateful data such as databases.
  system.stateVersion = "16.09";
}

WD passport USB 3.0 bug

with a WD passport USB 3.0 disk i can't boot the system since i hit this bug.

SeaBIOS (version ?-20160311_005214-0c3a223c2ee6)
XHCI init on dev 00:10.0: regs @ 0xfea22000, 4 ports, 32 slots, 32 byte contexts
XHCI    extcap 0x1 @ fea22500
XHCI    protocol USB  3.00, 2 ports (offset 1), def 0
XHCI    protocol USB  2.00, 2 ports (offset 3), def 10
XHCI    extcap 0xa @ fea22540
Found 2 serial ports
ATA controller 1 at 4010/4020/0 (irq 0 dev 88)
EHCI init on dev 00:13.0 (regs=0xfea25420)
ATA controller 2 at 4018/4024/0 (irq 0 dev 88)
Searching bootorder for: /pci@i0cf8/*@14,7
Searching bootorder for: /rom@img/memtest
Searching bootorder for: /rom@img/setup
ata0-0: KINGSTON SMS200S360G ATA-8 Hard-Disk (57241 MiBytes)
Searching bootorder for: /pci@i0cf8/*@11/drive@0/disk@0
XHCI port #3: 0x002202a0, powered, pls 5, speed 0 [ - ]
XHCI port #1: 0x00021203, powered, enabled, pls 0, speed 4 [Super]
Searching bootorder for: /pci@i0cf8/usb@10/storage@1/*@0/*@0,0
Searching bootorder for: /pci@i0cf8/usb@10/usb-*@1
USB MSC vendor='WD' product='My Passport 0827' rev='1012' type=0 removable=0
call16 with invalid stack
PCEngines apu2
coreboot build 20160311
4080 MB ECC DRAM

documentation

how to recover a bricked BIOS (after flashing)? on APU1 it was SPI, and there's a header, so like wires + a ch341a should do it.

wireless

the APU i'm using also has a Mini-PCIe wireless card built and you can choose from these two cards:

the access point works nicely with my android devices as well as my linux laptops.

buy the APU

if you want to buy an APU, buy the APU bundle.

summary

the APU is runnig NixOS and is very stable and fast while using little energy. would use/buy again!

by qknight at November 05, 2016 02:35 PM

November 04, 2016

Edward Tjörnhammar

Plant Based UML Wiki

Turn Gitit into a UML renderer

November 04, 2016 12:00 AM

October 26, 2016

Sander van der Burg

Push and pull deployment of Nix packages

In earlier blog posts, I have covered various general concepts of the Nix package manager. For example, I have written three blog posts explaining Nix from a system administrator, programming language, and a sales perspective.

Furthermore, I have written a number of blog posts demonstrating how Nix can be used, such as managing a set of private packages, packaging binary-only software, and Nix's declarative nature -- the Nix package manager (as well as other tools in the Nix project), are driven by declarative specifications describing the structure of the system to deploy (e.g. the packages and its dependencies). From such a specification, Nix carries out all required deployment activities, including building the packages from its sources, distributing packages from the producer to consumer site, installation and uninstallation.

In addition to the execution of these activities, Nix provides a number of powerful non-functional properties, such as strong guarantees that dependency specifications are complete, that package builds can be reproduced, and that upgrades are non-destructive, atomic, and can always be rolled back.

Although many of these blog posts cover the activities that the Nix package manager typically carries out (e.g. the three explanation recipes cover building and the blog post explaining the user environment concepts discusses installing, uninstalling and garbage collection), I have not elaborated much about the distribution mechanisms.

A while ago, I noticed that people were using one of my earlier blog post as a reference. Despite being able to set up a collection of private Nix expressions, there were still some open questions left, such as fully setting up a private package repository, including the distribution of package builds.

In this blog post, I will explain Nix's distribution concepts and show how they can be applied to private package builds. As with my earlier blog post on managing private packages, these steps should be relatively easy to repeat.

Source and transparent binary deployments


As explained in my earlier blog post on packaging private software, Nix is in principle a source-based package manager. Nix packages are driven by Nix expressions describing how to build packages from source code and all its build-time dependencies, for example:


{ stdenv, fetchurl
, pkgconfig, perl, glib, gpm, slang, zip, unzip, file, gettext
, libX11, libICE, e2fsprogs
}:

stdenv.mkDerivation {
name = "mc-4.8.12";

src = fetchurl {
url = http://www.midnight-commander.org/downloads/mc-4.8.12.tar.bz2;
sha256 = "15lkwcis0labshq9k8c2fqdwv8az2c87qpdqwp5p31s8gb1gqm0h";
};

buildInputs = [ pkgconfig perl glib gpm slang zip unzip file gettext
libX11 libICE e2fsprogs ];

meta = {
description = "File Manager and User Shell for the GNU Project";
homepage = http://www.midnight-commander.org;
license = "GPLv2+";
maintainers = [ stdenv.lib.maintainers.sander ];
};
}

The above expression (mc.nix) describes how to build Midnight Commander from source code and its dependencies, such as pkgconfig, perl and glib. Because the above expression does not specify any build procedure, the Nix builder environment reverts to the standard GNU Autotools build procedure, that typically consists of the following build steps: ./configure; make; make install.

Besides describing how to build a package, we must also compose a package by providing it the right versions or variants of the dependencies that it requires. Composition is typically done in a second expression referring to the former:


{ nixpkgs ? <nixpkgs>
, system ? builtins.currentSystem
}:

let
pkgs = import nixpkgs { inherit system; };

callPackage = pkgs.lib.callPackageWith (pkgs // pkgs.xlibs // self);

self = rec {
mc = callPackage ./mc.nix { };

# Other package imports
};
in
self

In the above expression (default.nix), the mc attribute imports our former expression and provides the build-time dependencies as function parameters. The dependencies originate from the Nixpkgs collection.

To build the Nix expression shown above, it typically suffices to run:


$ nix-build -A mc
/nix/store/jal99995sk6rixym4gfwcagmdiqrwv9a-mc-4.8.12

The result of a Nix build is a Nix store path in which the build result is stored.

As may be noticed, the prefix of the package name (jal99995sk6rixym4gfwcagmdiqrwv9a) is a SHA256 hash code that has been derived from all involved build dependencies, such as the source tarball, build-time dependencies and build scripts. Changing any of these dependencies (such as the version of the Midnight Commander) triggers a rebuild and yields a different hash code. Because hash codes ensure that the Nix store paths to packages will be unique, we can safely store multiple versions and variants of the same packages next to each other.

In addition to executing builds, Nix also takes as many precautions to ensure purity. For example, package builds are carried out in isolated environments in which only the specified dependencies can be found. Moreover, Nix uses all kinds of techniques to make builds more deterministic, such as resetting the timestamps of all files to 1 UNIX time, making build outputs read-only etc.

The combination of unique hash codes and pure builds results in a property called transparent binary deployments -- a package with an identical hash prefix results in a (nearly) bit-identical build regardless on what machine the build has been performed. If we want to deploy a package with a certain hash prefix that already exists on a trustable remote machine, then we can also transfer the package instead of building it again.

Distribution models


The Nix package manager (as well as its sub projects) support two kinds of distribution models -- push and pull deployment.

Push deployment


Push deployment is IMO conceptually the simplest, but at the same time, infrequently used on package management level and not very well-known. The idea of push deployment is that you take an existing package build on your machine (the producer) and transfer it elsewhere, including all its required dependencies.

With Nix this can be easily accomplished with the nix-copy-closure command, for example:


$ nix-copy-closure --to sander@consumermachine.org \
/nix/store/jal99995sk6rixym4gfwcagmdiqrwv9a-mc-4.8.12

The above command serializes the Midnight Commander store path including all its dependencies, transfers them to the provided target machine through SSH, and then de-serializes and imports the store paths into the remote Nix store.

An implication of push deployment is that the producer requires authority over the consumer machine. Moreover, nix-copy-closure can transfer store paths from one machine to another, but does not execute any additional deployment steps, such as the "installation" of packages (in Nix packages become available to end-users by composing a Nix user environment that is in the user's PATH).

Pull deployment


With pull deployment the consumer machine is control instead of the producer machine. As a result, the producer does not require any authority over another machine.

As with push deployment, we can also use the nix-copy-closure command for pull deployment:


$ nix-copy-closure --from sander@producermachine.org \
/nix/store/jal99995sk6rixym4gfwcagmdiqrwv9a-mc-4.8.12

The above command invocation is similar to the previous example, but instead copies a closure of the Midnight Commander from the producer machine.

Aside from nix-copy-closure, Nix offers another pull deployment mechanism that is more powerful and more commonly used, namely: channels. This is what people typically use when installing "end-user" packages with Nix.

The idea of channels is that they are remote HTTP/HTTPS URLs where you can subscribe yourself to. They provide a set of Nix expressions and a binary cache of substitutes:


$ nix-channel --add https://nixos.org/channels/nixos-unstable

The above command adds the NixOS unstable channel to the list of channels. Running the following command:


$ nix-channel --update

Fetches or updates the collection of Nix expressions from the channels allowing us to install any package that it provides. By default, the NIX_PATH environment variable is configured in such a way that they refer to the expressions obtained from channels:


$ echo $NIX_PATH
/nix/var/nix/profiles/per-user/root/channels/nixos:...

With a preconfigured channel, we can install any package we want including prebuilt substitutes, by running:


$ nix-env -i mc

The above command installs the Midnight Commander from the set of Nix expressions from the channel and automatically fetches the substitutes of all involved dependencies. After the installation succeeds, we can start it by running:


$ mc

The push deployment mechanisms of nix-copy-closure and pull deployment mechanisms of the channels can also be combined. When running the following command:


$ nix-copy-closure --to sander@consumermachine.org \
/nix/store/jal99995sk6rixym4gfwcagmdiqrwv9a-mc-4.8.12 \
--use-substitutes

The consumer machine first attempts to pull the substitutes of the dependency closure from the binary caches first, and finally the producer pushes the missing packages to the consumer machine. This approach is particularly useful if the connection between the producer and the consumer is slow, but the connection between the consumer and the binary cache is fast.

Setting up a private binary cache


With the concepts of push and pull deployment explained, you may probably wonder how these mechanisms can be used to your own private set of Nix packages?

Fortunately, with nix-copy-closure no additional work is required as it works on any store path, regardless of how it is produced. However, when it is desired to set up your own private binary cache, some additional work is required.

As channels/binary caches use HTTP as a transport protocol, we need to set up a web server with a document root folder serving static files. In NixOS, we can easily configure an Apache HTTP server instance by adding the following lines to a NixOS configuration: /etc/nixos/configuration.nix:


services.httpd = {
enable = true;
adminAddr = "admin@producer.org";
hostName = "producer";
documentRoot = "/var/www";
};

With the nix-push command we can generate a binary cache for our private package set (that includes the Midnight Commander and all its dependencies):


$ nix-push --dest /var/www $(nix-build default.nix)
/nix/store/jal99995sk6rixym4gfwcagmdiqrwv9a-mc-4.8.12

As may be observed by inspecting the document root folder, we now have a set of compressed NAR files representing the serialized forms of all packages involved and narinfo files capturing a package's metadata:


$ ls /var/www
007mwvk5kskrrgr0xgfnwxmwprd1jdvfxi079maq36mhg9dgwqlv.nar.xz
1c5z374p3z8065hfgyy82k2gi82wq147mrapkpivygcd6yjahcs6.nar.xz
a6gkm5vdylpjnsp9wzcb0c7kz11a3pak.narinfo
02jcvjlg2xha2bibl2ivb3nd0c3gh3nq.narinfo
1cgxv7679hqjvcs7jjjyjrk92n58ldj01yx1bsrvv6q2msh8r2m1.nar.xz
cfx2dvzj062qyj7qya9kcpwi1pk1bgin.narinfo
...

In addition to the binary cache, we also need to make the corresponding Nix expressions available. This can be done by simply creating a tarball out of the private Nix expressions and publishing the corresponding file through the web server:


tar cfvz /var/www/custompkgs.tar.gz custompkgs
custompkgs/
custompkgs/mc.nix
custompkgs/default.nix

On the customer machine, we need to configure the binary cache, by adding the following property to /etc/nix.conf:


binary-caches = http://producermachine.org/

In NixOS, this property can be set by adding the following property to /etc/nixos/configuration.nix:


nix.binaryCaches = [ "http://producermachine.org/" ];
nix.requireSignedBinaryCaches = false;

Additionally, we have to configure the NIX_PATH environment variable to refer to our tarball of Nix expressions:


$ export NIX_PATH=custompkgs=http://producermachine.org/custompkgs.tar.gz:$NIX_PATH

Now, when we run the following command-line instruction:


$ nix-env -f '<custompkgs>' -iA mc
downloading ‘http://producermachine.org/custompkgs.tar.gz’... [0/0 KiB, 0.0 KiB/s]
unpacking ‘http://producermachine.org/custompkgs.tar.gz’...
installing ‘mc-4.8.12’

We can install our custom Midnight Commander package by pulling the package from our own custom HTTP server without explicitly obtaining the set of Nix expressions or building it from source code.

Discussion


In this blog post, I have explained Nix's push and pull deployment concepts and shown how we can use them for a set of private packages, including setting up a private binary cache. The basic idea of binary cache distribution is quite simple: create a tarball of your private Nix expressions, construct a binary cache with nix-push and publish the files with an HTTP server.

In real-life production scenarios, there are typically more aspects you need to take into consideration beyond the details mentioned in this blog post.

For example, to make binary caches safe and trustable, it is also recommended to use HTTPS instead of plain HTTP connections. Moreover, you may want to sign the substitutes with a cryptographic key. The manual page of nix-push provides more details on how to set this up.

Some inconvenient aspects of the binary cache generation approach shown in this blog post (in addition to the fact that we need to set up an HTTP server), is that the approach is static -- whenever, we have a new version of a package built, we need to regenerate the binary cache and the package set.

To alleviate these inconveniences, there is also a utility called nix-serve that spawns a standalone web server generating substitutes on the fly.

Moreover, the newest version of Nix also provides a so-called binary cache Nix store. When Nix performs operations on the Nix store, it basically talks to a module with a standardized interface. When using the binary cache store module (instead of the standalone or remote Nix store plugin), Nix automatically generates NAR files for any package that gets imported into the Nix store, for example after the successful completion a build. Besides an ordinary binary cache store plugin, there is also plugin capable of uploading substitutes directly to an Amazon AWS S3 bucket.

Apart from the Nix package manager, also Nix-related projects use Nix's distribution facilities in some extent. Hydra, the Nix-based continuous integration server, also supports pull deployment as it can dynamically generate channels from jobsets. Users can subscribe to these channels to install the bleeding-edge builds of a project.

NixOps, a tool that deploys networks of NixOS configurations and automatically instantiates VMs in the cloud, as well as Disnix, a tool that deploys service-oriented systems (distributed systems that can be decomposed in a "distributable units", a.k.a. services) both use push deployment -- from a coordinator machine (that has authority over a collection of target machines) packages are distributed.

Concluding remarks


After writing this blog post and some thinking, I have become quite curious to see what a pull deployment variant of Disnix (and maybe NixOps) would look like.

Although Disnix and NixOps suit all my needs at the moment, I can imagine that when we apply the same concepts in large organizations with multiple distributed teams, it can no longer considered to be practical to work with a centralized deployment approach that requires authority over all build artefacts and the entire production environment.

by Sander van der Burg (noreply@blogger.com) at October 26, 2016 09:38 PM

Joachim Schiele

data-mining

motivation

inspired by the shackspace's grafana usage for moisture/temperature monitoring i wanted to use grafana myself. since i'm also active in the fablab neckar-alb and we are having a nice project, called cycle-logistics tübingen, to monitor this seemd to be a good opportunity to apply this toolchain.

we are interested in the voting behaviour:

this blog posting documents all steps needed to rebuild the setup so you can leverage this toolchain for your own projects!

here is a screenshot of how it looks:

setup

below you can find a detailed listing and discussion of the single programs used. the source code can be found on github, except the nixos specific parts which is listed below exclusively.

selenium

selenium is used to visit https://www.swt-umweltpreis.de/profile/, parse the DOM tree and to export the data as data.json.

to execute collect_data.py one needs an python environment with two additional libraries. nix-shell along with collect_data-environment.nix is used to create that environment on the fly.

collect_data.py

#! /usr/bin/env nix-shell
#! nix-shell collect_data-environment.nix --command 'python3 collect_data.py'

from selenium import webdriver
from selenium import selenium
from selenium.webdriver.common.keys import Keys

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait # available since 2.4.0
from selenium.webdriver.support import expected_conditions as EC # available since 2.26.0
from pyvirtualdisplay import Display

import sys
import os

display = Display(visible=0, size=(800, 600))
display.start()

from distutils.version import LooseVersion, StrictVersion
if LooseVersion(webdriver.__version__) < LooseVersion("2.51"):
    sys.exit("error: version of selenium ("
        + str(LooseVersion(webdriver.__version__))
        + ") is too old, needs 2.51 at least")

ff = webdriver.Firefox()

st = "https://www.swt-umweltpreis.de/profile/"
ff.get(st)

v = ff.execute_script("""
var t = document.getElementById('profile').childNodes; 
var ret = []
for (var i = 0; i < t.length; i++) {
  if ('id' in t[i]) {
    if(t[i].id.includes('profil-')) {
      var myID = t[i].id.replace("profil-","");
      var myVotes = t[i].getElementsByClassName('profile-txt-stimmen')[0].innerHTML;
      var myTitle = t[i].getElementsByClassName('archive-untertitel')[0].innerHTML;
      var myVerein = t[i].getElementsByClassName('archive-titel')[0].innerHTML;
      //console.log(myID,myVerein, myTitle, myVotes)
      var r = new Object();
      r.Id = parseInt(myID);
      r.Votes = parseInt(myVotes);
      r.Verein = myVerein; 
      r.Title = myTitle;
      ret.push(r)
    }
  }
}
var date = new Date();
var exp = {};
exp.Date = date;
exp.Data = ret;
var j = JSON.stringify(exp,  null, "\t");
console.log(j);
return j;
""")

print (v)

ff.quit()
display.stop()

collect_data-environment.nix

with import <nixpkgs> { };

let 
  pkgs1 = import (pkgs.fetchFromGitHub {
    owner = "qknight";
    repo = "nixpkgs";
    rev = "a1dd8b2a5b035b758f23584dbf212dfbf3bff67d";
    sha256 = "1zn9znsjg6hw99mshs0yjpcnh9cf2h0y5fw37hj6pfzvvxfrfp9j";
  }) {};

in

pkgs1.python3Packages.buildPythonPackage rec {
  name = "crawler";
  version = "0.0.1";

  buildInputs = [ pkgs.firefox xorg.xorgserver ];

  propagatedBuildInputs = with pkgs1.python3Packages; [
    virtual-display selenium
  ];

}

info: in the above environment, two different versions of nixpkgs are mixed, which is a nix speciality.

virtual-display and selenium from an older nixpkgs base called pkgs1 while firefox is the one coming with the nixos operating system called pkgs.

golang import

the go based importer is very simple and basically follows the example from the influxdb client code base.

if you want to build the inject_into_fluxdb binary you can simply use nix-shell and inside that shell simply type go build. you have to put that binary into the right place, which is /var/lib/crawler/, manually since this was only a prototype.

warning use nixos-rebuild switch with the nixos specific changes from below first so that the nixos system will create the user/group and directory (crawler/crawler and /var/lib/crawler). and when you deploy stuff into that directory, make sure you use chown crawler:crawler . -R in that directory.

inject_into_fluxdb

package main

import (
  "encoding/json"
  "github.com/fatih/structs"
  "github.com/influxdata/influxdb/client/v2"
  "io/ioutil"
  "log"
  "strconv"
  "time"
  "fmt"
//   "os"
)

type rec struct {
  Id     int
  Votes  int
  Verein string
  Title  string
}

type json_message struct {
  Date string
  Data []rec
}

const (
  MyDB     = "square_holes"
  username = "bubba"
  password = "bumblebeetuna"
)

func main() {

  f, err2 := ioutil.ReadFile("data.json")
  if err2 != nil {
    log.Fatalln("Error: ", err2)
    return
  }

  var l json_message
  err2 = json.Unmarshal(f, &l)
  if err2 != nil {
    log.Fatalln("Error: ", err2)
    return
  }

  // Make client
  c, err := client.NewHTTPClient(client.HTTPConfig{
    Addr:     "http://localhost:8086",
    Username: username,
    Password: password,
  })

  if err != nil {
    log.Fatalln("Error: ", err)
  }

  // Create a new point batch
  bp, err := client.NewBatchPoints(client.BatchPointsConfig{
    Database:  MyDB,
    Precision: "s",
  })

  if err != nil {
    log.Fatalln("Error: ", err)
  }
  
  layout := "2006-01-02T15:04:05.000Z"
  t, err3 := time.Parse(layout, l.Date)

  if err3 != nil {
      fmt.Println(err)
      return
  }
  
  for _, r := range l.Data {
    pt, err := client.NewPoint("frei", map[string]string{"Id": strconv.Itoa(r.Id)}, structs.Map(r), t.Local())
    bp.AddPoint(pt)
    if err != nil {
      log.Fatalln("Error: ", err)
    }
  }

  // Write the batch
  c.Write(bp)
}

deps.nix

go2nix in version 1.1.1 has been used to generate the default.nix and deps.nix automatically. this is also the reason for the weird directory naming inside the GIT repo.

warning there are two different implementation of a go to nix dependency converter and both are called go2nix. i was using the one from kamilchm the other never worked for me.

# This file was generated by go2nix.
[
  {
    goPackagePath = "github.com/fatih/structs";
    fetch = {
      type = "git";
      url = "https://github.com/fatih/structs";
      rev = "dc3312cb1a4513a366c4c9e622ad55c32df12ed3";
      sha256 = "0wgm6shjf6pzapqphs576dv7rnajgv580rlp0n08zbg6fxf544cd";
    };
  }
  {
    goPackagePath = "github.com/influxdata/influxdb";
    fetch = {
      type = "git";
      url = "https://github.com/influxdata/influxdb";
      rev = "6fa145943a9723f9660586450f4cdcf72a801816";
      sha256 = "14ggx1als2hz0227xlps8klhn5s478kczqx6i6l66pxidmqz1d61";
    };
  }
]

default.nix

info: go2nix generates a default.nix which is basically a dropin when used in nixpkgs but i wanted to use it with nix-shell so a few lines needed changes. just be awere of that!

{ pkgs ? import <nixpkgs>{} } :
let 
  stdenv = pkgs.stdenv;
  buildGoPackage = pkgs.buildGoPackage;
  fetchgit = pkgs.fetchgit;
  fetchhg = pkgs.fetchhg;
  fetchbzr = pkgs.fetchbzr; 
  fetchsvn = pkgs.fetchsvn;
in

buildGoPackage rec {
  name = "crawler-${version}";
  version = "20161024-${stdenv.lib.strings.substring 0 7 rev}";
  rev = "6159f49025fd5500e5c2cf8ceeca4295e72c1de5";

  goPackagePath = "fooooooooooo";

  src = ./.;

  goDeps = ./deps.nix;

  meta = {
  };
}

nixos

configuration.nix

info: in the configuration.nix excerpt apache, which is called httpd in nixos`, is used as a reverse proxy. you don't have to follow that example but it is a nice setup once one gets it working.

  ...
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
      ./crawler.nix
    ];
  ...
  services.grafana = {
    enable=true;
    port=3012;
    rootUrl="https://nixcloud.io/grafana";
    security = {
      adminPassword="supersecret";
      secretKey="+++evenmoresecret+++";
    };
    users = {
      allowSignUp = false;
      allowOrgCreate = true;
    };
    analytics.reporting.enable = false;

  };

  services.influxdb = {
    enable = true;
  };
  ...
  services.httpd = {
    enable = true;
    enablePHP = true;
    logPerVirtualHost = true;
    adminAddr="js@lastlog.de";
    hostName = "lastlog.de";
    extraModules = [
      { name = "php7"; path = "${pkgs.php}/modules/libphp7.so"; }
      { name = "deflate"; path = "${pkgs.apacheHttpd}/modules/mod_deflate.so"; }
      { name = "proxy_wstunnel"; path = "${pkgs.apacheHttpd}/modules/mod_proxy_wstunnel.so"; }
    ];
 virtualHosts =
    [ 
      # nixcloud.io (https)
      {
        hostName = "nixcloud.io";
        serverAliases = [ "nixcloud.io" "www.nixcloud.io" ];

        documentRoot = "/www/nixcloud.io/";
        enableSSL = true;
        sslServerCert = "/ssl/acme/nixcloud.io/fullchain.pem";
        sslServerKey = "/ssl/acme/nixcloud.io/key.pem";
       extraConfig = ''
          Alias /.well-known/acme-challenge /var/www/challenges/nixcloud.io/.well-known/acme-challenge
          <Directory "/var/www/challenges/nixcloud.io/.well-known/acme-challenge">
            Options -Indexes
            AllowOverride None
            Order allow,deny
            Allow from all
            Require all granted
          </Directory>
          RedirectMatch ^/$ /main/
          #Alias /main /www/nixcloud.io/main/page
          <Directory "/www/nixcloud.io/main/">
            Options -Indexes
            AllowOverride None
            Require all granted
          </Directory>

          SetOutputFilter DEFLATE

          <Directory "/www/nixcloud.io/">
            Options -Indexes
            AllowOverride None
            Order allow,deny
            Allow from all
          </Directory>

          # prevent a forward proxy! 
          ProxyRequests off

          # User-Agent / browser identification is used from the original client
          ProxyVia Off
          ProxyPreserveHost On

          RewriteEngine On
          RewriteRule ^/grafana$ /grafana/ [R]

          <Proxy *>
          Order deny,allow
          Allow from all
          </Proxy>

          ProxyPass /grafana/ http://127.0.0.1:3012/ retry=0
          ProxyPassReverse /grafana/ http://127.0.0.1:3012/
       '';
      }
    ]
   ...

crawler.nix

simply put crawler.nix into /etc/nixos and reference it from configuration.nix using the imports directive.

{ config, pkgs, lib, ... } @ args:

#with lib;

let
  cfg = config.services.crawler;
  stateDir = "/var/lib/crawler/";
in { 
  config = {
    users = {
      users.crawler= {
        #note this is a hack since this is not commited to the nixpkgs
        uid             = 2147483647;
        description     = "crawler server user";
        group           = "crawler";
        home            = stateDir;
        createHome      = true;
      };

      groups.crawler= {
        #note this is a hack since this is not commited to the nixpkgs
        gid = 2147483648;
      };
    };

    systemd.services.crawler= {
      script = ''
          source /etc/profile
          export HOME=${stateDir}
          ${stateDir}/collect_data.py > ${stateDir}/data.json
          cd ${stateDir}
          ${stateDir}/inject_into_fluxdb 
        '';
      serviceConfig = {
        Nice = 19;
        IOSchedulingClass = "idle";
        PrivateTmp = "yes";
        NoNewPrivileges = "yes";
        ReadWriteDirectories = stateDir;
        WorkingDirectory = stateDir;
      };
    };

    systemd.timers.crawler = { 
      description = "crawler service";
      partOf      = [ "crawler.service" ];
      wantedBy    = [ "timers.target" ];
      timerConfig = {
        OnCalendar = "*:0/30";
        Persistent = true;
      };
    };
  };
}

info: note the timerConfig.OnCalendar setting which starts the crawling every 30 minutes.

/var/lib/crawler

[root@nixcloud:/var/lib/crawler]# ls -lathr
total 5.6M
drwxr-xr-x 21 root    root    4.0K Oct 24 18:09 ..
drwx------  3 crawler crawler 4.0K Oct 24 23:24 .cache
drwx------  3 crawler crawler 4.0K Oct 24 23:24 .dbus
drwxr-xr-x  2 crawler crawler 4.0K Oct 24 23:24 Desktop
drwx------  4 crawler crawler 4.0K Oct 24 23:24 .mozilla
drwxr-xr-x  3 crawler crawler 4.0K Oct 25 13:37 github.com
-rw-r--r--  1 crawler crawler  490 Oct 25 13:37 collect_data-environment.nix
-rwxr-xr-x  1 crawler crawler 1.8K Oct 25 18:15 collect_data.py
drwx------  8 crawler crawler 4.0K Oct 25 18:15 .
drwxr-xr-x  8 crawler crawler 4.0K Oct 25 18:15 .git
-rwxr-xr-x  1 crawler crawler 5.6M Oct 25 18:16 inject_into_fluxdb
-rw-r--r--  1 crawler crawler 5.1K Oct 27 12:30 data.json

data.json

this is an example of the data.json which is generated by selenium and with jq, a very nice tool to process json in a shell, one can experiment with the values.

{
  "Date": "2016-10-27T11:00:55.123Z",
  "Data": [
    {
      "Id": 338,
      "Votes": 2252,
      "Verein": "Ziegenprojekt am Jusi und Florian",
      "Title": "Schwäbischer Albverein Kohlberg/Kappishäuseren"
    },
    {
      "Id": 215,
      "Votes": 2220,
      "Verein": "„Karl, der Käfer, wurde nicht gefragt …“ – ein Baumprojekt",
      "Title": "Waldkindergarten Schurwaldspatzen e.V."
    },
    {
      ...
    },
    {
      "Id": 194,
      "Votes": 34,
      "Verein": "Plankton: Das wilde Treiben im Baggersee!",
      "Title": "Tübinger Mikroskopische Gesellschaft e.V. (Tümpelgruppe)"
    }
  ]
}

jq usage example

cat data.json | jq '.Data[0]'
{
  "Id": 338,
  "Votes": 2252,
  "Verein": "Ziegenprojekt am Jusi und Florian",
  "Title": "Schwäbischer Albverein Kohlberg/Kappishäuseren"
}

grafana setup

first you need to add an influxdb data source:

based on that you need to configure the graph to use the influxdb source:

summary

hope you enjoyed reading this and if you have further questions, drop an email to: js@lastlog.de.

thanks,

qknight

by qknight at October 26, 2016 04:35 PM

October 16, 2016

Munich NixOS Meetup

Nix(OS) Introduction & Hackathon

photoMunich NixOS Meetup

Curry Club Augsburg & OpenLab Augsburg present: the second NixOS hackathon  //Augsburg  

We’ll have three days of hacking on all things Nix.

On Saturday there will be an extensive introduction to Nix(OS) for beginners. With NixOS install party!

If you want to present the project you are working on, we are going to have a time of 10–20 minute presentations on Friday (or alternatively Sunday).

Food & drinks are available, whether there will be sponsoring for catering will be clear in a few days.

Please RSVP so we can estimate the numbers.

Augsburg - Germany

Friday, November 4 at 11:00 AM

5

https://www.meetup.com/Munich-NixOS-Meetup/events/234896946/

October 16, 2016 04:32 PM

October 09, 2016

Rok Garbas

Updating your Nix sources

It feels a bit tedious that we (package maintainers of nixpkgs) are still updating version of packages manually. Especially in such a small community, we should be really careful where we use our resources.

In this blog post I will take you on a tour of things I learned in a project I am working on with Release Engineering team at Mozilla, mozilla-releng/services, and how we are continuously updating our nix sources.

Extending nixpkgs

Many times I was asked questions how to work efficiently with upstream nixpkgs and how to work with private nix package set.

For each project I usually create a private set of nix expressions. This could look like:

{ pkgs ? import <nixpkgs> {}
}:

let
  custom_pkgs = {
    inherit pkgs;
    packageA = ...;
    packageB = ...;
  };
in custom_pkgs

Our nix expression set is called custom_pkgs and it is a function which takes nixpkgs as an argument.

Pinning nixpkgs

In the past I already wrote about why pinning nixpkgs matters. To recap: We always want to have a stable development environment. Importing <nixpkgs> would depend on system's nixpkgs which is different from machine to machine. To pin down nixpkgs let us change previous example:

let
  _pkgs = import <nixpkgs> {};
in
{ pkgs ? import (_pkgs.fetchFromGitHub { owner = "NixOS";
                                         repo = "nixpkgs-channels";
                                         rev = "...";
                                         sha256 = "...";
                                       }) {}
}:

let
  custom_pkgs = {
    inherit pkgs;
    packageA = ...;
    packageB = ...;
  };
in custom_pkgs

We still depend on system's <nixpkgs>, but only to provide us with _pkgs.fetchFromGitHub function. We then pin NixOS/nixpkgs-channels repository to specific revision. I chose nixpkgs-channels repository, since that means I will also get binaries and wont have to recompile too often.

Update runner script

mozilla-releng/services's update runner script can be found in nix/update.nix. What this script does is, it checks which package has update attribute and then loops over and executes every update script. A minimal example would look like:

let
  _pkgs = import <nixpkgs> {};
in
{ pkgs ? import (_pkgs.fetchFromGitHub { owner = "NixOS";
                                         repo = "nixpkgs-channels";
                                         rev = "...";
                                         sha256 = "...";
                                       }) {}
}:

let
  packagesWith = name: attrs: ...; # function which searches attrs values
                                   # and checks for name attribute.
  custom_pkgs = import ./default.nix { inherit pkgs; };
  packages = packagesWith "update" attrs;
in pkgs.stdenv.mkDerivation {
  name = "update-releng-services";
  buildCommand = ''
    echo "+--------------------------------------------------------+"
    echo "| Not possible to update repositories using \`nix-build\`. \|"
    echo "|         Please run \`nix-shell update.nix\`.             \|"
    echo "+--------------------------------------------------------+"
    exit 1
  '';
  shellHook = ''
    export HOME=$PWD
    echo "Updating packages ..."
    ${builtins.concatStringsSep "\n\n" (
        map (pkg: ''echo ' - ${(builtins.parseDrvName pkg.name).name}';
                    ${pkg.update};
                  '') packages
    )}
    echo ""
    echo "Packages updated!"
    exit
  '';
}

If above script is ran with nix-build it will raise an error, saying this script can only be ran with nix-shell. Update scripts will need access to internet, and this is the reason we must run it with nix-shell.

When we run nix-shell update.nix we can see that packagesWith function currently does not find any package with update attribute, since we did not define any.

Update script

Lets first create an update script for tracking nixos-unstable branch for NixOS/nixpks-channels repository on github. Implementation of such script can be found here, updateFromGitHub.

Now lets use updateFromGitHub function and add update attribute to nixpkgs attribute in our custom_pkgs.

pkgs = pkgs // {
   update = releng_pkgs.lib.updateFromGitHub {
     owner = "garbas";
     repo = "nixpkgs";
     branch = "python-srcs";
     path = "nix/nixpkgs.json";
   };
 };
 let
   _pkgs = import <nixpkgs> {};
 in
 { pkgs ? import (_pkgs.fetchFromGitHub
                     (_pkgs.lib.importJSON ./nixpkgs.json)) {}
 }:

 let
   updateFromGitHub = ...;
   custom_pkgs = {
     pkgs = pkgs // {
       update = updateFromGitHub {
         owner = "NixOS";
         repo = "nixpkgs-channels";
         branch = "nixos-unstable";
         path = "nixpkgs.json";
       };
     };
     packageA = ...;
     packageB = ...;
   };
 in custom_pkgs

As you can see above our nixpkgs update script stores owner, repo, rev and sha256 to a nixpkgs.json file in a format which we also read and pass to fetchFromGitHub to be used to initialize custom_pkgs.

An example of update script for a python project, which uses pypi2nix, would look like this:

packageA = pkgs.stdenv.mkDerivation {
  ...
  passthru.update = writeScript "update-${name}" ''
    pushd src/packageA
    ${pkgs.pypi2nix}/bin/pypi2nix -v \
     -V 3.5 \
     -E "postgresql libffi openssl" \
     -r requirements.txt \
     -r requirements-dev.txt
    popd
  '';
  ...
}

Automation with Travis CI

Now that we can manually run update script we need to run it on a daily basis. Luckily for us Travis supports this.

  1. Enable Travis for your project.
  2. Ask Travis for cron support.
  3. Create .travis.yml such as:
language: nix
script:
- if [ "$TRAVIS_EVENT_TYPE" == "cron" ]; then
    nix-shell update.nix --pure;
  fi
- nix-build default.nix
before_deploy:
- eval "$(ssh-agent -s)"
- chmod 600 $TRAVIS_BUILD_DIR/deploy_rsa
- ssh-add $TRAVIS_BUILD_DIR/deploy_rsa
after_success:
- if [[ -n `git diff --exit-code` ]]; then
    git config user.name 'travis';
    git config user.email 'you@email';
    git stash
    git checkout -b result-$TRAVIS_BRANCH origin/$TRAVIS_BRANCH
    git pull
    git stash apply
    git add .
    git commit -m "Travis update [skip ci]";
    git push git@github.com:<owner>/<repo>.git HEAD:$TRAVIS_BRANCH;
  fi
  1. Create deploy_rsa using following commands:
% ssh-keygen -t rsa -b 4096 -C 'travis/<owner>/<repo>' -f ./deploy_rsa
% travis encrypt-file deploy_rsa --add
  1. Allow deploy_rsa.pub to push to your repository.

Add content of deploy_rsa.pub to Authorized keys for your github repository and make sure it has Push permission.

  1. Commit everything and watch updates coming in.
% git add deploy_rsa.enc deploy_rsa.pub .travis
% git commit -c "Travis will make us all lazy."
% git push

Conclusion

It does take quite some time to setup all of this, but benefits can be seen within a week once first (auto-)commits are coming in.

It is important to look at your project as a living system where latest version should be automatically picked, something we are used to from traditional packages managers. With Nix it is not possible to depend on future releases.

With above setup we can get best of both worlds. Despite constantly updating to latest versions, we don't update versions if build (including tests) is not passing. This way we have flexibility of traditional package managers and robustness of Nix.

Best of both worlds.

With large number of packages or projects a setup like this will save us a lot of time. Now imagine how much time we could save if we have similar setup for nixpkgs.

Let me know what you think. Nix out.

by Rok Garbas at October 09, 2016 10:00 PM

October 05, 2016

Rok Garbas

SystemD Conference 2016

I realized I have never really sat down and learned systemd. I was mostly exposed to it via NixOS when writing NixOS modules and time came to dig a bit deeper. Systemd man pages are invaluable source of information. Initially they might be a bit overwhelming, but soon you start to appreciate the long explanation.

journald

journalctl became one of the tools I can not really work without. To be worry free when it comes to logging and to not be a awk/grep wizard when it comes to accessing logs is something everybody new to systemd will appreciate.

Some (hopefully) useful commands I wrote down during Lennart's journald presentation:

  • journalctl -r - show logs in reverse order
  • journalctl -b - show logs since last boot
  • journalctl -k - show kernel logs
  • journalctl -p warning - show logs with warning priority
  • journalctl -p error - show logs with error priority
  • journalctl --since=2016-08-01 - show logs since
  • journalctl --until=2016-08-03 - show logs until
  • journalctl --until=today - show logs until midnight today
  • journalctl --since=yesterday - show logs since yesterday midnight
  • journalctl --since=-2week - show logs for last 2 weeks
  • journalctl -u <unit-name> - show logs of certain unit
  • journalctl /dev/sda - show kernel message of device
  • journalctl -o json - show logs in json format

And you can mix more or less or of the above options.

To serve journald events over the networks there is systemd-journal-gatewayd service which is disabled by default.

Every log message can now provide MESSAGE_ID (generated via journalctl --new-id128) to be able to use journal's catalog capabilities. If this gains traction, it could really help to have better documentation with every error that shows up in journal.

Currently we can limit globally how much space logs will take. Option to have this per service would be very nice and might happen in future.

Containers (systend-nspawn)

Containers were the hot topic at systemd conference. Not about their current usage, but more or less looking into the future and how containers might look.

Apart from great talks I was not aware that there are commands to pull / import / export containers from command line

% machinectl pull-tar <url>
% machinectl pull-raw <url>

% machinectl import-tar <name> <file>
% machinectl import-raw <name> <file>

% machinectl export-tar <name> <file>
% machinectl export-raw <name> <file>

Not something revolutionary, but this commands can be useful to quickly share images with coworkers.

Lennart also presented another tool called mkosi which is aiming to build legacy free OS images. You can consider it as a replacement for deboostrap and dnf. I wonder if this is a place were Nix could be used, since we already have a way of creating images of non NixOS distribution.

Embedded

All the negativity that some people have towards systemd, got me convinced that systemd is one big monolith and that it can not be stripped down to the bare minimum. Last and this year show us that systemd on embedded devices is growing and while there still are problems those problems are possible to solve.

Listening to all the talks about embedded systems got me thinking what kind of build systems do they use and could Nix be a possible solution. Knowing the strong points of Nix (the build tool), reproducibility, composability, etc... must be features that would get some developers in embedded world interested. But to unlock this space for Nix, a basic support for ARM should be there, maybe just in a form of binary channel for a small subset of packages.

Misc

One of pain points of my current setup (no desktop manager like kde/gnome/xfce, only i3 window manager) is that I have to manually mount every USB stick. I was not aware that systemd could also handle this.

/dev/sdb1   /mnt/usb     auto     noauto,x-systemd.automount     0 2

Looks like systemd community is really tackling the hard problems. While there is a networkd which manages networking (there was also a talk about it: Lennart Poettering, Tom Gundersen: What you didn't know about networkd) there is also work in progress to have have a utility to manage wireless networks, Marcel Holtmann: New Wireless Daemon for Linux.

I think the last area where Linux feels painful would be Printing. systemd-printing anybody? :)

Configuration Management

Systemd unit files allow us to specify services in a declarative way. With systemd growing in scope we are more and more seeing a need for a better way of managing services. We have seen to talk about configuration management.

We have seen an very nice presentation of a even more awesome tool called Mgmt (James (purpleidea) Shubin: Next Generation Config Mgmt). While still a bit unpolished you can easily see being it superior to many existing configuration tools out there.

Completely different approach to Configuration management took NixOS. And I had an honor to speak about it (NixOS - Reproducible Linux Distribution built around systemd). I tried my best to explain the core principles of how Nix works, but sadly I was short in time to show all the demos. Still I have learned few things when presenting Nix/NixOS:

  • Don't mention functional, ignore it
  • There is no Nix Expression Language, but a JSON-like syntax
  • Not everybody sees the benefits of reproducibility in first 5 seconds, show examples
  • Nix is not for everybody and for every use case

Conclusion

I had a wonderful 3 days, I hope to see all next year. Especially I would like to thank Kinvolk crew who organized the conference. Also we must not forget the amazing CCC VOC team for all the videos.

New NixOS release is here, what are you waiting! :)

by Rok Garbas at October 05, 2016 10:00 PM