As explained in many previous blog posts, most of the quality properties of the
Nix package manager (such as reliable deployment) stem from the fact that all packages are stored in a so-called
Nix store, in which every package resides in its own
isolated folder with a hash prefix that is derived from all build inputs (such as:
/nix/store/gf00m2nz8079di7ihc6fj75v5jbh8p8v-zlib-1.2.11).
This unorthodox naming convention makes it possible to safely store multiple versions and variants of the same package next to each other.
Although isolating packages in the Nix store provides all kinds of benefits, it also has a big drawback -- common components, such as shared libraries, can no longer be found in their "usual locations", such as
/lib.
For packages that are built from source with the Nix package manager this is typically not a problem:
- The Nix expression language computes the Nix store paths for the required packages. By simply referring to the variable that contains the build result, you can obtain the Nix store path of the package, without having to remember them yourself.
- Nix statically binds shared libraries to ELF binaries by modifying the binary's RPATH field. As a result, binaries no longer rely on the presence of their library dependencies in global locations (such as /lib), but use the libraries stored in isolation in the Nix store.
- The GNU linker (the ld command) has been wrapped to transparently add the paths of all the library package to the RPATH field of the ELF binary, whenever a dynamic library is provided.
As a result, you can build most packages from source code by simply executing their standardized build procedures in a Nix builder environment, such as:
./configure --prefix=$out; make; make install.
When it is desired to deploy prebuilt binary packages with Nix then you may probably run into various kinds of challenges:
- ELF executables require the presence of an ELF interpreter in /lib/ld-linux.so.2 (on x86) and /lib/ld-linux-x86-64.so.2 (on x86-64), which is impure and does not exist in NixOS.
- ELF binaries produced by conventional means typically have no RPATH configured. As a result, they expect libraries to be present in global namespaces, such as /lib. Since these directories do not exist in NixOS an executable will typically fail to work.
To make prebuilt binaries work in NixOS, there are basically two solutions -- it is possible to compose so-called
FHS user environments from a set of Nix packages in which shared components can be found in their "usual locations". The drawback is that it requires special privileges and additional work to compose such environments.
The preferred solution is to
patch prebuilt ELF binaries with
patchelf (e.g. appending the library dependencies to the
RPATH of the executable) so that their dependencies are loaded from the Nix store.
I wrote a guide that demonstrates how to do this for a number of relatively simple packages.
Although it is possible to patch prebuilt ELF binaries to make them run work from the Nix store, such a process is typically tedious and time consuming -- you must dissect a package, search for all relevant ELF binaries, figure out which libraries a binary requires, find the corresponding packages that provide them and then update the deployment instructions to patch the ELF binaries.
For small projects, a manual binary patching process is still somewhat manageable, but for a complex project such as the
Android SDK, that provides a large collection of plugins containing a mix of many 32-bit and 64-bit executables, manual patching is quite labourious, in particular when it is desired to keep all plugins up to date -- plugin packages are updated quite frequently forcing the packager to re-examine all binaries over and over again.
To make the Android SDK patching process easier, I wrote a small tool that can mostly automate it. The tool can also be used for other kinds of binary packages.
Automatic searching for library locations
In order to make ELF binaries work, they must be patched in such a way that they use an ELF interpreter from the Nix store and their RPATH fields should contain all paths to the libraries that they require.
We can gather a list of required libraries for an executable, by running:
$ patchelf --print-needed ./zipmix
libm.so.6
libc.so.6
Instead of manually patching the executable with this provided information, we can also create a function that searches for the corresponding libraries in a list of search paths. The tool could take the first path that provides the required libraries.
For example, by setting the following colon-separated seach environment variable:
$ export libs=/nix/store/7y10kn6791h88vmykdrddb178pjid5bv-glibc-2.27/lib:/nix/store/xh42vn6irgl1cwhyzyq1a0jyd9aiwqnf-zlib-1.2.11/lib
The tool can automatically discover that the path:
/nix/store/7y10kn6791h88vmykdrddb178pjid5bv-glibc-2.27/lib provides both
libm.so.6 and
libc.so.6.
We can also run into situations in which we cannot find any valid path to a required library -- in such cases, we can throw an error and notify the user.
It is also possible extend the searching approach to the ELF interpreter. The following command provides the path to the required ELF interpreter:
$ patchelf --print-interpreter ./zipmix
/lib64/ld-linux-x86-64.so.2
We can search in the list of library packages for the ELF interpreter as well so that we no longer have to explicitly specify it.
Dealing with multiple architectures
Another problem with the Android SDK is that plugin packages may provide both x86 and x86-64 binaries. You cannot link libraries compiled for x86 against an x86-64 executable and vice versa. This restriction could introduce a new kind of risk in the automatic patching process.
Fortunately, it is also possible to figure out for what kind of architecture a binary was compiled:
$ readelf -h ./zipmix
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: EXEC (Executable file)
Machine: Advanced Micro Devices X86-64
The above command-line instruction shows that we have a 64-bit binary (
Class: ELF64) compiled for the x86-64 architecture (
Machine: Advanced Micro Devices X86-64)
I have also added a check that ensures that the tool will only add a library path to the RPATH if the architecture of the library is compatible with the binary. As a result, it is not possible to accidentally link a library with an incompatible architecture to a binary.
Patching collections of binaries
Another inconvenience is the fact that Android SDK plugins typically provide more than one binary that needs to be patched. We can also recursively search an entire directory for ELF binaries:
$ autopatchelf ./bin
The above command-line instruction recursively searches for binaries in the
bin/ sub directory and automatically patches them.
Sometimes recursively patching executables in a directory hierarchy could have undesired side effects. For example, the Android SDK also provides emulators having their own set of ELF binaries that need to run in the emulator. Patching these binaries typically breaks the software running in the emulator. We can also disable recursion if this is desired:
$ autopatchelf --no-recurse ./bin
or revert to patching individual executables:
$ autopatchelf ./zipmix
The result
The result of having most aspects automated of a binary patching process results in a substantial reduction in code size for the Nix expressions that need to deploy prebuilt packages.
In
my previous blog post, I have shown two example cases for which I manually derived the
patchelf instructions that I need to run. By using the
autopatchelf tool I can significantly decrease the size of the corresponding Nix expressions.
For example, the following expression deploys
kzipmix:
{stdenv, fetchurl, autopatchelf, glibc}:
stdenv.mkDerivation {
name = "kzipmix-20150319";
src = fetchurl {
url = http://static.jonof.id.au/dl/kenutils/kzipmix-20150319-linux.tar.gz;
sha256 = "0fv3zxhmwc3p34larp2d6rwmf4cxxwi71nif4qm96firawzzsf94";
};
buildInputs = [ autopatchelf ];
libs = stdenv.lib.makeLibraryPath [ glibc ];
installPhase = ''
${if stdenv.system == "i686-linux" then "cd i686"
else if stdenv.system == "x86_64-linux" then "cd x86_64"
else throw "Unsupported system architecture: ${stdenv.system}"}
mkdir -p $out/bin
cp zipmix kzip $out/bin
autopatchelf $out/bin
'';
}
In the expression shown above, it suffices to simply move the executable to
$out/bin and running
autopatchelf.
I have also shown a more complicated example demonstrating how to patch the
Quake 4 demo. I can significantly reduce the amount of code by substituting all the
patchelf instructions by a single
autopatchelf invocation:
{stdenv, fetchurl, glibc, SDL, xlibs}:
stdenv.mkDerivation {
name = "quake4-demo-1.0";
src = fetchurl {
url = ftp://ftp.idsoftware.com/idstuff/quake4/demo/quake4-linux-1.0-demo.x86.run;
sha256 = "0wxw2iw84x92qxjbl2kp5rn52p6k8kr67p4qrimlkl9dna69xrk9";
};
buildInputs = [ autopatchelf ];
libs = stdenv.lib.makeLibraryPath [ glibc SDL xlibs.libX11 xlibs.libXext ];
buildCommand = ''
# Extract files from the installer
cp $src quake4-linux-1.0-demo.x86.run
bash ./quake4-linux-1.0-demo.x86.run --noexec --keep
# Move extracted files into the Nix store
mkdir -p $out/libexec
mv quake4-linux-1.0-demo $out/libexec
cd $out/libexec/quake4-linux-1.0-demo
# Remove obsolete setup files
rm -rf setup.data
# Patch ELF binaries
autopatchelf .
# Remove libgcc_s.so.1 that conflicts with Mesa3D's libGL.so
rm ./bin/Linux/x86/libgcc_s.so.1
# Create wrappers for the executables and ensure that they are executable
for i in q4ded quake4
do
mkdir -p $out/bin
cat > $out/bin/$i <<EOF
#! ${stdenv.shell} -e
cd $out/libexec/quake4-linux-1.0-demo
./bin/Linux/x86/$i.x86 "\$@"
EOF
chmod +x $out/libexec/quake4-linux-1.0-demo/bin/Linux/x86/$i.x86
chmod +x $out/bin/$i
done
'';
}
For the Android SDK, there is even a more substantial win in code size reductions. The following Nix expression is used to patch the Android build-tools plugin package:
{deployAndroidPackage, lib, package, os, autopatchelf, makeWrapper, pkgs, pkgs_i686}:
deployAndroidPackage {
inherit package os;
buildInputs = [ autopatchelf makeWrapper ];
libs_x86_64 = lib.optionalString (os == "linux")
(lib.makeLibraryPath [ pkgs.glibc pkgs.zlib pkgs.ncurses5 ]);
libs_i386 = lib.optionalString (os == "linux")
(lib.makeLibraryPath [ pkgs_i686.glibc pkgs_i686.zlib pkgs_i686.ncurses5 ]);
patchInstructions = ''
${lib.optionalString (os == "linux") ''
export libs_i386=$packageBaseDir/lib:$libs_i386
export libs_x86_64=$packageBaseDir/lib64:$libs_x86_64
autopatchelf $packageBaseDir/lib64 libs --no-recurse
autopatchelf $packageBaseDir libs --no-recurse
''}
wrapProgram $PWD/mainDexClasses \
--prefix PATH : ${pkgs.jdk8}/bin
'';
noAuditTmpdir = true;
}
The above expression specifies the search libraries per architecture for x86 (
i386) and
x86_64 and automatically patches the binaries in the
lib64/ sub folder and base directories. The
autopatchelf tool ensures that no library of an incompatible architecture gets linked to a binary.
Discussion
The automated patching approach described in this blog post is not entirely a new idea -- in Nixpkgs, Aszlig Neusepoff created an autopatchelf hook that is integrated into the fixup phase of the
stdenv.mkDerivation {} function. It shares a number of similar features -- it accepts a list of library packages (the
runtimeDependencies environment variable) and automatically adds the provided runtime dependencies to the RPATH of all binaries in all the output folders.
There are also a number of differences -- my approach provides an
autopatchelf command-line tool that can be invoked in any stage of a build process and provides full control over the patching process. It can also be used outside a Nix builder environment, which is useful for experimentation purposes. This increased level of flexibility is required for more complex prebuilt binary packages, such as the Android SDK and its plugins -- for some plugins, you cannot generalize the patching process and you typically require more control.
It also offers better support to cope with repositories providing binaries of multiple architectures -- while the Nixpkgs version has a check that prevents incompatible libraries from being linked, it does not allow you to have fine grained control over library paths to consider for each architecture.
Another difference between my implementation and the autopatchelf hook is that it works with colon separated library paths instead of white space delimited Nix store paths. The autopatchelf hook makes the assumption that a dependency (by convention) stores all shared libraries in the
lib/ sub folder.
My implementation works with arbitrary library paths and arbitrary environment variables that you can specify as a parameter. To patch certain kinds of Android plugins, you must be able to refer to libraries that reside in unconventional locations in the same package. You can even use the
LD_LIBRARY_PATH environment variable (that is typically used to dynamically load libraries from a set of locations) in conjunction with
autopatchelf to make dynamic library references static.
There is also a use case that the
autopatchelf command-line tool does not support -- the autopatchelf hook can also be used for source compiled projects whose executables may need to dynamically load dependencies via the
dlopen() function call.
Dynamically loaded libraries are not known at link time (because they are not provided to the Nix-wrapped
ld command), and as a result, they are not added to the
RPATH of an executable. The Nixpkgs autopatchelf hook allows you to easily supplement the library paths of these dynamically loaded libraries after the build process completes.
Availability
The
autopatchelf command-line tool can be found in the
nix-patchtools repository. The goal of this repository to provide a collection of tools that help making the patching processes of complex prebuilt packages more convenient. In the future, I may identify more patterns and provide additional tooling to automate them.
autopatchelf is prominently used in
my refactored version of the Android SDK to automatically patch all ELF binaries. I have the intention to integrate this new Android SDK implementation into Nixpkgs soon.
Follow up
UPDATE: In the meantime, I have been working with Aszlig, the author of the autopatchelf hook, to get the functionality I need for auto patching the Android SDK integrated in Nixpkgs.
The result is that the Nixpkgs' version now implements a number of similar features and is also capable of patching the Android SDK. The
build-tools expression shown earlier, is now implemented as follows:
{deployAndroidPackage, lib, package, os, autoPatchelfHook, makeWrapper, pkgs, pkgs_i686}:
deployAndroidPackage {
inherit package os;
buildInputs = [ autoPatchelfHook makeWrapper ]
++ lib.optionalString (os == "linux") [ pkgs.glibc pkgs.zlib pkgs.ncurses5 pkgs_i686.glibc pkgs_i686.zlib pkgs_i686.ncurses5 ];
patchInstructions = ''
${lib.optionalString (os == "linux") ''
addAutoPatchelfSearchPath $packageBaseDir/lib
addAutoPatchelfSearchPath $packageBaseDir/lib64
autoPatchelf --no-recurse $packageBaseDir/lib64
autoPatchelf --no-recurse $packageBaseDir
''}
wrapProgram $PWD/mainDexClasses \
--prefix PATH : ${pkgs.jdk8}/bin
'';
noAuditTmpdir = true;
}
In the above expression we do the following:
- By adding the autoPatchelfHook package as a buildInput, we can invoke the autoPatchelf function in the builder environment and use it in any phase of a build process. To prevent the fixup hook from doing any work (that generalizes the patch process and makes the wrong assumptions for the Android SDK), the deployAndroidPackage function propagates the dontAutoPatchelf = true; parameter to the generic builder so that this fixup step will be skipped.
- The autopatchelf hook uses the packages that are specified as buildInputs to find the libraries it needs, whereas my implementation uses libs, libs_i386 or libs_x86_64 (or any other environment variable that is specified as a command-line parameter). It is robust enough to skip incompatible libraries, e.g. x86 libraries for x86-64 executables.
- My implementation works with colon separated library paths whereas autopatchelf hook works with Nix store paths making the assumption that there is a lib/ sub folder in which the libraries can be found that it needs. As a result, I no longer use the lib.makeLibraryPath function.
- In some cases, we also want the autopatchelf hook to inspect non-standardized directories, such as uncommon directories in the same package. To make this work, we can add additional paths to the search cache by invoking the addAutoPatchelfSearchPath function.
