In two earlier blog posts, I have described various
pieces of
my custom web framework that I used to actively develop many years ago. The framework is quite
modular -- every concern, such as layout management, data management, the editor, and the gallery, are separated into packages that can be deployed independently, so that web applications only have to include what they actually need.
Although modularity is quite useful for a variety of reasons, the framework did not start out as being modular in the beginning -- when I just started developing web applications in PHP, I did not reuse anything at all. Slowly, I discovered similarities between my projects and started sharing snippets of common functionality between them. Gradually, I learned that keeping these common aspects up to date became a burden. As a result, I developed a "
common framework" that I reused among all my PHP projects.
Having a common framework for my web application projects reduced the amount of required maintenance, but introduced a new drawback -- its
size kept growing and growing. As a result, many simple web applications that only required a small subset of the framework's functionality still had to embed the entire framework, making them unnecessarily big.
Today, a bit of extra PHP code is not so much of a problem, but around the time I was still actively developing web applications, many shared web hosting providers only offered a small amount of storage capacity, typically just a few megabytes.
To cope with the growing size of the framework, I decided to
modularize the code by separating the framework's concerns into packages that can be deployed independently. I "invented" my own conventions to integrate the framework packages into web applications:
- In the base directory of the web application project, I create a lib/ directory that contains symlinks to the framework packages.
- In every PHP script that displays a page (typically only index.php), I configure the include path to refer to the packages' content in the lib/ folder, such as:
set_include_path("./lib/sblayout:./lib/sbdata:./lib/sbcrud");
- Each PHP module is responsible for loading the desired classes or utility functions from the framework packages. As a result, I ended up writing a substantial amount of require() statements, such as:
require_once("data/model/Form.class.php");
require_once("data/model/field/HiddenField.class.php");
require_once("data/model/field/TextField.class.php");
require_once("data/model/field/DateField.class.php");
require_once("data/model/field/TextAreaField.class.php");
require_once("data/model/field/URLField.class.php");
require_once("data/model/field/FileField.class.php");
After my (approximately) 8 years of absence from the PHP domain, I discovered that a tool has been developed to support convenient construction of modular PHP applications:
composer. Composer is heavily inspired by the
NPM package manager, that is the defacto package delivery mechanism for
Node.js applications.
In the last couple of months (it progresses quite slowly as it is a non-urgent side project), I have decided to get rid of my custom modularity conventions in my framework packages, and to adopt composer instead.
Furthermore, composer is a useful deployment tool, but its scope is limited to PHP applications only. As frequent readers may probably already know, I use
Nix-based solutions to deploy
entire software systems (that are also composed of non-PHP packages) from a single declarative specification.
To be able to include PHP composer packages in a Nix deployment process, I have developed a generator named:
composer2nix that can be used to generate Nix deployment expressions from composer configuration files.
In this blog post, I will explain the concepts of
composer2nix and show how it can be used.
Using composer
Using composer is generally quite straight forward. In the most common usage scenario, there is typically a PHP project (often a web application) that requires a number of dependencies. By changing the current working folder to the project directory, and running:
$ composer install
Composer will obtain all required dependencies and stores them in the
vendor/ sub directory.
The
vendor/ folder follows a very specific organisation:
$ find vendor/ -maxdepth 2 -type d
vendor/bin
vendor/composer
vendor/phpdocumentor
vendor/phpdocumentor/fileset
vendor/phpdocumentor/graphviz
vendor/phpdocumentor/reflection-docblock
vendor/phpdocumentor/reflection
vendor/phpdocumentor/phpdocumentor
vendor/svanderburg
vendor/svanderburg/pndp
...
The
vendor/ folder structure (mostly) consists two levels: the outer directory defines the namespace of the packages and the inner directory the package names.
There are a couple of folders deviating from this convention -- most notably, the
vendor/composer directory, that is used by composer to track package installations:
$ ls vendor/composer
autoload_classmap.php
autoload_files.php
autoload_namespaces.php
autoload_psr4.php
autoload_real.php
autoload_static.php
ClassLoader.php
installed.json
LICENSE
In addition to obtaining packages and storing them in the
vendor/ folder,
composer also generates
autoload scripts (as shown above) that can be used to automatically make code units (typically classes) provided by the packages available for use in the project. Adding the following statement to one of your project's PHP scripts:
require_once("vendor/autoload.php");
suffices to load the functionality exposed by the packages that composer installs.
Composer can be used to install both
runtime and
development dependencies. Many development dependencies (such as
phpunit or
phpdocumentor) provide command-line utilities to carry out tasks. Composer packages can also declare which executables they provide. Composer automatically generates symlinks for all provided executables in the:
vendor/bin folder:
$ ls -l vendor/bin/
lrwxrwxrwx 1 sander users 29 Sep 26 11:49 jsonlint -> ../seld/jsonlint/bin/jsonlint
lrwxrwxrwx 1 sander users 41 Sep 26 11:49 phpdoc -> ../phpdocumentor/phpdocumentor/bin/phpdoc
lrwxrwxrwx 1 sander users 45 Sep 26 11:49 phpdoc.php -> ../phpdocumentor/phpdocumentor/bin/phpdoc.php
lrwxrwxrwx 1 sander users 34 Sep 26 11:49 pndp-build -> ../svanderburg/pndp/bin/pndp-build
lrwxrwxrwx 1 sander users 46 Sep 26 11:49 validate-json -> ../justinrainbow/json-schema/bin/validate-json
For example, you can run the following command-line instruction from the base directory of a project to generate API documentation:
$ vendor/bin/phpdocumentor -d src -t out
In some cases (the composer documentation often discourages this) you may want to install end-user packages
globally. They can be installed into the global composer configuration directory by running:
$ composer global require phpunit/phpunit
After installing a package globally (and adding:
$HOME/.config/composer/vendor/bin directory to the
PATH environment variable), we should be able to run:
$ phpunit --help
The composer configuration
The deployment operations that composer carries out are driven by a configuration file named:
composer.json. An example of such a configuration file could be:
{
"name": "svanderburg/composer2nix",
"description": "Generate Nix expressions to build PHP composer packages",
"type": "library",
"license": "MIT",
"authors": [
{
"name": "Sander van der Burg",
"email": "svanderburg@gmail.com",
"homepage": "http://sandervanderburg.nl"
}
],
"require": {
"svanderburg/pndp": "0.0.1"
},
"require-dev": {
"phpdocumentor/phpdocumentor": "2.9.x"
},
"autoload": {
"psr-4": { "Composer2Nix\\": "src/Composer2Nix" }
},
"bin": [ "bin/composer2nix" ]
}
The above configuration file declares the following configuration properties:
- A number of meta attributes, such as the package name, description, license and authors.
- The package type. The type: library indicates that this project is a library that can be used in another project.
- The project's runtime (require) and development (require-dev) dependencies. In a dependency object, the keys refer to the package names and the values to version specifications that can be either:
- A semver compatible version specifier that can be an exact version (e.g. 0.0.1), wildcard (e.g. 1.0.x), or version range (e.g. >= 1.0.0).
- A version alias that directly (or indirectly) resolves to a branch in the VCS repository of the dependency. For example, the dev-master version specifier refers to the current master branch of the Git repository of the package.
- The autoloader configuration. In the above example, we configure the autoloader to load all classes belonging to the Composer2Nix namespace, from the src/Composer2Nix sub directory.
By default, composer obtains all packages from the
Packagist repository. However, it is also possible to consult other kinds of repositories, such as external HTTP sites or VCS repositories of various kinds (including Git, Mercurial and Subversion).
External repositories can be specified by adding a '
repositories' object to the composer configuration:
{
"name": "svanderburg/composer2nix",
"description": "Generate Nix expressions to build PHP composer packages",
"type": "library",
"license": "MIT",
"authors": [
{
"name": "Sander van der Burg",
"email": "svanderburg@gmail.com",
"homepage": "http://sandervanderburg.nl"
}
],
"repositories": [
{
"type": "vcs",
"url": "https://github.com/svanderburg/pndp"
}
],
"require": {
"svanderburg/pndp": "dev-master"
},
"require-dev": {
"phpdocumentor/phpdocumentor": "2.9.x"
},
"autoload": {
"psr-4": { "Composer2Nix\\": "src/Composer2Nix" }
},
"bin": [ "bin/composer2nix" ]
}
In the above example, we have defined
PNDP's GitHub repository as an external repository and changed the version specifier of
svanderburg/pndp to use the latest Git master branch.
Composer uses a version resolution strategy that will parse composer configuration files and branch names in all repositories to figure out where a version can be obtained from and takes the first option that matches the dependency specification. Packagist is consulted last, making it possible for the user to override dependencies.
Pinpointing dependency versions
The version specifiers of dependencies in a
composer.json configuration file are
nominal and have some drawbacks when it comes to reproducibility -- for example, the version specifier:
>= 1.0.1 may resolve to version
1.0.2 today and to
1.0.3 tomorrow, making it very difficult to exactly reproduce a deployment elsewhere at a later point in time.
Although direct dependencies can be easily controlled by the user, it is quite difficult to control the version resolutions of the transitive dependencies. To cope with this problem, composer will always generate
lock files (
composer.lock) that pinpoint the exact dependency versions (including all transitive dependencies) the first time when it gets invoked (or when
composer update is called):
{
"_readme": [
"This file locks the dependencies of your project to a known state",
"Read more about it at https://getcomposer.org/doc/01-basic-usage.md#composer-lock-the-lock-file",
"This file is @generated automatically"
],
"content-hash": "ca5ed9191c272685068c66b76ed1bae8",
"packages": [
{
"name": "svanderburg/pndp",
"version": "v0.0.1",
"source": {
"type": "git",
"url": "https://github.com/svanderburg/pndp.git",
"reference": "99b0904e0f2efb35b8f012892912e0d171e9c2da"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/svanderburg/pndp/zipball/99b0904e0f2efb35b8f012892912e0d171e9c2da",
"reference": "99b0904e0f2efb35b8f012892912e0d171e9c2da",
"shasum": ""
},
"bin": [
"bin/pndp-build"
],
...
}
...
]
}
By bundling the
composer.lock file with the package, it becomes possible to reproduce a deployment elsewhere with the exact same package versions.
The Nix package manager
Nix is a package manager whose main purpose is to build all kinds of software packages from source code, such as GNU Autotools, CMake, Perl's MakeMaker, Apache Ant, and Python projects.
Nix's main purpose is not be a build tool (it can actually also be used for building projects, but this application area is still highly experimental). Instead, Nix manages dependencies and
complements existing build tools by providing dedicated build environments to make deployments reliable and reproducible, such as clearing all environment variables, making files read-only after the package has been built, restricting network access and resetting the files' timestamps to 1.
Most importantly, in these dedicated environments Nix ensures that only
specified dependencies can be found. This may probably sound inconvenient at first, but this property exists for a good reason: if a package unknowingly depends on another package then it may work on the machine where it has been built, but may fail on another machine because this unknown dependency is missing. By building a package in a pure environment in which all dependencies are known, we eliminate this problem.
To provide stricter purity guarantees, Nix isolates packages by storing them in a so-called "Nix store" (that typically resides in:
/nix/store) in which every directory entry corresponds to a package. Every path in the Nix store is prefixed by hash code, such as:
/nix/store/2gi1ghzlmb1fjpqqfb4hyh543kzhhgpi-firefox-52.0.1
The hash is derived from all build-time dependencies to build the package.
Because every package is stored in its own path and variants of packages never share the same name because of the hash prefix, it becomes harder for builds to accidentally succeed because of undeclared dependencies. Dependencies can only be found if the environment has been configured in such a way that the Nix store paths to the packages are known, for example, by configuring environment variables, such as:
export PATH=/nix/store/5vyssyqvbirdihqrpqhbkq138ax64bjy-gnumake-4.2.1/bin.
The Nix expression language and build environment abstractions have all kinds of facilities to make the configuration of dependencies convenient.
Integrating composer deployments into Nix builder environments
Invoking
composer in a Nix builder environment introduces an additional challenge -- composer is not only a tool that does build management (e.g. it can execute script directives that can carry out arbitrary build steps), but also
dependency management. The latter property conflicts with the Nix package manager.
In a Nix builder environment, network access is typically restricted, because it affects reproducibility (although it still possible to hack around this restriction) -- when downloading a file from an external site it is not known in advance what you will get. An unknown artifact influences the outcome of a package build in unpredictable ways.
Network access in Nix build environments is only permitted in so-called
fixed output derivations. For a fixed output derivation, the output hash must be known in advance so that Nix can verify whether we have obtained the artifact we want.
The solution to cope with a conflicting dependency manager is by
substituting it -- we must let Nix obtain the dependencies and force the tool to only execute its build management tasks.
We can populate the
vendor/ folder ourselves. As explained earlier, the
composer.lock file stores the exact versions of pinpointed dependencies including all transitive dependencies. For example, when a project declares
svanderburg/pndp version
0.0.1 as a dependency, it may translate to the following entry in the
composer.lock file:
"packages": [
{
"name": "svanderburg/pndp",
"version": "v0.0.1",
"source": {
"type": "git",
"url": "https://github.com/svanderburg/pndp.git",
"reference": "99b0904e0f2efb35b8f012892912e0d171e9c2da"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/svanderburg/pndp/zipball/99b0904e0f2efb35b8f012892912e0d171e9c2da",
"reference": "99b0904e0f2efb35b8f012892912e0d171e9c2da",
"shasum": ""
},
...
}
...
]
As can be seen in the code fragment above, the dependency translates to two kinds of pinpointed source objects -- a
source reference to a specific revision in a Git repository and a
dist reference to a zipball containing a snapshot of the given Git revision.
The reason why every dependency translates to two kinds of objects is that composer supports two kinds of installation modes: source (to obtain a dependency directly from a VCS) and dist (to obtain a dependency from a zipball).
We can translate the 'dist' reference into the following Nix function invocation:
"svanderburg/pndp" = {
targetDir = "";
src = composerEnv.buildZipPackage {
name = "svanderburg-pndp-99b0904e0f2efb35b8f012892912e0d171e9c2da";
src = fetchurl {
url = https://api.github.com/repos/svanderburg/pndp/zipball/99b0904e0f2efb35b8f012892912e0d171e9c2da;
sha256 = "19l7i7adp76bjf32x9a2ykm0r5cgcmi4wf4cm4127miy3yhs0n4y";
};
};
};
and the 'source' reference to the following Nix function invocation:
"svanderburg/pndp" = {
targetDir = "";
src = fetchgit {
name = "svanderburg-pndp-99b0904e0f2efb35b8f012892912e0d171e9c2da";
url = "https://github.com/svanderburg/pndp.git";
rev = "99b0904e0f2efb35b8f012892912e0d171e9c2da";
sha256 = "15i311dc0123v3ppa69f49ssnlyzizaafzxxr50crdfrm8g6i4kh";
};
};
(As a sidenote: we need the
targetDir property to provide compatibility with
the deprecated PSR-0 autoloading standard. Old autoload packages can be stored in a sub folder of a package residing in the
vendor/ structure.)
To generate the above function invocations, we need more than just the properties provided by the
composer.lock file. Since download functions in Nix are fixed output derivations, we must compute the output hashes of the downloads by invoking a Nix prefetch script, such as
nix-prefetch-url or
nix-prefetch-git. The
composer2nix generator will automatically invoke the appropriate prefetch script to augment the generated expressions with output hashes.
To ensure maximum compatibility with composer's behaviour, the dependencies obtained by Nix must be
copied into to the
vendor/ folder. In theory, symlinking would be more space efficient, but experiments have shown that some packages (such as
phpunit) may attempt to load the project's autoload script, e.g. by invoking:
require_once(realpath("../../autoload.php"));
The above require invocation does not work if the dependency is a symlink -- the require path resolves to a path in the Nix store (e.g.
/nix/store/...). The parent's parent path corresponds to
/nix where no autoload script is stored. (As a sidenote: I have decided to still provide symlinking as an option for deployment scenarios where this is not an issue).
After some experimentation, I discovered that composer uses the following file to track which packages have been installed:
vendor/composer/installed.json. The contents appears to be quite similar to the
composer.lock file:
[
{
"name": "svanderburg/pndp",
"version": "v0.0.1",
"version_normalized": "0.0.1.0",
"source": {
"type": "git",
"url": "https://github.com/svanderburg/pndp.git",
"reference": "99b0904e0f2efb35b8f012892912e0d171e9c2da"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/svanderburg/pndp/zipball/99b0904e0f2efb35b8f012892912e0d171e9c2da",
"reference": "99b0904e0f2efb35b8f012892912e0d171e9c2da",
"shasum": ""
},
...
},
...
]
Reconstructing the above file can be done by merging the contents of the
packages and
packages-dev objects in the
composer.lock file.
Another missing piece in the puzzle is the autoload scripts. We can force composer to dump the autoload script, by running:
$ composer dump-autoload --optimize
The above command generates an optimized autoloader script. A non-optimized autoload script dynamically inspects the contents of the package folders to load modules. This is convenient in the development stage of a project, in which the files continuously change, but in production environments this introduces quite a bit of load time overhead.
Since packages in the Nix store can never change after they have been built, it makes no sense to generate a non-optimized autoloader script.
Finally, the last remaining practical issue, is PHP packages providing command-line utilities. Most executables have the following
shebang line:
#!/usr/bin/env php
To ensure that these CLI tools work in Nix builder environments, the above shebang must be subsituted by the PHP executable that resides in the Nix store.
After carrying out the above described steps, running the following command:
$ composer install --optimize-autoloader
is simply just a formality -- it will not download or change anything.
Use cases
composer2nix has a variety of use cases. The most obvious one is to use it to package a web application project with Nix instead of
composer. Running the following command generates Nix expressions from the composer configuration files:
$ composer2nix
By running the following command, we can use Nix to obtain the dependencies and generate a package with a
vendor/ folder:
$ nix-build
$ ls result/
index.php vendor/
In addition to web applications, we can also deploy command-line utility projects implemented in PHP. For these kinds of projects it make more sense generate a
bin/ sub folder in which the executables can be found.
For example, for the
composer2nix project, we can generate a CLI-specific expression by adding the
--executable parameter:
$ composer2nix --executable
We can install the composer2nix executable in our Nix profile by running:
$ nix-env -f default.nix -i
and then invoke
composer2nix as follows:
$ composer2nix --help
We can also deploy third party command-line utilities directly from the Packagist repository:
$ composer2nix -p phpunit/phpunit
$ nix-env -f default.nix -iA phpunit-phpunit
$ phpunit --version
The most powerful application is not the integration with Nix itself, but the integration with other Nix projects. For example, we can define a
NixOS configuration running an Apache HTTP server instance with PHP and our example web application:
{pkgs, config, ...}:
let
myexampleapp = import /home/sander/myexampleapp {
inherit pkgs;
};
in
{
services.httpd = {
enable = true;
adminAddr = "admin@localhost";
extraModules = [
{ name = "php7"; path = "${pkgs.php}/modules/libphp7.so"; }
];
documentRoot = myexampleapp;
};
...
}
We can deploy the above NixOS configuration as follows:
$ nixos-rebuild switch
By running only one simple command-line instruction, we have a running system with the Apache webserver serving our web application.
Discussion
In addition to
composer2nix, I have also been responsible for developing
node2nix, a tool that generates Nix expressions from NPM package configurations. Because composer is heavily inspired by NPM, we see many similarities in the architecture of both generators. For example, both generate the same kinds of expressions (a builder environment, a packages expression and a composition expression), have a similar separation of concerns, and both use an internal DSL for generating Nix expressions (
NiJS and
PNDP).
There are also a number of conceptual differences -- dependencies in NPM can be private to a package or shared among multiple packages. In composer, all dependencies in a project are shared.
The reason why NPM's dependency management is more powerful is because Node.js uses the
CommonJS module system. CommonJS considers each file to be a unique module. This, for example, makes it possible for one module to load a version of a package from a certain filesystem location and another version of the same package from another filesystem location within the same project.
By contrast, in PHP, isolation is accomplished by the namespace declarations in each file. Namespaces can not be dynamically altered so that multiple versions can safely coexist in one project. Furthermore, the
vendor/ directory structure makes it possible to store only one variant of a package.
Despite the fact that composer's dependency management is less powerful makes constructing a generator much more straightforward compared to NPM.
Another feature that composer supports for quite some time, and NPM until very recently is pinpointing/locking dependencies. When generating Nix expressions from NPM package configurations, we must replicate NPM's dependency resolving algorithm. In composer, we can simply take whatever the
composer.lock file provides. The lock file saves us from replicating the dependency lookup process making the generation process considerably easier.
Acknowledgments
My implementation is not the first attempt that tries to integrate composer with Nix. After a few days of developing, I discovered
another attempt that seems to be in a very early development stage. I did not try or use this version.
Availability
composer2nix can be obtained from
Packagist and
my GitHub page.
Despite the fact that I did quite a bit of research,
composer2nix should still be considered a prototype. One of its known limitations is that it does not support fossil repositories yet.
Munich NixOS Meetup
