In
my previous blog post, I have described a minimalistic architecture documentation approach for service-oriented systems based on
my earlier experiences with setting up basic configuration management repositories. I used this approach to construct a documentation catalog for the platform I have been developing at Mendix.
I also explained my motivation -- it improves developer effectiveness, team consensus and the on-boarding of new team members. Moreover, it is a crucial ingredient in improving the quality of a system.
Although we are quite happy with the documentation, my biggest inconvenience is that I had to derive it
entirely by hand -- I consulted various kinds of sources, but since existing documentation and information provided by people may be incomplete or inconsistent, I considered the source code and deployment configuration files the
ultimate source of truth, because no matter how elegantly a diagram is drawn, it is useless if it does not match the actual implementation.
Because a manual documentation process is very costly and time consuming, a more ideal situation would be to have an
automated approach that automatically derives architecture documentation from deployment specifications.
Since I am developing a deployment framework for service-oriented systems myself (
Disnix), I have decided to extend it with a
generator that can derive architecture diagrams and supplemental descriptions from the deployment models using the conventions I have described in my previous blog post.
Visualizing deployment architectures in Disnix
As explained in my previous blog post, the notation that I used for the diagrams was not something I invented from scratch, but something I borrowed from Disnix.
Disnix already has a feature (for quite some time) that can visualize
deployment architectures referring to a description that shows how the functional parts (the services/components) are mapped to physical resources (e.g. machines/containers) in a network.
For example, after deploying a service-oriented system, such as
my example web application system, by running:
$ disnix-env -s services.nix -i infrastructure.nix \
-d distribution-bundles.nix
You can visualize the corresponding deployment architecture of the system, by running:
$ disnix-visualize > out.dot
The above command-line instruction generates a directed graph in the
DOT language. The resulting dot file can be converted into a displayable image (such as a PNG or SVG file) by running:
$ dot -Tpng out.dot > out.png
Resulting in a diagram of the deployment architecture that may look as follows:
The above diagram uses the following notation:
- The light grey boxes denote machines in a network. In the above deployment scenario, we have two them.
- The ovals denote services (more specifically: in a Disnix-context, they reflect any kind of distributable deployment unit). Services can have almost any shape, such as web services, web applications, and databases. Disnix uses a plugin system called Dysnomia to make sure that the appropriate deployment steps are carried out for a particular type of service.
- The arrows denote inter-dependencies. When a service points to another service means that the latter is an inter-dependency of the former service. Inter-dependency relationships ensure that the dependent service gets all configuration properties so that it knows how to reach the dependency and the deployment system makes sure that inter-dependencies of a specific service are deployed first.
In some cases, enforcing the right activation order of activation may be expensive. It is also possible to drop the ordering requirement, as denoted by the dashed arrows. This is acceptable for redirects from the portal application, but not acceptable for database connections. - The dark grey boxes denote containers. Containers can be any kind of runtime environment that hosts zero or more distributable deployment units. For example, the container service of a MySQL database is a MySQL DBMS, whereas the container service of a Java web application archive can be a Java Servlet container, such as Apache Tomcat.
Visualizing the functional architecture of service-oriented systems
The services of which a service-oriented systems is composed are
flexible -- they can be deployed to various kinds of environments, such a test environment, a second fail-over production environment or a local machine.
Because services can be deployed to a variety of targets, it may also be desired to get an architectural view of the
functional parts only.
I created a new tool called:
dydisnix-visualize-services that can be used to generate
functional architecture diagrams by visualizing the services in the Disnix services model:
The above diagram is a visual representation of the services model of the example web application system, using a similar notation as the deployment architecture without showing any environment characteristics:
- Ovals denote services and arrows denote inter-dependency relationships.
- Every service is annotated with its type, so that it becomes clear what kind of a shape a service has and what kind of deployment procedures need to be carried out.
Despite the fact that the above diagram is focused on the functional parts, it may still look quite detailed, even from a functional point of view.
Essentially, the architecture of my example web application system is a "
system of sub systems" -- each sub system provides an isolated piece of functionality consisting of a database backend and web application front-end bundle. The portal sub system is the entry point and responsible for guiding the users to the sub systems implementing the functionality that they want to use.
It is also possible to annotate services in the Disnix services model with a group and description property:
{distribution, invDistribution, pkgs, system}:
let
customPkgs = import ../top-level/all-packages.nix {
inherit pkgs system;
};
groups = {
homework = "Homework";
literature = "Literature";
...
};
in
{
homeworkdb = {
name = "homeworkdb";
pkg = customPkgs.homeworkdb;
type = "mysql-database";
group = groups.homework;
description = "Database backend of the Homework subsystem";
};
homework = {
name = "homework";
pkg = customPkgs.homework;
dependsOn = {
inherit usersdb homeworkdb;
};
type = "apache-webapplication";
appName = "Homework";
group = groups.homework;
description = "Front-end of the Homework subsystem";
};
...
}
In the above services model, I have grouped every database and web application front-end bundle in a group that represents a sub system (such as
Homework). By adding the
--group-subservices parameter to the
dydisnix-visualize-services command invocation, we can simplify the diagram to only show the sub systems and how these sub systems are inter-connected:
$ dydisnix-visualize-services -s services.nix -f png \
--group-subservices
resulting in the following functional architecture diagram:
As may be observed in the picture above, all services have been grouped. The service groups are denoted by ovals with dashed borders.
We can also query sub architecture diagrams of every group/sub system. For example, the following command generates a sub architecture diagram for the
Homework group:
$ dydisnix-visualize-services -s services.nix -f png \
--group Homework --group-subservices
resulting in the following diagram:
The above diagram will only show the the services in the
Homework group and their context -- i.e. non-transitive dependencies and services that have a dependency on any service in the requested group.
Services that exactly fit the group or any of its parent groups will be displayed verbatim (e.g. the homework database back-end and front-end). The other services will be categorized into in the lowest common sub group (the
Users and
Portal sub systems).
For more complex architectures consisting of many layers, you may probably want to generate all available architecture diagrams in one command invocation. It is also possible to run the visualization tool in
batch mode. In batch mode, it will recursively generate diagrams for the top-level architecture and every possible sub group and stores them in a specified output folder:
$ dydisnix-visualize-services --batch -s services.nix -f svg \
--output-dir out
Generating supplemental documentation
Another thing I have explained in my previous blog post is that providing diagrams is useful, but they cannot clear up all confusion -- you also need to document and clarify additional details, such as the purposes of the services.
It also possible to generate a documentation page for each group showing a table of services with their descriptions and types:
The following command generates a documentation page for the
Homework group:
$ dydisnix-document-services -s services.nix --group Homework
It is also possible to adjust the generation process by providing a documentation configuration file (by using the
--docs parameter):
$ dydisnix-document-services -f services.nix --docs docs.nix \
--group Homework
The are a variety of settings that can be provided in a documentation configuration file:
{
groups = {
Homework = "Homework subsystem";
Literature = "Literature subsystem";
...
};
fields = [ "description" "type" ];
descriptions = {
type = "Type";
description = "Description";
};
}
The above configuration file specifies the following properties:
- The descriptions for every group.
- Which fields should be displayed in the overview table. It is possible to display any property of a service.
- A description of every field in the services model.
Like the visualization tool, the documentation tool can also be used in batch mode to generate pages for all possible groups and sub groups.
Generating a documentation catalog
In addition to generating architecture diagrams and descriptions, it is also possible to combine both tools to automatically generate a
complete documentation
catalog for a service-oriented system, such as the web application example system:
$ dydisnix-generate-services-docs -s services.nix --docs docs.nix \
-f svg --output-dir out
By opening the entry page in the output folder, you will get an overview of the top-level architecture, with a description of the groups.
By clicking on a group hyperlink, you can inspect the sub architecture of the corresponding group, such as the 'Homework' sub system:
The above page displays the sub architecture diagram of the 'Homework' subsystem and a description of all services belonging to that group.
Another particularly interesting aspect is the 'Portal' sub system:
The portal's purpose is to redirect users to functionality provided by the other sub systems. The above architecture diagram displays all the sub systems in grouped form to illustrate that there is a dependency relationship, but without revealing all their internal details that clutters the diagram with unnecessary implementation details.
Other features
The tools support more use cases than those described in this blog post -- it is also possible, for example, to create arbitrary layers of sub groups by using the '/' character as a delimiter in the group identifier. I also used the company platform as an example case, that can be decomposed into four layers.
Availability
The tools described in this blog post are part of the latest development version of
Dynamic Disnix -- a very experimental extension framework built on top of Disnix that can be used to make service-oriented systems self-adaptive by redeploying their services in case of events.
The reason why I have added these tools to Dynamic Disnix (and not the core Disnix toolset) is because the extension toolset has an infrastructure to parse and reflect over individual Disnix models.
Although I promised to make an official release of Dynamic Disnix a very long time ago, this still has not happened yet. However, the documentation feature is a compelling reason to stabilize the code and make the framework more usable.
Munich NixOS Meetup
