Building AppImages with Nix

by Brian Picciano - (9 min read)

It’s been a bit since I’ve written an update on the cryptic nebula project, almost 5 months (since this post, which wasn’t officially part of the blog series but whatever). Since then it’s switched names to “cryptic-net”, and that we would likely use MinIO as our network storage service, but neither of those is the most interesting update.

The project had been stalled because of a lack of a build system which could fulfill the following requirements:

  • Network configuration (static IP, VPN certificates) of individual hosts is baked into the binary they run.

  • Binaries are completely static; no external dependencies need to exist on the host in order to run them.

  • Each binary runs a composition of multiple sub-services, each being a separate sub-process, and all of them having been configured to work together (with some possible glue code on our side) to provide the features we want.

  • The builder itself should be deterministic; no matter where it runs it should produce the same binary given the same input parameters.

Lacking such a build system we’re not able to distribute cryptic-net in a way which “just works”; it would require some kind of configuration, or some kind of runtime environment to be set up, both of which would be a pain for users. And lacking a definite build system makes it difficult to move forward on any other aspect of a project, as it’s not clear what may need to be redone in the future when the build system is decided upon.

Why not nix-bundle?

My usage of nix-bundle in a previous post was an attempt at fulfilling these requirements. Nix in general does very well in fulfilling all but the second requirement, and nix-bundle was supposed to fulfill even that by packaging a nix derivation into a static binary.

And all of this it did! Except that the mechanism of nix-bundle is a bit odd. The process of a nix-bundle’d binary jails itself within a chroot, which it then uses to fake the /nix/store path which nix built binaries expect to exist.

This might work in a lot of cases, but it did not work in ours. For one, nebula can’t create its network interface when run from inside nix-bundle’s chroot. For another, being run in a chroot means there’s going to be strange restrictions on what our binary is able to do and not.

AppImage

What we really needed was an AppImage. AppImages are static binaries which can bundle complex applications, even those which don’t expect to be bundled into single binaries. In this way the end result is the same as nix-bundle, but the mechanism AppImage uses is different and places far fewer restrictions on what we can and can’t do with our program.

Building Sub-Services Statically with Nix

It’s probably possible to use nix to generate an AppImage which has the /nix/store built into it, similar to what nix-bundle does, and therefore not worry about whether the binaries it’s bundling are static or not. But if your services are written in sane languages it’s not that difficult to build them statically and dodge the issue.

For example, here is how you build a go binary statically:

{
    buildGoModule,
    fetchFromGitHub,
}:
    buildGoModule rec {
        pname = "nebula";
        version = "1.4.0";

        src = fetchFromGitHub {
            owner = "slackhq";
            repo = pname;
            rev = "v${version}";
            sha256 = "lu2/rSB9cFD7VUiK+niuqCX9CI2x+k4Pi+U5yksETSU=";
        };

        vendorSha256 = "p1inJ9+NAb2d81cn+y+ofhxFz9ObUiLgj+9cACa6Jqg=";

        doCheck = false;

        subPackages = [ "cmd/nebula" "cmd/nebula-cert" ];

        CGO_ENABLED=0;
        tags = [ "netgo" ];
        ldflags = [
            "-X main.Build=${version}"
            "-w"
            "-extldflags=-static"
        ];
    };

And here’s how to statically build a C binary:

{
    stdenv,
    glibcStatic, # e.g. pkgs.glibc.static
}:
    stdenv.mkDerivation rec {
        pname = "dnsmasq";
        version = "2.85";

        src = builtins.fetchurl {
          url = "https://www.thekelleys.org.uk/dnsmasq/${pname}-${version}.tar.xz";
          sha256 = "sha256-rZjTgD32h+W5OAgPPSXGKP5ByHh1LQP7xhmXh/7jEvo=";
        };

        nativeBuildInputs = [ glibcStatic ];

        makeFlags = [
            "LDFLAGS=-static"
            "DESTDIR="
            "BINDIR=$(out)/bin"
            "MANDIR=$(out)/man"
            "LOCALEDIR=$(out)/share/locale"
        ];
    };

The derivations created by either of these expressions can be plugged right into the pkgs.buildEnv used to create the AppDir (see AppDir section below).

Process Manager

An important piece of the puzzle for getting cryptic-net into an AppImage was a process manager. We need something which can run multiple service processes simultaneously, restart processes which exit unexpectedly, gracefully handle shutting down all those processes, and coalesce the logs of all processes into a single stream.

There are quite a few process managers out there which could fit the bill, but finding any which could be statically compiled ended up not being an easy task. In the end I decided to see how long it would take me to implement such a program in go, and hope it would be less time than it would take to get circus, a python program, bundled into the AppImage.

2 hours later, pmux was born! Check it out. It’s a go program so building it looks pretty similar to the nebula builder above, so I won’t repeat it. However I will show the configuration we’re using for it within the AppImage, to show how it ties all the processes together:

processes:
    - name: nebula
      cmd: bin/nebula
      args:
        - "-config"
        - etc/nebula/nebula.yml

    - name: dnsmasq
      cmd: bin/dnsmasq
      args:
        - "-d"
        - "-C"
        - ${dnsmasq}/etc/dnsmasq/dnsmasq.conf

AppDir -> AppImage

Generating an AppImage requires an AppDir. An AppDir is a directory which contains all files required by a program, rooted to the AppDir. For example, if the program expects a file to be at /etc/some/conf, then that file should be places in the AppDir at <AppDir-path>/etc/some/conf.

These docs were very helpful for me in figuring out how to construct the AppDir. I then used the pkgs.buildEnv utility to create an AppDir derivation containing everything cryptic-net needs to run:

    appDir = pkgs.buildEnv {
        name = "cryptic-net-AppDir";
        paths = [

            # real directory containing non-built files, e.g. the pmux config
            ./AppDir

            # static binary derivations shown previously
            nebula
            dnsmasq
            pmux
        ];
    };

Once the AppDir is built one needs to use appimagetool to turn it into an AppImage. There is an appimagetool build in the standard nixpkgs, but unfortunately it doesn’t seem to actually work…

Luckily nix-bundle is working on AppImage support, and includes a custom build of appimagetool which does work!

{
    fetchFromGitHub,
    callPackage,
}: let
    src = fetchFromGitHub {
        owner = "matthewbauer";
        repo = "nix-bundle";
        rev = "223f4ffc4179aa318c34dc873a08cb00090db829";
        sha256 = "0pqpx9vnjk9h24h9qlv4la76lh5ykljch6g487b26r1r2s9zg7kh";
    };
in
    callPackage "${src}/appimagetool.nix" {}

Using callPackage on this expression will give you a functional appimagetool derivation. From there’s it’s a simple matter of writing a derivation which generates the AppImage from a created AppDir:

{
    appDir,
    appimagetool,
}:
    pkgs.stdenv.mkDerivation {
        name = "cryptic-net-AppImage";

        src = appDir;
        buildInputs = [ appimagetool ];
        ARCH = "x86_64"; # required by appimagetool

        builder = builtins.toFile "build.sh" ''
            source $stdenv/setup
            cp -rL "$src" buildAppDir
            chmod +w buildAppDir -R
            mkdir $out

            appimagetool cryptic-net "$out/cryptic-net-bin"
        '';
    }

Running that derivation deterministically spits out a binary at result/cryptic-net-bin which can be executed and run immediately, on any system using the x86_46 CPU architecture.

Fin

I’m extremely hyped to now have the ability to generate binaries for cryptic-net that people can just run, without them worrying about which sub-services that binary is running under-the-hood. From a usability perspective it’s way nicer than having to tell people to “install docker” or “install nix”, and from a dev perspective we have a really solid foundation on which to build a quite complex application.