Using Nix to build multi-package, full stack Haskell apps
July 16, 2020 - 6 min read
This post has been updated on June 16 following the finalization of the Nix port.
As part of my job working on an open source logic textbook, I picked up a Haskell codebase that was rather hard to build. This was problematic for new contributors getting started, so I wanted to come up with a better process. Further, because of legal requirements for public institutions in BC, I need to be able to host this software in Canada, for which it would be useful to be able to have CI and containerization (where it is directly useful to have an easy to set up build environment).
The value proposition of Nix is that it ensures that regardless of who is building the software or where it is being built, it is possible to ensure the environment where this is done is exactly the same. It also makes it fairly easy for users to set up that environment. Finally, it has a package and binaries for GHCJS, which provides extraordinary time and effort savings by avoiding the process of setting up dependencies for and compiling GHCJS.
A lot of the documentation around Nix is styled like programming documentation rather than like packaging documentation, which makes it harder to figure out where to start with packaging. For example, it is not really clear what exactly the "correct" way to structure a multiple package Haskell project is: are you supposed to use overlays, overrides or other methods? I chose to use overlays based on the nixpkgs documentation's suggestions that they are the most advanced (and thus modern?) way of putting stuff into nixpkgs.
The most significant tip I can give for doing Nix development and especially reading other Nix package source code is that the best way of understanding library calls is to read the nixpkgs source. This is for a couple of reasons: for one, the user facing documentation seems to be less complete than the documentation comments on functions, and often it is useful to read the library function source alongside the documentation.
Usually I keep a tab in my neovim open to the nixpkgs source and use either nix-doc or ripgrep to search for the function I am interested in.
This post summarizes the design decisions that went into implementing Nix for this existing full stack app. If you'd like to read the source, it is available on GitHub.
I have a top-level default.nix
that imports nixpkgs with overlays for each
conceptual part of the application (this could all be done in one but it is
useful to separate them for maintenance purposes). A simplified version is
below:
{ compiler ? "ghc865",
ghcjs ? "ghcjs"
}:
let nixpkgs = import (builtins.fetchTarball {
name = "nixpkgs-20.03-2020-06-28";
url = "https://github.com/NixOS/nixpkgs/archive/f8248ab6d9e69ea9c07950d73d48807ec595e923.zip";
sha256 = "009i9j6mbq6i481088jllblgdnci105b2q4mscprdawg3knlyahk";
}) {
config = {
# Use this if you use 'broken' packages that are fixed in an overlay
allowBroken = true;
};
overlays = [
(import ./client.nix { inherit ghcjs; })
(import ./server.nix { inherit ghcjs compiler; })
];
};
in {
client = nixpkgs.haskell.packages."${ghcjs}".Client;
server = nixpkgs.haskell.packages."${compiler}".Server;
}
In each Haskell package, use cabal2nix .
to generate nix files for the
package. These nix files can then be picked up with
lib.callPackage
in an overlay:
{ ghcjs ? "ghcjs", compiler ? "ghc865" }:
self: super:
let overrideCabal = super.haskell.lib.overrideCabal;
in {
haskell = super.haskell // {
packages = super.haskell.packages // {
"${compiler}" = super.haskell.packages."${compiler}".override {
overrides = newpkgs: oldpkgs: {
Common1 = oldpkgs.callPackage ./Common1/Common1.nix { };
# ...
};
};
};
};
}
Shells
You could normally use
nixpkgs.haskell.packages.${ghcVer}.shellFor
to construct a shell. However, this is not ideal for multiple package projects
since it will invariably make Nix build some of your projects because they are
"dependencies".
There does not appear to be any built in resolution for this. However,
reflex-platform, has
integrated a module called
workOnMulti
.
I thus took the opportunity to extricate it from its dependencies on the rest
of reflex-platform to be able to use it independently. This extracted version
is available here.
It can be used thus:
let # import nixpkgs with overlays...
workOnMulti = import ./nix/work-on-multi.nix {
inherit nixpkgs;
# put whatever tools you want in the shell environments here
generalDevTools = _: {
inherit (nixpkgs) cabal2nix;
inherit (nixpkgs.haskell.packages."${ghcVer}")
Cabal
cabal-install
ghcid
hasktags;
};
};
in {
ghcShell = workOnMulti {
envPackages = [
"Common1"
"Common2"
"Server"
];
env = with nixpkgs.haskell.packages."${ghcVer}"; {
# enable hoogle in the environment
ghc = ghc.override {
override = self: super: {
withPackages = super.ghc.withHoogle;
};
};
inherit Common1 Common2 Server mkDerivation;
};
};
}
Then, you can use nix-shell
with this attribute: nix-shell -A ghcShell
.
Build with Cabal as usual (cabal new-build all
), assuming you've built the
GHCJS parts already (see below).
GHCJS
GHCJS breaks many unit tests such that they freeze the Nix build process. You
can override mkDerivation
to disable most packages' unit tests. For some,
this does not work because nixpkgs puts test runs in a conditional already,
which causes the mkDerivation
override to be ignored.
haskell.lib.dontCheck
can be used to deal with these cases.
# inside the config.packageOverrides.haskell.packages.${compiler}.override call
mkDerivation = args: super.mkDerivation (args // {
doCheck = false;
enableLibraryProfiling = false;
});
To integrate the GHCJS-built browser side code with the rest of the project, a
method inspired by
reflex-platform
is used. Namely, nix-build -o client-out -A client
is used to build the
client and put a symbolic link in a known place, then manually created symbolic links are
placed in the static folder pointing back into this client output link.
For package builds, a preConfigure
script
is used with
haskell.lib.overrideCabal
to replace these links with paths in the Nix store for the browser JavaScript.
A dependency on the built JavaScript is also added so it gets pulled in.
Custom dependencies
Larger projects have a higher likelihood of having dependencies on Hackage
packages that are not in nixpkgs, or absolutely need to be a specific version.
It's easy to integrate these into the nix project using cabal2nix
:
$ cabal2nix cabal://your-package-0.1.0.0 | tee nix/your-package.nix
These can then be integrated into the project by using
lib.callPackage
.
While it is also possible to use
callCabal2nix
,
I choose not to for reasons of initial build performance and reproducibility:
cabal2nix
is not fast, and inadvertent updates could happen when updates are
made on the Hackage side, whereas checking in cabal2nix
output ensures that
exactly the same package is used.
Final thoughts
This project was very stimulating and challenging, and I learned a lot about both my own learning process for new complex technologies and the technologies themselves. Throughout this process, for the first time, I treated learning a new technology like class material and read the documentation from top to bottom, taking notes on the useful parts. This made it easier to keep the unfamiliar language behaviour in mind while simultaneously reading code.
I will use this strategy again because although it is slightly slower, it seemed to result in fewer trips to Google to check things and generally better comprehension.