A scientific review written in 2024 at the Faculty of Mathematics and Computer Science, Ovidius University of Constanța.

Forewordh2

Software distribution is fundamental to the IT industry. The famous phrase “software is eating the world”¹ would not ring true if we could not run source code on our own computers or servers; software would remain only a dream on a programmer’s machine.

Simply put, software distribution means successfully running a program written on one machine on another machine. Microsoft’s greatest challenge was not purely a software challenge, but a distribution challenge: it could not have become the giant it is today if Windows had not been able to run on computers around the world.²

This essay first proposes a modern solution and then argues that, despite the substantial progress made in recent years, the most widely used tools in the industry do not truly solve the problem because they begin from flawed assumptions.

Defining the problem spaceh2

Modern software is built on the work and achievements of others. A modern programmer does not need to reinvent the wheel. Whether it is a file, a library, a compiler, or an operating-system program, anything used to build or run a program becomes one of its dependencies.

The challenge of software distribution is ensuring that a program can be built, compiled, or run in an environment other than its original developer’s machine—an environment that may or may not contain all its dependencies. The software industry has generally settled on two major distribution methods, with only minor variations between implementations.

Relying on dependencies provided by the systemh3

The first method assumes that a program’s dependencies already exist on the system where it will run. The program only needs to locate and use them. In principle, this minimizes the disk space occupied by the program because some dependencies may already be present and shared with other applications.

Unfortunately, software develops quickly. Programs often require a slightly modified dependency or a different version from the one installed by the user. The program may consequently fail to run—or its installation may break the rest of the system. Modern system package managers such as RPM, APT, Chocolatey, and Homebrew still suffer from versions of the same problem.

Bundling dependencies with the programh3

The second method was born from pragmatism: distribute the dependencies together with the program, eliminating the possibility that it will fail because a dependency is absent.

The obvious disadvantage is that every program occupies more storage. Some dependencies may also be loaded into memory even when another running program has already loaded a separate copy. For languages such as Go, the idea of a single executable containing all its dependencies—also known as a fat binary—has even become a selling point.

For other languages, bundling dependencies has become common enough to receive dedicated tooling and formats. Java has JAR files,³ while newer GraalVM native images⁴ can include the runtime alongside the dependencies, bringing Java closer to Go’s single-binary model.

Virtual machines and containersh3

Virtual machines such as VirtualBox and VMware, followed by containers such as Docker, combine aspects of both distribution methods. If the operating system can be considered a dependency, why not distribute it together with the program?

These technologies emerged in response to the complexity of dependency management in modern software, aided by advances in hardware and operating systems. They allow programs to keep running long after they were created, reduce the risk caused by dependency upgrades, and let teams use conflicting dependencies in better-isolated environments.

Although convenient and operationally useful, this approach inherits problems from both earlier methods. A virtual machine or container still contains an ordinary operating system with an ordinary dependency manager. In theory, we should be able to reproduce an image with the same dependencies every time. In practice, reproducibility is surprisingly easy to break.

General problems with software-distribution toolsh2

Most software-distribution failures originate in poor dependency management. Best practices—many popularized by Google engineers—recommend pinning every dependency to a specific version to prevent unexpected breakage. Even in modern systems, this advice can be difficult to follow.

What happens when a program needs two different versions of the same dependency? Package managers often report a conflict and reject the situation entirely. Dependencies normally have a standard installation path, and package managers correctly recognize that one version would overwrite another. Popular operating-system package managers—including APT on Ubuntu and Debian, RPM on Red Hat systems, Homebrew on macOS, and Chocolatey on Windows—are all affected. Because of class-resolution conflicts, multiple versions can also be difficult or impossible with project-level package managers such as Composer for PHP and NuGet for C#.

Pinning versions therefore introduces its own common problem and makes software production less convenient. In practical terms, ensuring that a program continues to behave in exactly the same way becomes difficult under these conditions.

Virtualization and containerization do not automatically provide reproducibility either. Their operating systems inherit the limitations of the same dependency managers. Docker is the most popular system for building and running containers and is consequently often used as a distribution system. This is not inherently wrong: the images it consumes can be versioned and identified cryptographically by a hash. The problem is that rebuilding a derived image from scratch may still retrieve different dependency versions.

Consider this common Dockerfile fragment:

FROM ubuntu:22.04

RUN apt-get update && apt-get -y install build-essential

Figure 1: A Docker build based on a common base image.

Every clean image build may retrieve a newer version of GCC, potentially changing how the application is compiled. Docker effectively moves responsibility for dependencies from the application developer to the image maintainer—usually a DevOps engineer or system administrator—who must ensure that the image’s dependencies remain unchanged.

Another issue is that most operating-system package managers are either source-based, compiling every piece of code, or binary-based, downloading prebuilt dependencies. They rarely do both. Source-based systems require considerable build time but can pursue reproducibility. Binary-based systems are often tied to weak versioning schemes and cannot always prove that a binary was built from the source revision it claims to represent.

Hermetic, deterministic, and reproducible systemsh2

A hermetic build system builds a program in isolation, without uncontrolled external influences. A deterministic build system fixes its inputs and dependencies so that a program can be produced from a precisely defined set of inputs. Together, these concepts provide the foundation for reproducible software.

A build is reproducible when rebuilding a program produces a bit-for-bit identical result every time. However, as we will see, a system can be hermetic and deterministic without every program built by it necessarily being reproducible.

Nix and the promise of complete reproducibilityh2

In 2006, Eelco Dolstra defended his doctoral dissertation at Utrecht University in the Netherlands. He had studied many of the software-distribution problems described above and created Nix to address them.⁵

At its core, Nix is a functional programming language that forms part of a powerful dependency-management and build system. It was designed around this purpose from the beginning and follows several important principles:

• Determinism through closed functions: every component in the Nix system is built by invoking a function whose inputs must be specified in advance. Changing an input causes the function to be evaluated again and the component to be rebuilt.
• Isolation: every version of a component or dependency receives an exact, unique path in the Nix store. Two components cannot share a path, even when they are different versions of the same software. The set of build inputs is encoded into a cryptographic hash that helps identify the evaluated component.
• Derivability: because components are constructed by evaluating functions, users can change how any component is built or how components are combined. Evaluating a component produces a derivation: an intermediate description between a function definition and its result that records dependency paths in the Nix store.

Because everything is expressed as a function of declared inputs, every dependency relationship—including runtime dependencies—is known before the build. This made it possible to create nixpkgs,⁶ a distribution channel whose components are designed to work together. Community contributions have grown it to more than 100,000 packages.

So far, this essay has avoided the word package. The term is common but has a fairly restrictive meaning in software distribution: a package is the result of a software build placed into a distributable form. In nixpkgs, a package is fundamentally a function definition with a set of inputs. Evaluated results are kept in binary caches so that users do not have to realize every derivation locally. From this pragmatic perspective, the name nixpkgs is appropriate, but software is not the only thing that we may want to distribute.

Returning to our original definition, distributing software means running it successfully on another machine. In modern systems, “successfully” often requires one or more configuration files. Configuration is not quite software, but it makes a program behave as its user intends. A configuration file may become invalid when the software it controls is upgraded. In an ideal system, this component should therefore be subject to the same rigor as the software itself. NixOS makes this possible by integrating the Nix language throughout the operating system’s configuration.

Nix promises reproducibility through hermeticity and determinism. Reproducible builds are prominently presented as a NixOS advantage,⁷ but the statement is not universally true. Consider a program that embeds its build date. That date changes with every build, so its output cannot be bit-for-bit identical even if neither the dependencies nor the source code changed.

Complete reproducibility therefore remains a goal rather than an automatic property. Only a subset of all packages in nixpkgs is reproducible. With more than 100,000 packages, verifying and correcting every build is an enormous effort, but the community continues to pursue it. At the time the original essay was written, the minimal NixOS image was reported as 99.81% reproducible—only one package out of 538 was not reproducible.⁸

The following shortened definition builds the C3 compiler with LLVM:

{
  llvmPackages,
  lib,
  fetchFromGitHub,
  cmake,
  curl,
  libxml2,
  libffi,
  xar,
}:

llvmPackages.stdenv.mkDerivation rec {
  src = fetchFromGitHub {
    owner = "c3lang";
    repo = "c3c";
    rev = "refs/tags/0.5.5";
    hash = "sha256-iOljE1BRVc92NJZj+nr1G6KkBTCwJEUOadXHUDNoPGk=";
  };

  nativeBuildInputs = [ cmake ];

  buildInputs = [
    llvmPackages.llvm
    llvmPackages.lld
    curl
    libxml2
    libffi
  ] ++ lib.optionals llvmPackages.stdenv.isDarwin [ xar ];

  # ...
}

Figure 2: Nix code for building a package.

In plain English, the definition says: “Create an LLVM-based derivation that builds source code fetched from GitHub and depends on LLVM, LLD, curl, and other libraries. On Darwin (macOS), also include xar. CMake is required only at build time.” The code is expressive enough to communicate its purpose without requiring deep familiarity with the Nix language.

Solutions provided by Nixh2

Nix’s flexibility and deterministic model allow it to distribute many kinds of software. The community’s effort to bring more software into the ecosystem has also produced foundational tools in nixpkgs that make future contributions easier.

One example—and a personal contribution—is the C3 compiler,⁹ which has been available in nixpkgs since version 24.05. Its package definition is represented in Figure 2.

Nix can do considerably more. The following examples revisit problems raised earlier and show how Nix addresses them.

Multiple versions of the same dependencyh3

How can software coexist with several versions of one dependency? Suppose we want an environment in which multiple versions of the C3 compiler can be tested against the same source code. A conventional industry solution would normally require virtual machines or containers. Nix provides a more direct approach because every dependency is stored at a unique path.

Placing a definition like the following at the root of a project and evaluating it with Nix opens a new shell containing both the nixpkgs 23.05 and nixpkgs 24.05 versions of c3c. If the two compiler versions share dependencies, those dependencies do not need to be downloaded or stored twice.

{ pkgs }:

let
  nixpkgs2305 = import (builtins.fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/refs/tags/23.05.tar.gz";
    sha256 = "...";
  }) {};

  nixpkgs2405 = import (builtins.fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/refs/tags/24.05.tar.gz";
    sha256 = "...";
  }) {};

  c3cVersions = pkgs.runCommandLocal "c3c-versions" {} ''
    mkdir -p $out/bin
    ln -s ${nixpkgs2305.c3c}/bin/c3c $out/bin/c3c-old
    ln -s ${nixpkgs2405.c3c}/bin/c3c $out/bin/c3c-new
  '';
in
pkgs.mkShell {
  packages = [ c3cVersions ];
}

Figure 3: A multiple-dependency example. Evaluating the definition makes both target versions available only within the current shell session.

Hermetic and deterministic Docker image buildsh3

Nix’s efficient dependency storage also makes it useful for building Docker images, sometimes more efficiently than Docker’s native build process. The word Docker can refer to several distinct things: the image format, the software that builds images, and the software that runs them. The image specification is now part of the Open Container Initiative (OCI),¹⁰ so Docker itself is not required to construct a compatible image.

Teams can therefore introduce Nix gradually into an existing container workflow without substantially changing how developers run the result. One example from the company where I work involved a PHP image. The Docker-built version occupied 1.4 GB uncompressed on disk. After switching the image build to Nix, it occupied only 400 MB because it contained only the dependencies we actually needed—which the migration also forced us to declare explicitly.

The definition below builds a standard OCI image containing all required dependencies. The PHP application is included because it is referenced by the image’s working directory, while PHP itself is included because it is a dependency and appears in the default command. Nix assembles these paths into a filesystem and packages it as an OCI image that Docker can load and run.

{ pkgs ? import <nixpkgs> {} }:

let
  app = pkgs.php82.buildComposerProject {
    src = ./src;
    pname = "api";
    version = "1.0.0";
    vendorHash = "sha256-...";
    composerLock = ./src/composer.lock;
  };
in
pkgs.dockerTools.buildImage {
  config = {
    WorkingDir = "${app}/share/php/api";
    Cmd = [ "${pkgs.php82}/bin/php" "-S" "0.0.0.0:80" ];
  };
}

Figure 4: Building a Docker-compatible image for a PHP application.

In this case, Nix solves the determinism problem in container-image construction, and the resulting image is reproducible as well.

Disadvantages of Nixh2

Nix is many things at once: a language, a software build system, a distribution system, and a philosophy. To begin using one aspect, a newcomer often has to learn all of them at the same time. This is its greatest disadvantage and a major reason for its relatively limited adoption in the industry. Nix does not offer a magic command that immediately solves every problem; it requires an upfront investment of time. Being a functional programming language does not help its popularity either.

For many problems, the solutions already used by the industry are good enough. Even if some of them begin from questionable assumptions, organizations can continue using them without retraining their workforce, and extensive testing can validate the resulting distributions. The inefficiency of current systems is often not considered a problem at all, especially as network speeds and storage capacities continue to grow.

Nix was never a purely academic language or tool. Yet, with one of the largest software distribution collections ever created, the project increasingly struggles under its own weight. Its advantages—especially deterministic dependencies, which require maintaining an enormous graph of relationships between components—make the infrastructure expensive to sustain. The cost of this correctness risks affecting the entire Nix project, which continually seeks sponsorship and tries to keep expenses under control.

Finally, the way Nix stores dependencies and forces programs to reference them makes it incompatible with native Windows usage. Nix is clearly designed for Unix systems such as Linux, FreeBSD, and macOS.

Conclusionh2

Software distribution and dependency management are not fully solved problems. Modern solutions remain incomplete and often begin from flawed assumptions. By applying strong principles, Nix makes efficient software distribution systems possible, while its functional and deterministic approach eliminates many dependency conflicts.

Although Nix predates newer and more popular distribution systems such as Docker, it can be used successfully alongside them. This combination can make software distribution more efficient and help teams adopt Nix incrementally as part of their existing development process.

Bibliographyh2

1. Dolstra, Eelco. The Purely Functional Software Deployment Model (Dissertation). Utrecht University, 2006. dspace.library.uu.nl
2. Dolstra, Eelco, and Andres Löh. “NixOS: A Purely Functional Linux Distribution.” Proceedings of the 13th ACM SIGPLAN International Conference on Functional Programming, 2008, pp. 367–378. doi.org/10.1145/1411204.1411255
3. Beyer, Betsy, et al., editors. Site Reliability Engineering: How Google Runs Production Systems. O’Reilly, 2016. sre.google