Building Multiplatform Container Images the Easy Way

While doing some work on Trow, our image management solution for Kubernetes, we discovered how easy new tooling makes it to produce container images for multiple architectures. This post will show how with just a little work we were able to create Trow images for both ARM and Intel platforms.

Docker has for some time supported multiplatform images where an image can have multiple different versions for different architectures and platforms. Pulling a multiplatform image will pull only the version for your current architecture. For example, docker pull debian:latest will pull the amd64 image on my laptop and the armv7 image on my Raspberry Pi. The way this works is through the use of manifest lists aka “fat manifests” , which contain pointers to images for different architectures.

The Trow project has had multiplatform builds for some time, but the code to do it was frustratingly complex and dependent on Phil Estes’ manifest-tool. With recent updates to buildx, the GitHub Container Registry now supporting manifest lists, and some inspiration from Tõnis Tiigi’s xx project, I realised it was possible to simplify things a lot. It’s taken a bit of work but we now have a new build for Trow that only requires a single Dockerfile:

# syntax=docker/dockerfile:1

#Note we build on host platform and cross-compile to target arch
FROM --platform=$BUILDPLATFORM rust:latest as cross
ARG TARGETARCH
WORKDIR /usr/src/trow
COPY docker/platform.sh .
RUN ./platform.sh # should write /.platform and /.compiler
RUN rustup component add rustfmt
RUN rustup target add $(cat /.platform) 
RUN apt-get update && apt-get install -y unzip $(cat /.compiler)

COPY Cargo.toml .
COPY Cargo.lock .
COPY .cargo/config .cargo/config

COPY lib lib
COPY src src

# Build protobuf first for generated code
RUN cd lib/protobuf && cargo build --release --target $(cat /.platform)
RUN cargo build --release --target $(cat /.platform)
RUN cp /usr/src/trow/target/$(cat /.platform)/release/trow /usr/src/trow/ # Get rid of this when build --out is stable

FROM debian:stable-slim
RUN groupadd -r -g 333333 trow && useradd -r -g trow -u 333333 trow

# Note that certs are needed for proxying
RUN apt-get update \
  && apt-get install -y --no-install-recommends openssl libssl-dev ca-certificates\
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

COPY quick-install/self-cert /install/self-cert
COPY start-trow.sh /
RUN mkdir --parents /data/layers && mkdir /data/scratch && mkdir /certs
# keep this late for concurrency
COPY --from=cross /usr/src/trow/trow /trow 
RUN chown -R trow /data /certs /install
USER trow
ENTRYPOINT ["/start-trow.sh"]

ARG VCS_REF
ARG VCS_BRANCH
ARG DATE
ARG VERSION
ARG REPO
ARG TAG
ENV CREATED=$DATE
ENV VCS_REF=$VCS_REF
ENV VCS_BRANCH=$VCS_BRANCH
ENV VERSION=$VERSION

LABEL org.opencontainers.image.created=$DATE \
  org.opencontainers.image.authors="Container Solutions Labs" \
  org.opencontainers.image.url="https://trow.io" \
  org.opencontainers.image.source="https://github.com/ContainerSolutions/trow" \
  org.opencontainers.image.version=$VERSION \
  org.opencontainers.image.revision=$VCS_REF \
  git.branch=$VCS_BRANCH \
  org.opencontainers.image.title="Trow Cluster Registry" \
  repository=$REPO \
  tag=$TAG

Using Buildx builders

To figure out what’s going on, it’s easiest to start at the end and work backwards. We can run the following to use the Dockerfile:

docker buildx build -f Dockerfile ../

Note that we are using buildx, a CLI plugin for Docker which enables usage of advanced buildkit features from Docker. In this blog I’ve used the full version of buildkit from the GitHub repository rather than the inbuilt Docker version (typically enabled with DOCKER_BUILDKIT=1).

We can also use the --platform argument e.g:

docker buildx build --platform linux/arm/v7 -f Dockerfile ../

Assuming you have a builder instance that supports the target platform Docker will happily build the appropriate image. Most of us don’t have a build cluster to target, but we can use Docker’s QEMU support to create a build instance that can emulate the target architecture. It is also possible to specify multiple platforms, which will be important later.

You can see what platforms you can build for by running docker buildx ls e.g:

NAME/NODE DRIVER/ENDPOINT STATUS PLATFORMS
default    docker 
default    default        running linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

If the specified platform isn’t the same as the host platform, buildx can use QEMU to emulate the target platform (you can also specify separate build instances for different platforms).

Cross-Compilation to the rescue

But as cool as this is, there’s a major problem. It’s slow. Not just go-for-a-coffee slow, but maybe-get-a-three-course-meal-actually-just-come-back-tomorrow slow if you’re doing something CPU intensive like compilation. Thankfully, there is a workaround - a lot of compilers, including those for Go and Rust, will allow you to do cross-compilation i.e. performing the compilation step on the host architecture but targeting a different one. In our Dockerfile, that’s exactly what’s happening in this line:

 RUN cargo build --release --target $(cat /.platform)

The exact platform (e.g. gcc-arm-linux-gnueabihf) is specified in the contents of the .platform file.

Having got that clear, let’s go back to the, err, end. If we look at the second FROM command in the Dockerfile, you will see it’s pretty standard:

FROM debian:stable-slim

There is no platform specified, so it will use the same platform as specified in the buildx build command. If this platform is different from the host, this is going to use QEMU, but that’s ok as the only RUN commands are trivial shell operations (mkdir and chown) that won’t stress the CPU.

Compare this to the first FROM line:

FROM --platform=$BUILDPLATFORM rust:latest as cross

Here we say that the “cross” build image will be built on BUILDPLATFORM, a variable that buildx sets to the host build platform (so for my laptop, that’s amd64), regardless of the platform specified in the build command. The idea being that in this non-emulated build stage we do our CPU expensive compilation and then copy the result into the second stage which builds the final image.

The slightly tricky bit

But how do we tell the compiler what platform to build for? This is where we need to be a little clever. Take a look at these lines:

COPY docker/platform.sh .
RUN ./platform.sh # should write /.platform and /.compiler

Here we run a short script which populates the files /.platform and /.compiler with values derived from the TARGETARCH variable set by buildx. The script is actually pretty simple:

#!/bin/bash

# Used in Docker build to set platform dependent variables

case $TARGETARCH in

    "amd64")
	echo "x86_64-unknown-linux-gnu" > /.platform
	echo "" > /.compiler 
	;;
    "arm64") 
	echo "aarch64-unknown-linux-gnu" > /.platform
	echo "gcc-aarch64-linux-gnu" > /.compiler
	;;
    "arm")
	echo "armv7-unknown-linux-gnueabihf" > /.platform
	echo "gcc-arm-linux-gnueabihf" > /.compiler
	;;
esac

We can then refer to these values in RUN steps via:

RUN rustup target add $(cat /.platform) 
RUN apt-get update && apt-get install -y unzip $(cat /.compiler)

RUN cargo build --release --target $(cat /.platform)

One gotcha is that you have to declare an ARG in the Dockerfile to expose the TARGETARCH environment variable e.g:

ARG TARGETARCH

Note that the “cat” is needed as we’re using sh, not bash. In a bash script we could instead use $(< /.compiler).

But that’s about it—the rest of the Dockerfile is just setting labels and copying files about.

The really great thing about this is we can now build for multiple architectures at once e.g:

docker buildx build --platform linux/arm/v7,linux/arm64,linux/amd64 \
    -t myrepo/multi-image \
    --push \
     -f Dockerfile ../

This will build our Dockerfile for all the above platforms, create a manifest list and push the images up to myrepo/multi-image on the Docker Hub. Pulling that tag will download the correct image for your current architecture. (If you’re curious about how manifest lists work, try using the imagetools command in buildx to take a look at a remote manifest e.g. docker buildx imagetools inspect containersol/trow:latest).

This solution saved a lot of previous trouble; using the buildx variables means we now only need one Dockerfile and I don’t have to do any manual work to stitch together the manifest list since, as it is one Dockerfile, buildx takes care of it.

Taking it to the next level

If you like this approach, take a look at the xx project by Tõnis Tiigi, who is also the main developer behind buildx. xx provides both a Docker image and helper scripts to abstract a lot of the fiddly details away e.g. you can use xx-apt-get to install the appropriate versions of libraries without having to figure out the right names for different platforms.

Hopefully this helps some of you create your first multi-arch builds and some others to simplify existing builds. Let us know if you found this post useful!