While doing some work on Trow, our image management solution for Kubernetes, we discovered how easy new tooling makes it to produce container images for multiple architectures. This post will show how with just a little work we were able to create Trow images for both ARM and Intel platforms.
Docker has for some time supported multiplatform images where an image can have multiple different versions for different architectures and platforms. Pulling a multiplatform image will pull only the version for your current architecture. For example,
docker pull debian:latest will pull the
amd64 image on my laptop and the
armv7 image on my Raspberry Pi. The way this works is through the use of manifest lists aka “fat manifests” , which contain pointers to images for different architectures.
The Trow project has had multiplatform builds for some time, but the code to do it was frustratingly complex and dependent on Phil Estes’ manifest-tool. With recent updates to buildx, the GitHub Container Registry now supporting manifest lists, and some inspiration from Tõnis Tiigi’s xx project, I realised it was possible to simplify things a lot. It’s taken a bit of work but we now have a new build for Trow that only requires a single Dockerfile:
# syntax=docker/dockerfile:1 #Note we build on host platform and cross-compile to target arch FROM --platform=$BUILDPLATFORM rust:latest as cross ARG TARGETARCH WORKDIR /usr/src/trow COPY docker/platform.sh . RUN ./platform.sh # should write /.platform and /.compiler RUN rustup component add rustfmt RUN rustup target add $(cat /.platform) RUN apt-get update && apt-get install -y unzip $(cat /.compiler) COPY Cargo.toml . COPY Cargo.lock . COPY .cargo/config .cargo/config COPY lib lib COPY src src # Build protobuf first for generated code RUN cd lib/protobuf && cargo build --release --target $(cat /.platform) RUN cargo build --release --target $(cat /.platform) RUN cp /usr/src/trow/target/$(cat /.platform)/release/trow /usr/src/trow/ # Get rid of this when build --out is stable FROM debian:stable-slim RUN groupadd -r -g 333333 trow && useradd -r -g trow -u 333333 trow # Note that certs are needed for proxying RUN apt-get update \ && apt-get install -y --no-install-recommends openssl libssl-dev ca-certificates\ && apt-get clean \ && rm -rf /var/lib/apt/lists/* COPY quick-install/self-cert /install/self-cert COPY start-trow.sh / RUN mkdir --parents /data/layers && mkdir /data/scratch && mkdir /certs # keep this late for concurrency COPY --from=cross /usr/src/trow/trow /trow RUN chown -R trow /data /certs /install USER trow ENTRYPOINT ["/start-trow.sh"] ARG VCS_REF ARG VCS_BRANCH ARG DATE ARG VERSION ARG REPO ARG TAG ENV CREATED=$DATE ENV VCS_REF=$VCS_REF ENV VCS_BRANCH=$VCS_BRANCH ENV VERSION=$VERSION LABEL org.opencontainers.image.created=$DATE \ org.opencontainers.image.authors="Container Solutions Labs" \ org.opencontainers.image.url="https://trow.io" \ org.opencontainers.image.source="https://github.com/ContainerSolutions/trow" \ org.opencontainers.image.version=$VERSION \ org.opencontainers.image.revision=$VCS_REF \ git.branch=$VCS_BRANCH \ org.opencontainers.image.title="Trow Cluster Registry" \ repository=$REPO \ tag=$TAG
Using Buildx builders
To figure out what’s going on, it’s easiest to start at the end and work backwards. We can run the following to use the Dockerfile:
docker buildx build -f Dockerfile ../
Note that we are using buildx, a CLI plugin for Docker which enables usage of advanced buildkit features from Docker. In this blog I’ve used the full version of buildkit from the GitHub repository rather than the inbuilt Docker version (typically enabled with
We can also use the
--platform argument e.g:
docker buildx build --platform linux/arm/v7 -f Dockerfile ../
Assuming you have a builder instance that supports the target platform Docker will happily build the appropriate image. Most of us don’t have a build cluster to target, but we can use Docker’s QEMU support to create a build instance that can emulate the target architecture. It is also possible to specify multiple platforms, which will be important later.
You can see what platforms you can build for by running
docker buildx ls e.g:
NAME/NODE DRIVER/ENDPOINT STATUS PLATFORMS default docker default default running linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
If the specified platform isn’t the same as the host platform, buildx can use QEMU to emulate the target platform (you can also specify separate build instances for different platforms).
Cross-Compilation to the rescue
But as cool as this is, there’s a major problem. It’s slow. Not just go-for-a-coffee slow, but maybe-get-a-three-course-meal-actually-just-come-back-tomorrow slow if you’re doing something CPU intensive like compilation. Thankfully, there is a workaround - a lot of compilers, including those for Go and Rust, will allow you to do cross-compilation i.e. performing the compilation step on the host architecture but targeting a different one. In our Dockerfile, that’s exactly what’s happening in this line:
RUN cargo build --release --target $(cat /.platform)
The exact platform (e.g.
gcc-arm-linux-gnueabihf) is specified in the contents of the
Having got that clear, let’s go back to the, err, end. If we look at the second FROM command in the Dockerfile, you will see it’s pretty standard:
There is no platform specified, so it will use the same platform as specified in the buildx build command. If this platform is different from the host, this is going to use QEMU, but that’s ok as the only RUN commands are trivial shell operations (mkdir and chown) that won’t stress the CPU.
Compare this to the first FROM line:
FROM --platform=$BUILDPLATFORM rust:latest as cross
Here we say that the “cross” build image will be built on BUILDPLATFORM, a variable that buildx sets to the host build platform (so for my laptop, that’s amd64), regardless of the platform specified in the build command. The idea being that in this non-emulated build stage we do our CPU expensive compilation and then copy the result into the second stage which builds the final image.
The slightly tricky bit
But how do we tell the compiler what platform to build for? This is where we need to be a little clever. Take a look at these lines:
COPY docker/platform.sh . RUN ./platform.sh # should write /.platform and /.compiler
Here we run a short script which populates the files
/.compiler with values derived from the TARGETARCH variable set by buildx. The script is actually pretty simple:
#!/bin/bash # Used in Docker build to set platform dependent variables case $TARGETARCH in "amd64") echo "x86_64-unknown-linux-gnu" > /.platform echo "" > /.compiler ;; "arm64") echo "aarch64-unknown-linux-gnu" > /.platform echo "gcc-aarch64-linux-gnu" > /.compiler ;; "arm") echo "armv7-unknown-linux-gnueabihf" > /.platform echo "gcc-arm-linux-gnueabihf" > /.compiler ;; esac
We can then refer to these values in RUN steps via:
RUN rustup target add $(cat /.platform) RUN apt-get update && apt-get install -y unzip $(cat /.compiler)
RUN cargo build --release --target $(cat /.platform)
One gotcha is that you have to declare an ARG in the Dockerfile to expose the TARGETARCH environment variable e.g:
Note that the “cat” is needed as we’re using sh, not bash. In a bash script we could instead use
But that’s about it—the rest of the Dockerfile is just setting labels and copying files about.
The really great thing about this is we can now build for multiple architectures at once e.g:
docker buildx build --platform linux/arm/v7,linux/arm64,linux/amd64 \ -t myrepo/multi-image \ --push \ -f Dockerfile ../
This will build our Dockerfile for all the above platforms, create a manifest list and push the images up to
myrepo/multi-image on the Docker Hub. Pulling that tag will download the correct image for your current architecture. (If you’re curious about how manifest lists work, try using the
imagetools command in buildx to take a look at a remote manifest e.g.
docker buildx imagetools inspect containersol/trow:latest).
This solution saved a lot of previous trouble; using the buildx variables means we now only need one Dockerfile and I don’t have to do any manual work to stitch together the manifest list since, as it is one Dockerfile, buildx takes care of it.
Taking it to the next level
If you like this approach, take a look at the xx project by Tõnis Tiigi, who is also the main developer behind buildx. xx provides both a Docker image and helper scripts to abstract a lot of the fiddly details away e.g. you can use
xx-apt-get to install the appropriate versions of libraries without having to figure out the right names for different platforms.
Hopefully this helps some of you create your first multi-arch builds and some others to simplify existing builds. Let us know if you found this post useful!