Introduction⌗

Previously, we used Intel chip Macs for development, but later we all switched to Apple M1 series MacBook Pro. This led to an issue: images built on the Intel platform would show compatibility warnings on Apple M1 series computers.

Additionally, if we modified the Dockerfile, such as upgrading the PHP Runtime, images built on ARM chips would not run on the linux/amd64 platform.

After research, we discovered that the Docker team had been working on moby/buildkit for quite some time.

Drivers⌗

Docker buildx drivers include:

  • docker: The built-in Builder in Docker Engine, which doesn’t support building multi-platform images;
  • docker-container: Uses BuildKit Container for building multi-platform images;
  • kubernetes: Schedules BuildKit Container to run on Kubernetes Pods for building multi-platform images;
  • remote: Deploy buildkitd service on a remote server, then use buildx to create a remote builder instance.

Specific differences are as follows:

Featuredockerdocker-containerkubernetesremote
Automatically load image✅
Cache exportInline only✅✅✅
Tarball output✅✅✅
Multi-arch images✅✅✅
BuildKit configuration✅✅Managed externally

Requirements⌗

  • Docker Desktop: latest
  • Docker buildx: latest

buildx is a plugin for the docker CLI, similar to the compose plugin, used to call BuildKit’s build functionality.

Start Building Multi-Platform Images⌗

There are basically two approaches to building multi-platform images:

  • Using QEMU on ARM chips to emulate linux/amd64
  • Using BuildKit Container on remote servers to achieve native platform compilation

Of these two approaches, the first is the simplest, but compiling linux/amd64 images on ARM is much slower! The second method distributes different platform image build tasks to different remote servers for native building, which has an absolute advantage in speed, although the configuration is relatively more complex.

Using QEMU Emulation Locally⌗

The advantage of this approach is that if you don’t have a linux/amd64 architecture remote server, you can use QEMU for emulation locally. The disadvantage is that it’s considerably slower (5-10 times slower during the build process).

docker buildx create --name local-builder --driver docker-container --bootstrap
[+] Building 18.2s (1/1) FINISHED
 => [internal] booting buildkit                                     18.2s
 => => pulling image moby/buildkit:buildx-stable-1                  17.6s
 => => creating container buildx_buildkit_local-builder0            0.6s
local-builder

docker buildx ls
NAME/NODE        DRIVER/ENDPOINT             STATUS  BUILDKIT PLATFORMS
local-builder    docker-container
  local-builder0 unix:///var/run/docker.sock running v0.10.5  linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
default *        docker
  default        default                     running 20.10.17 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
desktop-linux    docker
  desktop-linux  desktop-linux               running 20.10.17 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

As you can see, a builder instance named local-builder has been created, which runs a Docker Container locally:

docker ps
CONTAINER ID   IMAGE                           COMMAND                  CREATED         STATUS         PORTS                        NAMES
999b76900e61   moby/buildkit:buildx-stable-1   "buildkitd"              5 minutes ago   Up 5 minutes                                buildx_buildkit_local-builder0

Use the created Builder instance to build images:

docker buildx build --builder local-builder --platform linux/amd64,linux/arm64 --tag betterde/php:8.1-fpm --push .
[+] Building 20.4s (4/16)
 => [linux/arm64 1/6] FROM docker.io/library/php:8.1-fpm@sha256:66f22e43c5b2546cdc953d8fdd6ce994925207f3c3c958f59da00519afc8c548                                                     16.3s
 => => sha256:d4cbe7e5b3a1e3bf8dd36ca0f6c0975474c90c6a7002db1eab83ee683cdbf530 245B / 245B                                                                                            2.3s
 => => sha256:51922d488e38ac090f75c9f825d4030e42811615d2727509ff0e0bda221e15f9 2.45kB / 2.45kB                                                                                        2.3s
 => => sha256:db98555fb860484aa5b7e93135449c9efa6513b035bed0ec52ff312dbd09aa46 26.05MB / 26.05MB                                                                                      5.3s
 => => sha256:0643c2113c832fd423cbebe2ee0c779f249e90bf494e32e09fe7edc7e762d666 492B / 492B                                                                                            0.5s
 => => sha256:677ee647cc49e1ae8e670f191f4eac1ed4c2fb0f22d6a19ecef70f6469f199b6 11.90MB / 11.90MB                                                                                      4.8s
 => => sha256:632ec72a2e3295f16e0814b1568d1498d4940618e1cdf95dd056118af372eee8 224B / 224B                                                                                            0.5s
 => => sha256:abafaf8826eab2c82e152b7058ed06211207f6c7027c37225e1f1c19ebba793d 31.46MB / 86.72MB                                                                                     13.5s
 => => sha256:df8e44b0463f16c791d040e02e9c3ef8ec2a84245d365f088a80a22a455c71e8 27.26MB / 30.06MB                                                                                     13.4s
 => => sha256:2a21100b08aef7a04fe5e61ba611f278fa0bfe87fd02bbf28823b0df1dcea3b1 225B / 225B                                                                                            0.4s
 => [linux/amd64 1/6] FROM docker.io/library/php:8.1-fpm@sha256:66f22e43c5b2546cdc953d8fdd6ce994925207f3c3c958f59da00519afc8c548                                                     16.3s
 => => resolve docker.io/library/php:8.1-fpm@sha256:66f22e43c5b2546cdc953d8fdd6ce994925207f3c3c958f59da00519afc8c548                                                                  0.0s
 => => sha256:06ce6b583d54b66d9e12dd7504fd42abf3d3d8af02d5d63ab9ac34436e6e6081 8.62kB / 8.62kB                                                                                        0.4s
 => => sha256:454617d5c6c725ec1d97e379553019fb6fa4af02cf400ad221101790f69b7888 247B / 247B                                                                                            0.4s
 => => sha256:9e1d78cd066abba93a02e92179987869c2019532b221af8174fbde18e154cacb 2.45kB / 2.45kB                                                                                        0.4s
 => => sha256:c3fb5aa0621191d99a0819209f8b4d7a57d94c28aefaf1dd628d9111d053d650 26.22MB / 26.22MB                                                                                      7.0s
 => => sha256:a18ae08838ec8895e844b9547716fa8a9ff4ab5bee1747edb7d8669e183d3242 494B / 494B                                                                                            0.5s
 => => sha256:77f71a584e447d63c4666a2b48a6f88599ce646645ce553d76b1a4c66566fcba 12.12MB / 12.12MB                                                                                      3.5s
 => => sha256:4220e0c033772bd27ff577c6cc30afe1cd81296bf0d58615a56a8923c07c655a 271B / 271B                                                                                            0.4s
 => => sha256:e7793be89e9cdecbe7f44d7cdb803c08522f057e4fbdd2b0cc72019e378bb660 11.53MB / 91.63MB                                                                                      4.7s
 => => sha256:1e83b070fd9716453dd96b679a05d9d11c1f66d95582736a8d809d73a3f70c0d 227B / 227B                                                                                            0.4s
 => => sha256:bd159e379b3b1bc0134341e4ffdeab5f966ec422ae04818bb69ecef08a823b05 3.15MB / 31.42MB

You can see that images for both platforms are being built simultaneously, but the linux/amd64 architecture image is much slower than the linux/arm64 one.

After everything is built, you can use the following command to check the built image information:

docker buildx imagetools inspect betterde/php:8.1-fpm
Name:      docker.io/betterde/php:8.1-fpm
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:341d1a09197003b9e4f16abe46aa1b700d54d2b60c4514f760bcaa33ce849dc9

Manifests:
  Name:      docker.io/betterde/php:8.1-fpm@sha256:3433c5247713f44a57bceddb7d463c01783eaf0694f22884887f7b245ae64fb0
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64

  Name:      docker.io/betterde/php:8.1-fpm@sha256:1e34a95b96eb31bb32089f6435176f701b54d79b3f7108162674cee6f0bdf314
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64

You can see that the image now supports both linux/amd64 and linux/arm64 platforms.

Docker Hub

Building on Remote Servers⌗

Building on remote servers can be broadly divided into three approaches:

  • Running Buildkit Container on a Kubernetes cluster using QEMU emulation;
  • Using Buildx CLI with the docker-container driver to connect to remote Docker via SSH or TCP Socket to automatically create Buildkit Container instances;
  • Pre-deploying Buildkit Container (container or host), exposing TCP port 1234, and then connecting using Buildx CLI’s remote driver.

I’ll skip the Kubernetes-related multi-platform build here and mainly analyze the differences between the latter two approaches.

Using docker-container driver⌗

docker buildx create --name builder \
  --node linux-amd64 \
  --driver docker-container \
  --platform  linux/amd64,linux/386 \
  --bootstrap \
  ssh://USER@HOST
[+] Building 17.7s (1/1) FINISHED
 => [internal] booting buildkit                                17.3s
 => => pulling image moby/buildkit:buildx-stable-1             15.7s
 => => creating container buildx_buildkit_staging

After this, log into the remote server to check the container running status:

docker ps | grep buildx
75001bb07b22   moby/buildkit:buildx-stable-1    "buildkitd"    1 minutes ago   Up 1 minutes    buildx_buildkit_staging

Using remote driver⌗

Run the following command on the remote server to create a Buildkit Container:

docker run -d --rm \
  --name=remote-buildkitd \
  --privileged \
  -p 1234:1234 \
  -v /root/smallstep:/etc/buildkit/certs \
  moby/buildkit:latest \
  --addr tcp://0.0.0.0:1234 \
  --tlscacert /etc/buildkit/certs/daemon/root_ca.crt \
  --tlscert /etc/buildkit/certs/daemon/docker-daemon.crt \
  --tlskey /etc/buildkit/certs/daemon/docker-daemon.key

docker ps | grep remote-buildkitd
8ea4c52b97a8   moby/buildkit:latest   "buildkitd --addr tc…"   1 minutes ago   Up 1 minutes   0.0.0.0:1234->1234/tcp, :::1234->1234/tcp        remote-buildkitd

Use Buildx CLI to create a remote instance:

docker buildx create \
  --use \
  --name builder \
  --node linux-amd64 \
  --platform linux/amd64,linux/386 \
  --driver remote \
  --driver-opt key=/Users/George/Desktop/smallstep/docker-client.key \
  --driver-opt cert=/Users/George/Desktop/smallstep/docker-client.crt \
  --driver-opt cacert=/Users/George/Desktop/smallstep/root_ca.crt \
  tcp://HOST:1234

docker buildx ls
NAME/NODE     DRIVER/ENDPOINT           STATUS  BUILDKIT PLATFORMS
builder *       remote
  linux-amd64   tcp://47.103.204.136:1234 running          linux/amd64*, linux/386*
default         docker
  default       default                   running 20.10.21 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
desktop-linux   docker
  desktop-linux desktop-linux             running 20.10.21 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

Although docker buildx create [OPTIONS] [CONTEXT|ENDPOINT] supports Docker Context, CONTEXT is only applicable for docker and docker-container drivers!

Building Multi-Platform Images⌗

Before building, we also need to add a linux/arm64 Builder on ARM macOS:

docker run -d \
  --name=remote-buildkitd \
  --privileged \
  -p 1234:1234 \
  -v /YOURPATH/certs:/etc/buildkit/certs \
  moby/buildkit:latest \
  --addr tcp://0.0.0.0:1234 \
  --tlscacert /etc/buildkit/certs/root_ca.crt \
  --tlscert /etc/buildkit/certs/docker-daemon.crt \
  --tlskey /etc/buildkit/certs/docker-daemon.key

docker buildx create \
  --append \
  --name builder \
  --node linux-arm64 \
  --platform linux/arm64 \
  --driver remote \
  --driver-opt key=/YOURPATH/docker-client.key \
  --driver-opt cert=/YOURPATH/docker-client.crt \
  --driver-opt cacert=/YOURPATH/root_ca.crt \
  tcp://127.0.0.1:1234

docker buildx ls
builder         remote
  linux-amd64   tcp://47.103.204.136:1234 running          linux/amd64, linux/386
  linux-arm64   tcp://127.0.0.1:1234      running v0.11.1  linux/arm64*
default *       docker
  default       default                   running 20.10.21 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
desktop-linux   docker
  desktop-linux desktop-linux             running 20.10.21 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

docker buildx build --builder builder --platform linux/amd64,linux/arm64 --tag betterde/php:8.2-fpm --push .

Other Solutions⌗

Depot is a cloud build platform that provides native ARM build environments and also supports self-hosted build nodes.

I hope this is helpful, Happy hacking…