Building Docker Images with bake

2024-12-03 4 min read Martin

Platform engineers and build pipeline engineers constantly are faced with the challenge of building container images for providing images to be used as application base or for running pipeline tasks. Probably such images have to be provided as multiple variants for multiple versions of software packages to be included.

The challenge is how to build such images efficiently in terms of build time and the required code and configuration of the pipeline.

Let’s say we need to provide a container image that includes a Java Development Kit (JDK) and Gradle for building Java based applications. We are working with the latest Java 21 LTS but also need to allow at least the next following Java release so application developers can start testing with a newer JDK. Also, we have teams that need multiple versions of Gradle. Let’s say we are using 8.10 and 8.10.2. So we have a matrix of software packages to package as container image.

It’s been a while since I published my post about ways to optimize a container image build. Since then the Docker tooling has received plenty of valuable improvements.

Today, let’s look into the options we have to implement the above required container images and then dive into our options to optimize for short build cycles.

The container image Dockerfile

The Dockerfile below packages a JDK with a Gradle distribution. Versions can be chosen based on variables we can pass into the build command.

Dockerfile

ARG JAVA_VERSION="21"
FROM eclipse-temurin:${JAVA_VERSION}-jdk-alpine

ARG CI_JOB_STARTED_AT
ARG CI_PROJECT_URL
ARG CI_COMMIT_REF_NAME
ARG CI_COMMIT_SHA

LABEL org.opencontainers.image.created=${CI_JOB_STARTED_AT}
LABEL org.opencontainers.image.source=${CI_PROJECT_URL}
LABEL org.opencontainers.image.version=${CI_COMMIT_REF_NAME}
LABEL org.opencontainers.image.revision=${CI_COMMIT_SHA}

WORKDIR /tmp/.cache

ARG GRADLE_VERSION="8.10.2"

RUN wget "https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip" && \
    mkdir -p /opt/gradle && \
    unzip -d /opt/gradle gradle-${GRADLE_VERSION}-bin.zip && \
    rm gradle-${GRADLE_VERSION}-bin.zip

Building the images

In order to build the needed images we can issue a sequence of commands and we will notice that the commands take a while even when we re-run the commands.

# Create a dedicated builder
docker buildx create --name container --driver docker-container --bootstrap --use
BUILD_OPTIONS="--builder=container"
docker image build --build-arg JAVA_VERSION=22 --build-arg GRADLE_VERSION="8.10" --tag "pipeline-sdk-17-gradle-8.10" ${BUILD_OPTIONS} .
docker image build --build-arg JAVA_VERSION=22 --build-arg GRADLE_VERSION="8.10.2" --tag "pipeline-sdk-17-gradle-8.10.2" ${BUILD_OPTIONS} .
docker image build --build-arg JAVA_VERSION=21 --build-arg GRADLE_VERSION="8.10" --tag "pipeline-sdk-21-gradle-8.10" ${BUILD_OPTIONS} .
docker image build --build-arg JAVA_VERSION=21 --build-arg GRADLE_VERSION="8.10.2" --tag "pipeline-sdk-21-gradle-8.10.2" ${BUILD_OPTIONS} .

Even though layers will be used from the cache, it takes a while since for each command the client has to connect to the Docker daemon and eventually upload the Docker context. So let’s look into how to eliminate the client-daemon interactions.

Enter Docker Bake

Bake is a feature of Docker Buildx that lets you define your build configuration using a declarative file, as opposed to specifying a complex CLI expression.

Using bake we can describe a set of images to be built using multiple formats (YAML, JSON, and HCL), we are using HCL here. The bake script below describes all images to be built and target the required software packages. In addition, we have also added another dimension, we are building multi-platform images to make things a bit more challenging.

group "all" {
  targets = [
    "java-21-gradle-8-10-2",
    "java-21-gradle-8-10",
    "java-22-gradle-8-10-2",
    "java-22-gradle-8-10"
  ]
}

target "java-22-gradle-8-10" {
  args = {
    JAVA_VERSION   = "22"
    GRADLE_VERSION = "8.10"
  }
  tags = ["pipeline-jdk:22-gradle-8-10"]
  platforms = ["linux/amd64", "linux/arm64"]
}

target "java-22-gradle-8-10-2" {
  args = {
    JAVA_VERSION   = "22"
    GRADLE_VERSION = "8.10.2"
  }
  tags = ["pipeline-jdk:22-gradle-8-10-2"]
  platforms = ["linux/amd64", "linux/arm64"]
}

target "java-21-gradle-8-10" {
  args = {
    JAVA_VERSION   = "21"
    GRADLE_VERSION = "8.10"
  }
  tags = ["pipeline-jdk:21-gradle-8-10"]
  platforms = ["linux/amd64", "linux/arm64"]
}

target "java-21-gradle-8-10-2" {
  args = {
    JAVA_VERSION   = "21"
    GRADLE_VERSION = "8.10.2"
  }
  tags = ["pipeline-jdk:21-gradle-8-10-2"]
  platforms = ["linux/amd64", "linux/arm64"]
}

# Create a dedicated builder
docker buildx create --name container --driver docker-container --bootstrap --use
docker buildx bake --builder container -f docker-bake.hcl

This time we will notice that first we have just one client daemon interaction but also that bake is trying to run image builds in parallel thus reducing the time to build.

We can further simplify the bake script. Obviously each image is identical, just the versions for the JDK image and the Gradle package are different. Looks like a matrix build operation typically offered by pipeline implementations.

target "all" {
  name = "pipeline-jdk-${java}-gradle-${replace(gradle, ".", "-")}"
  matrix = {
    java = [ "21", "22" ]
    gradle = [ "8.10.2", "8.10" ]
  }
  args = {
    JAVA_VERSION="${java}"
    GRADLE_VERSION="${gradle}"
  }
  platforms = ["linux/amd64", "linux/arm64"]
  tags = [ "pipeline-jdk:${java}-gradle-${replace(gradle, ".", "-")}"]
}

Bake is going to iterate the matrix variables and generate a target for each instance.

We’ve explored how Docker Bake streamlines and simplifies the container image build process, particularly for complex workflows involving multiple images. Bake files introduce powerful features such as inheritance, allowing you to define shared configurations in a base target and reuse or extend them across multiple targets. This approach minimizes redundancy, ensures consistency, and enhances maintainability.

For teams working with mono-repositories generating multiple container images, Docker Bake is especially beneficial. Its declarative syntax and parallel execution capabilities can significantly optimize build times, making it an invaluable tool for scaling CI/CD pipelines and managing large-scale containerized projects efficiently.