Skip to content

Docker Builds on GKE

Docker Builds on Kubernetes

Besides the ability to build docker images using a dedicated docker_builder task which runs on VMs, it is also possible to run docker builds on Kubernetes. To do so we are leveraging the additional_containers and docker-in-docker functionality.

Currently cirrus-ci supports running builds on following Kubernetes distributions:

  • Google Kubernetes Engine (GKE)
  • AWS Elastic Kubernetes Service (EKS)

For Generic Kubernetes Support follow this issue.

Comparison of docker builds on VMs vs Kubernetes

  • VMs
    • complex builds are potentially faster than docker-in-docker
    • safer due to better isolation between builds
  • Kubernetes
    • much faster start - creating a new container usually takes few seconds vs creating a VM which takes usually about a minute on GCP and even longer on AWS.
    • ability to use an image with your custom tools image (e.g. containing Skaffold) to invoke docker instead of using a fixed VM image.

How to

This a full example of how to build a docker image on GKE using docker and pushing it to GCR. While not required, the script section in this example also has some best practice cache optimizations and pushes the image to GCR.

AWS EKS support

While the steps below are specifically written for and tested with GKE (Google Kubernetes Engine), it should work equally on AWS EKS.

docker_build_task:
  gke_container: # for AWS, replace this with `aks_container`
    image: docker:latest # This image can be any custom image. The only hard requirement is that it needs to have `docker-cli` installed.
    cluster_name: cirrus-ci-cluster # your gke cluster name
    zone: us-central1-b # zone of the cluster
    namespace: cirrus-ci # namespace to use
    cpu: 1
    memory: 1500Mb
    additional_containers:
      - name: dockerdaemon
        privileged: true # docker-in-docker needs to run in privileged mode
        cpu: 4
        memory: 3500Mb
        image: docker:dind
        port: 2375
        env:
          DOCKER_DRIVER: overlay2 # this speeds up the build
  env:
    DOCKER_HOST: tcp://localhost:2375 # this is required so that docker cli commands connect to the "additional container" instead of `docker.sock`.
    GOOGLE_CREDENTIALS: ENCRYPTED[qwerty239abc] # this should contain the json key for a gcp service account with the `roles/storage.admin` role on the `artifacts.<your_gcp_project>.appspot.com` bucket as described here https://cloud.google.com/container-registry/docs/access-control. This is only required if you want to pull / push to gcr. If we use dockerhub you need to use different credentials.
  login_script:
    echo $GOOGLE_CREDENTIALS | docker login -u _json_key --password-stdin https://gcr.io
  build_script:
    - docker pull gcr.io/my-project/my-app:$CIRRUS_LAST_GREEN_CHANGE || true
    - docker build
      --cache-from=gcr.io/my-project/my-app:$CIRRUS_LAST_GREEN_CHANGE
      -t gcr.io/my-project/my-app:$CIRRUS_CHANGE_IN_REPO 
      .   
  push_script:
    - docker push gcr.io/my-project/my-app:$CIRRUS_CHANGE_IN_REPO 

Caveats

Since the additional_container needs to run in privileged mode, the isolation between the docker build and the host are somewhat limited, hence you should create a separate cluster for cirrus-ci builds ideally. If this a concern you can also try out Kaniko or Makisu to run builds in unprivileged containers.