↓Skip to main content
IT Blog IT Blog
  1. All Posts/

k8s-deploy-helper

Tools Shell

Description

k8s-deploy-helper (KDH) is a tool to help build and deploy containerized applications into Kubernetes using GitLab CI along with templated manifest files. Major features include:

  • Automated Kubernetes Secret Management using GitLab’s UI. Version 3.0.0 can automatically insert secrets into Kubernetes manifests and Dockerfiles to lessen manual work.
  • Build via Heroku buildpacks or via Dockerfiles starting in 3.1.0.
  • Automated canary deployments. Dynamic creation is included in version 3.0.0.
  • Automated review app deployments.
  • Automated deployment of applications using Heroku Procfile in 3.1.0.
  • Deployment notifications to New Relic, Datadog and Slack.
  • Templated manifest deployments to Kubernetes living in the same repo as the code, giving developers more control.
  • Uses kubeval to evaluate manifest yaml before any are deployed.
  • Easy, standardized image creation with build arguments and multiple Dockerfile support.
  • Standardized building conventions to allow for easy rollbacks through GitLab UI and better caching.

This project is not endorsed or affiliated with GitLab in any way.

Examples

In addition to this documentation, the best way to get started is to look at our example repository.
Need some help getting started? Feel free to join us on Open Digerati Slack in #k8s and we’ll be more than happy to assist.

Why?

GitLab’s Auto DevOps initiative is amazing for getting simple apps running quickly, but for slightly more complex and production workloads, you need more control in the process. For instance, what if you have a pod with sidecar containers? What if you want a deployment of worker pods using something like celery for async work? You’ll need to interact with Kubernetes at a deeper level to do stuff like this, and that’s where our project comes in.
At Life.Church, we wanted to create a standardized tool along with corresponding conventions that our developers could use with GitLab CI to allow us to get up and running with manifest-based deployments as quickly and easily as possible. So, we took the work that GitLab started, and used it as the base of a new project that would meet our needs.
This tool was built akin to an airplane that was built while we were flying it. Our process is constantly maturing as we learn more about how to deploy into k8s. Our goal isn’t to say ‘this is the best way of deploying’, but simply to share how we’re doing it now, knowing that it will at least be a helpful starting place for others who are embarking on their Kubernetes journey.

Prerequisites

  • GitLab w/customizable runners
  • Kubernetes

Configuring GitLab Runner

There is a lot of discussion around the best way to build docker images from within Docker. We ended up going the route of sharing the Docker socket. Here is a sample GitLab Runner configuration. Of particular note, is the volumes section, to share the socket the way we expect. Additionally, we use dind for some stages, so privileged needs to be turned on as well.

[[runners]]
  name = "runner01"
  limit = 16
  url = "https://runneriurl/ci"
  token = "token"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = true
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    shm_size = 0
  [runners.cache]

Integrate Kubernetes into your Project

In your GitLab Project, go to Operations->Kubernetes to give GitLab the ability to talk to your Kubernetes cluster. See GitLab’s documentation on how to do this properly.

GitLab Credentials

GitLab has finally introduced a way to have persistent deploy tokens that can fetch things from the Docker registry. k8s-deploy-helper 3.0 now uses this more secure token. You can create a deploy token at Settings->Repository->Deploy Tokens and make one named gitlab-deploy-token with read_registry access. It HAS to be called gitlab-deploy-token due to GitLab limitations. Once this token is created, k8s-deploy-helper will pick it up automatically.

Building Docker Images

Our goal was to make sure Docker containers could be built as quickly as possible without the developers having to micromanage each docker build command on a per-project basis.
Here is a quick example from the .gitlab-ci.yml:

build_container:
  stage: build
  script:
    - command build
  only:
    - branches

Notice the script only has one command: command build – k8s-deploy-helper takes it from there, building the container, tagging it with a unique id (commit hash), and pushing it into the GitLab docker registry.

Caching Docker FS Layers

By default, k8s-deploy-helper and the accompanying examples use a convention where the build that is deployed to production is given the latest tag after successful deployment. When building Docker containers, we use –cache-from your docker image’s latest tag. This will allow for more optimized caching when using multiple GitLab runners or runners without persistent storage.
If you’re managing your own runners, and you only have one, then you may want to think about setting a variable named KDH_SKIP_LATEST to true in your build stages or in the GitLab variables UI. When k8s-deploy-helper finds this variable set to true, we don’t use –cache-from, and will just build the Docker container normally, which will try and make use of the cache that is already present in the runner.

Build Arguments

Sometimes you need to pass in arguments to containers at build time to do things like putting a token in place to pull from a private npm registry. To pass in build arguments, simply go to your GitLab project and go to Settings->CI/CD->Variables and create a variable in this form:
BUILDARG_npmtoken=1111
When we build the Docker container, we look for all environment variables that start with BUILDARG_, strip that prefix out, and pass it into docker via –build-arg. In the example above, this will create a build argument npmtoken with a value of 1111
In the example above, you would have to put the following line into your Dockerfile in order to use it at build time:
ARG npmtoken

Automatic ARG Insertion

Starting in 3.0, you can set a variable named KDH_INSERT_ARGS to true, and k8s-deploy-helper will automatically take all your build arguments and insert corresponding ARG commands into your Dockerfile at build time, immediately after FROM lines. This makes GitLab the source of truth and you as a developer will no longer have to worry about setting things in multiple places.
Inserting things into your Dockerfile at runtime is definitely considered a bit magic though, so we make this a feature you have to opt into.

Secrets as Buildargs

Sometimes you may have an application that has logic that looks to make sure all environment variables are present before you can run any command (like asset generation). k8s-deploy-helper has the ability to automatically create secrets and use them in your Kubernetes manifests, which we will go into later on. If you set the variable BUILDARGS_FROM to production, it will take all the secrets that would be created to run in the production stage and automatically use them as build arguments when creating your Docker container. This will also turn on the KDH_INSERT_ARGS feature and will insert ARG statements into your Dockerfile automatically.

Build Multiple Dockerfiles

If your project needs to build multiple Dockerfiles, the helper will automatically handle all the naming convention management to avoid collisions. All you need to do is pass in the file name of the Dockerfile that is in the root of your repository. For example, if you have two Dockerfiles, Dockerfile-app, and Dockerfile-worker, this is what your .gitlab-ci.yml would look like:

build_app:
  stage: build
  script:
    - command build Dockerfile-app
  only:
    - branches

build_worker:
  stage: build
  script:
    - command build Dockerfile-worker
  only:
    - branches

Buildpack Builds

Starting in 3.1.0, KDH can build applications using Heroku Buildpacks via herokuish. To do…



↑