Post

Shipping Helm Charts to Air-Gapped Kubernetes Clusters

Deploying Kubernetes applications to air-gapped environments requires shipping both Helm charts and their Docker images. Let's see how we can package and deploy Helm charts to air-gapped Kubernetes clusters.

Posted Updated
Preview Image
By Bikramjeet Singh
8 min read
Shipping Helm Charts to Air-Gapped Kubernetes Clusters

Background

I was working on a security product that handles sensitive data for clients. The product was using microservice architecture. We selected Kubernetes for orchestrating the deployment because it made easier to scale up and down based on load. Kubernetes has many way to install an application. We selected Helm charts because it provided a template engine which was helpful in reducing the number of manifest files by using templates.

As we grew, we started to get bigger clients. The Clients wanted to install the application in an air gapped environment due to the nature of the data being processed. This means that we can no longer pull images from our artifactory and we have to come up with a different way to ship our docker images.

From my previous, I knew how to ship containers to an airgapped environment. But in this case I was suppose to ship a whole product which had mutliple dependencies and required complex configuration.

Requirements

Kubernetes is a distributed system which can schedule the pods/application containers to any available node and the node will pull the image from Internet. We needed a local version of docker hub since there was no option to pull the images from Internet.

Also, Since the application was being installed in customer’s environment we needed a rollback mechanism as there was no way we could fix any issue during the deployment.

Packaging Process

Package Docker Registry

  1. On a machine with internet access, pull and save the Docker registry image:

    1
2
3
4
5

     # Pull the official Docker registry image
 docker image pull registry:2

 # Save the image to a tar file
 docker image save registry:2 | gzip > "$PACKAGE_DIR/images/registry.tar.gz"

    Docker image save gives out the image in tar format which is an uncompressed archive. We are using gzip to compress the tar archive to reduce the size of the image.

  2. We need to have a Deployment manifest in order to deploy the registry on the cluster:

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

     # registry-deployment.yaml
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: docker-registry
   namespace: container-registry
 spec:
   replicas: 1
   selector:
     matchLabels:
       app: docker-registry
   template:
     metadata:
       labels:
         app: docker-registry
     spec:
       nodeSelector:
         primary-registry: "true"
       containers:
       - name: registry
         image: registry:2
         imagePullPolicy: IfNotPresent
         ports:
         - containerPort: 5000
         env:
         - name: REGISTRY_AUTH
           value: "htpasswd"
         - name: REGISTRY_AUTH_HTPASSWD_REALM
           value: "Registry Realm"
         - name: REGISTRY_AUTH_HTPASSWD_PATH
           value: "/auth/htpasswd"
         - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
           value: "/var/lib/registry"
         volumeMounts:
         - name: registry-storage
           mountPath: /var/lib/registry
         - name: registry-auth
           mountPath: /auth
           readOnly: true
       volumes:
       - name: registry-storage
         persistentVolumeClaim:
           claimName: registry-pvc
       - name: registry-auth
         secret:
           secretName: registry-auth

    We added auth to the registry so that only authenticated clients can upload/donwload the images. Here you can notice that we have added a nodeSelector to the deployment. This is to ensure that the registry will get deployed on the same node where we will import the registry image during installation.

  3. We need to have a persistent volume to store the images uploaded to the registry. To do this we created a PersistentVolumeClaim resource:

    1
2
3
4
5
6
7
8
9
10
11
12
13

     # registry-pvc.yaml
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: registry-pvc
   namespace: container-registry
 spec:
   accessModes:
     - ReadWriteMany
   resources:
     requests:
       storage: 20Gi
   storageClassName: nfs-client

    Customer was required to setup a Storage Class and provide the name of the storage class during installation.

  4. We need to expose the registry so that it can be accessed by our installer script to import the images. To expose the registry we created a Service and Ingress resource:

    1
2
3
4
5
6
7
8
9
10
11
12
13

     # registry-service.yaml
 apiVersion: v1
 kind: Service
 metadata:
   name: docker-registry
   namespace: container-registry
 spec:
   selector:
     app: docker-registry
   ports:
   - port: 5000
     targetPort: 5000
   type: ClusterIP

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

     # registry-ingress.yaml
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
   name: docker-registry
   namespace: container-registry
 spec:
   ingressClassName: nginx
   rules:
   - host: registry.airgapped.local
     http:
       paths:
       - path: /
         pathType: Prefix
         backend:
           service:
             name: docker-registry
             port:
               number: 5000

    Customer was required to setup a DNS record for the registry and provide the domain name during installation. Customer also provided the ingress class name to create the ingress.

Package the Helm Chart

We need to follow the following steps in order to package the images used in helm chart:

  1. Download the dependency charts for our helm chart.

    1

     helm dependency update "$CHART_PATH"

  2. Extract all the unique images being used in the helm chart.

    1
2

    IMAGES=$(helm template "$CHART_PATH" | grep -oP '(?<=image: ).*' | sort -u)
echo "$IMAGES" > "$PACKAGE_DIR/images/image-list.txt"

  3. Pull all the images from the docker registry.

    1
2
3

    for image in ${IMAGES}; do
     docker image pull "$image"
done

  4. Export all the images as archives files.

    1
2
3
4

    for image in ${IMAGES}; do
     IMAGE_NAME=$(echo "$image" | cut -d: -f1)
     docker save "$image" | gzip > "$PACKAGE_DIR/images/${IMAGE_NAME}.tar.gz"
done

  5. Create Helm package:

    1

     helm package "$CHART_PATH" --destination "$PACKAGE_DIR"

    We are not updating dependencies since we already downloaded them in first step

Creating Installer Script

We need to perform the following steps in order to install our application from all the resources we exported so far:

  1. Creating the Private registry:

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

     # Select a worker node
 WORKER_NODE=$(kubectl get nodes -l 'node-role.kubernetes.io/worker' -o json | jq -r '.items[0].metadata.labels."kubernetes.io/hostname"')

 # Copy the registry image to one of the worker nodes of Kubernetes
 scp "$PACKAGE_DIR/registry/registry.tar.gz" ${WORKER_NODE}:/tmp/
 ssh ${WORKER_NODE} "bash -c 'ctr image import /tmp/registry.tar.gz'"

 # Label the node as primary-registry so kubernetes will schedule the registry pod to this node
 kubectl label nodes ${WORKER_NODE} primary-registry=true

 # Create a username and password for the registry
 htpasswd -Bbn admin password > /tmp/htpasswd

 # Create the registry namespace and install registry resources to it
 kubectl create namespace container-registry
 kubectl create secret generic registry-auth --from-file=htpasswd=/tmp/htpasswd -n container-registry
 kubectl apply -f "$PACKAGE_DIR/registry/registry-pvc.yaml"
 kubectl apply -f "$PACKAGE_DIR/registry/registry-deployment.yaml"
 kubectl apply -f "$PACKAGE_DIR/registry/registry-service.yaml"
 kubectl apply -f "$PACKAGE_DIR/registry/registry-ingress.yaml"

    Customer was expected to have already setup ssh keys to access the worker nodes of the Kubernetes cluster.

  2. Import the registry image to the private registry:

    1
2
3
4
5
6
7
8
9
10
11

     # Login to the registry
 docker login registry.airgapped.local -u admin -p password

 # Load the registry image into the local docker daemon
 gunzip -c "$PACKAGE_DIR/registry/registry.tar.gz" | docker image load

 # Tag the registry image with new hostname
 docker tag registry:2 registry.airgapped.local/registry:2

 # Push to the registry
 docker push registry.airgapped.local/registry:2

  3. Update registry deployment to point to the private registry:

    1
2
3
4
5
6
7
8

     # Update the registry deployment to point to the private registry
 kubectl set image deployment/docker-registry registry=registry.airgapped.local/registry:2 -n container-registry

 # Update the registry deployment to remove the `nodeSelector`
 kubectl patch deployment docker-registry -n container-registry -p '{"spec":{"template":{"spec":{"nodeSelector":null}}}}'

 # Scale the registry deployment to 3 replicas
 kubectl scale deployment docker-registry --replicas=3 -n container-registry

  4. Importing the container images for helm charts to the private registry:

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

     while IFS= read -r IMAGE; do
   IMAGE_NAME=$(echo "$IMAGE" | tr '/:' '-')
   IMAGE_FILE="$PACKAGE_DIR/images/${IMAGE_NAME}.tar.gz"

   if [ -f "$IMAGE_FILE" ]; then
   echo "Loading image: $IMAGE"
   gunzip -c "$IMAGE_FILE" | docker load

   # Tag for local registry
   LOCAL_IMAGE="$REGISTRY_URL/${IMAGE#*/}"
   docker tag "$IMAGE" "$LOCAL_IMAGE"

   # Push to registry
   echo "Pushing to registry: $LOCAL_IMAGE"
   docker push "$LOCAL_IMAGE"
 fi
 done < "$PACKAGE_DIR/images/image-list.txt"

  5. Install the helm chart:

    1
2
3
4
5

     helm install "$CHART_NAME" "$CHART_FILE" \
 --namespace "$NAMESPACE" \
 --create-namespace \
 --set global.imageRegistry="$REGISTRY_URL" \
 --wait

  6. For Upgrading the helm chart:

    1
2
3
4
5
6
7
8
9
10

     if helm list -n "$NAMESPACE" | grep -q "$CHART_NAME"; then
   # Upgrade the chart
   sh db-backup.sh
   helm upgrade "$CHART_NAME" "$CHART_FILE" \
   --namespace "$NAMESPACE" \
   --set global.imageRegistry="$REGISTRY_URL" \
   --wait
 else
   # Install the chart
 fi

  7. For Uninstalling the helm chart:

    1

     helm uninstall "$CHART_NAME" -n "$NAMESPACE"

  8. For Rollback:

    1
2
3
4
5
6
7

     kubectl scale deployment -n "$NAMESPACE" – replicas 0 – all
    
 sh db-rollback.sh
    
 REVISION=$(helm history "$CHART_NAME" -n "$NAMESPACE" -o json | jq -r '.[-2].revision')

 helm rollback "$CHART_NAME" "$REVISION" -n "$NAMESPACE"

Conclusion

Distributing a Highly available application to air-gapped Kubernetes cluster requires you do have a private docker registry to store the images so that the Kubernetes is able to schedule the pods to any available node and the node is able to pull the image as it would have done in a non-airgapped cluster.

Helm charts make it easy to bundle the dependencies but you need to extract all the dependency images from it to distribute your application to air-gapped clusters.

For distributing an application to OnPrem Kubernetes clusters, you need to have a robust deployment/installation process that is able to install/upgrade the application and rollback the whole system if anything fails.

These days various Kubernetes distributions offer built-in support for air-gapped installations, making this process easier. For example, K3S provides a registry mirror, OpenShift also has an inbuilt registry.

blogs, tech
kubernetes helm docker airgapped
This post is licensed under CC BY 4.0 by the author.