Shipping Helm Charts to Air-Gapped Kubernetes Clusters

Background

I was working on a security product that handles sensitive data for clients. The product was using microservice architecture. We selected Kubernetes for orchestrating the deployment because it made easier to scale up and down based on load. Kubernetes has many way to install an application. We selected Helm charts because it provided a template engine which was helpful in reducing the number of manifest files by using templates.

As we grew, we started to get bigger clients. The Clients wanted to install the application in an air gapped environment due to the nature of the data being processed. This meant that we had to come up with a custom installation process which would install the application without internet.

In my previous blog, I discussed how to ship Docker images to air-gapped systems. However, when working with Kubernetes and Helm, the challenge becomes more complex. You are not delivering just one Docker image. You are delivering a full application stack with many images, configurations, and dependencies.

The main challenges we faced were:

• Managing all Docker images referenced in the Helm chart
• Ensuring images were available on all worker nodes
• Handling upgrades and ensuring the latest images were deployed
• Distributing Docker images along with the Helm chart in a single package

What is Helm?

Before we dive into the solution, let’s understand what Helm is. Helm is a package manager for Kubernetes, similar to how apt is for Ubuntu or dnf/yum is for RHEL. It allows you to define, install, and upgrade Kubernetes applications using charts.

A Helm chart is a collection of files that describe a related set of Kubernetes resources. A single chart might deploy a simple application like nginx, or something complex like a full web application stack with databases, caches, and multiple services.

The Challenge with Air-Gapped Deployments

When you deploy a Helm chart in a normal environment with internet access, Kubernetes pulls the Docker images from public registries like Docker Hub or Google Container Registry. However, in an air-gapped environment:

1. Kubernetes cannot pull images from public registries
2. All images must be available locally
3. The Helm chart’s image references must point to local registries
4. You need a way to package and transport everything together

The Solution: Private Registry + Automated Packaging

After many iterations, we developed a solution that involved:

1. Setting up a private Docker registry inside the air-gapped Kubernetes cluster
2. Creating a packaging script that bundles Helm charts along with all the required Docker images
3. Creating an installation script that automates the deployment process

Let’s see how we implemented this step by step.

Setting Up the Private Registry

Prerequisites

Before we begin, you need:

• A Kubernetes cluster in the air-gapped environment
• A storage class available in the cluster for persistent volumes
• kubectl configured to access the cluster
• helm installed on your local machine
• docker or podman installed

Step 1: Prepare the Registry Image

On a machine with internet access, pull and save the Docker registry image:

# Pull the official Docker registry image
docker pull registry:2

# Save the image to a tar file
docker image save registry:2 | gzip > registry.tar.gz

Step 2: Transfer and Load the Registry Image

Copy the registry image to the air-gapped environment:

# Copy to the air-gapped machine
scp registry.tar.gz airgapped-node.local:/tmp/

# SSH to the air-gapped machine and load the image
ssh airgapped-node.local "bash -c 'gunzip -c /tmp/registry.tar.gz | docker image load'"

If you have multiple worker nodes, you need to load the image on at least one node initially. We’ll use node labels to control where the registry pod runs.

Step 3: Create Registry Authentication

Create a password file for registry authentication:

# Install htpasswd if not available
dnf install httpd-tools -y

# Create auth directory
mkdir -p /tmp/registry-auth

# Create a user (replace 'admin' and 'password' with your credentials)
htpasswd -Bbn admin password > /tmp/registry-auth/htpasswd

Step 4: Create Kubernetes Resources for Registry

Create a namespace for the registry:

kubectl create namespace container-registry

Create a secret with the authentication file:

kubectl create secret generic registry-auth \
  --from-file=htpasswd=/tmp/registry-auth/htpasswd \
  -n container-registry

Create a PersistentVolumeClaim for registry storage:

# registry-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: registry-pvc
  namespace: container-registry
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 20Gi
  storageClassName: your-storage-class

Apply the PVC:

kubectl apply -f registry-pvc.yaml

Replace your-storage-class with the storage class provided by your cluster administrator.

Step 5: Label the Node for Initial Registry Deployment

Label the node where you loaded the registry image:

# List nodes
kubectl get nodes

# Label a node (replace node-name with actual node name)
kubectl label nodes node-name primary-registry=true

Step 6: Deploy the Registry

Create the registry deployment:

# registry-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: docker-registry
  namespace: container-registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: docker-registry
  template:
    metadata:
      labels:
        app: docker-registry
    spec:
      nodeSelector:
        primary-registry: "true"
      containers:
      - name: registry
        image: registry:2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 5000
        env:
        - name: REGISTRY_AUTH
          value: "htpasswd"
        - name: REGISTRY_AUTH_HTPASSWD_REALM
          value: "Registry Realm"
        - name: REGISTRY_AUTH_HTPASSWD_PATH
          value: "/auth/htpasswd"
        - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
          value: "/var/lib/registry"
        volumeMounts:
        - name: registry-storage
          mountPath: /var/lib/registry
        - name: registry-auth
          mountPath: /auth
          readOnly: true
      volumes:
      - name: registry-storage
        persistentVolumeClaim:
          claimName: registry-pvc
      - name: registry-auth
        secret:
          secretName: registry-auth

Apply the deployment:

kubectl apply -f registry-deployment.yaml

Step 7: Expose the Registry

Create a service for the registry:

# registry-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: docker-registry
  namespace: container-registry
spec:
  selector:
    app: docker-registry
  ports:
  - port: 5000
    targetPort: 5000
  type: ClusterIP

Create an ingress for external access:

# registry-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: docker-registry
  namespace: container-registry
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
spec:
  ingressClassName: nginx
  rules:
  - host: registry.airgapped.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: docker-registry
            port:
              number: 5000

Apply the service and ingress:

kubectl apply -f registry-service.yaml
kubectl apply -f registry-ingress.yaml

Make sure to add registry.airgapped.local to your /etc/hosts file or DNS pointing to your ingress controller IP.

Step 8: Push Registry Image to Registry

Now we need to push the registry image to itself:

# Login to the registry
docker login registry.airgapped.local -u admin

# Tag the registry image
docker tag registry:2 registry.airgapped.local/registry:2

# Push to the registry
docker push registry.airgapped.local/registry:2

Step 9: Scale Up the Registry

Update the deployment to remove node selector and scale up:

# Edit the deployment
kubectl edit deployment docker-registry -n container-registry

# Remove the nodeSelector section and update image to use registry URL
# Change image from 'registry:2' to 'registry.airgapped.local/registry:2'
# Save and exit

# Scale up the deployment
kubectl scale deployment docker-registry --replicas=3 -n container-registry

Now the registry is running on multiple nodes with high availability!

Packaging Helm Charts with Docker Images

Now that we have a registry, let’s create a script to package Helm charts with their Docker images.

Step 1: Create the Packaging Script

Create a script called package-helm-chart.sh:

#!/bin/bash

set -e

CHART_PATH=$1
OUTPUT_DIR=${2:-./helm-package}

if [ -z "$CHART_PATH" ]; then
  echo "Usage: $0 <chart-path> [output-dir]"
  exit 1
fi

CHART_NAME=$(basename "$CHART_PATH")
PACKAGE_DIR="$OUTPUT_DIR/$CHART_NAME"

echo "Packaging Helm chart: $CHART_NAME"

# Create output directory
mkdir -p "$PACKAGE_DIR/images"
mkdir -p "$PACKAGE_DIR/charts"

# Package the Helm chart
echo "Creating Helm chart package..."
helm package "$CHART_PATH" -d "$PACKAGE_DIR/charts"

# Extract image references from the chart
echo "Extracting Docker images from chart..."
IMAGES=$(helm template "$CHART_PATH" | grep -oP '(?<=image: ).*' | sort -u)

# Pull and save each image
for IMAGE in $IMAGES; do
  echo "Processing image: $IMAGE"
  IMAGE_NAME=$(echo "$IMAGE" | tr '/:' '-')
  
  # Pull the image
  docker pull "$IMAGE"
  
  # Save the image
  docker save "$IMAGE" | gzip > "$PACKAGE_DIR/images/${IMAGE_NAME}.tar.gz"
done

# Create image list file
echo "$IMAGES" > "$PACKAGE_DIR/images/image-list.txt"

echo "Package created successfully at: $PACKAGE_DIR"

Make the script executable:

chmod +x package-helm-chart.sh

Step 2: Create the Installation Script

Create an installation script called install-chart.sh:

#!/bin/bash

set -e

PACKAGE_DIR=$1
REGISTRY_URL=${2:-registry.airgapped.local}
REGISTRY_USER=${3:-admin}
NAMESPACE=${4:-default}

if [ -z "$PACKAGE_DIR" ]; then
  echo "Usage: $0 <package-dir> [registry-url] [registry-user] [namespace]"
  exit 1
fi

echo "Installing from package: $PACKAGE_DIR"

# Login to registry
echo "Logging in to registry..."
docker login "$REGISTRY_URL" -u "$REGISTRY_USER"

# Load and push images
echo "Loading and pushing Docker images..."
while IFS= read -r IMAGE; do
  IMAGE_NAME=$(echo "$IMAGE" | tr '/:' '-')
  IMAGE_FILE="$PACKAGE_DIR/images/${IMAGE_NAME}.tar.gz"
  
  if [ -f "$IMAGE_FILE" ]; then
    echo "Loading image: $IMAGE"
    gunzip -c "$IMAGE_FILE" | docker load
    
    # Tag for local registry
    LOCAL_IMAGE="$REGISTRY_URL/${IMAGE#*/}"
    docker tag "$IMAGE" "$LOCAL_IMAGE"
    
    # Push to registry
    echo "Pushing to registry: $LOCAL_IMAGE"
    docker push "$LOCAL_IMAGE"
  fi
done < "$PACKAGE_DIR/images/image-list.txt"

# Find the chart file
CHART_FILE=$(ls "$PACKAGE_DIR/charts"/*.tgz | head -n 1)
CHART_NAME=$(basename "$CHART_FILE" .tgz | sed 's/-[0-9].*//')

# Check if release exists
if helm list -n "$NAMESPACE" | grep -q "$CHART_NAME"; then
  echo "Release exists, performing upgrade..."
  
  # Backup database if configured
  if [ -n "$DB_BACKUP_SCRIPT" ]; then
    echo "Creating database backup..."
    bash "$DB_BACKUP_SCRIPT"
  fi
  
  helm upgrade "$CHART_NAME" "$CHART_FILE" \
    --namespace "$NAMESPACE" \
    --set global.imageRegistry="$REGISTRY_URL" \
    --wait
else
  echo "Installing new release..."
  helm install "$CHART_NAME" "$CHART_FILE" \
    --namespace "$NAMESPACE" \
    --create-namespace \
    --set global.imageRegistry="$REGISTRY_URL" \
    --wait
fi

echo "Installation completed successfully!"
echo ""
echo "To rollback if needed:"
echo "  helm rollback $CHART_NAME -n $NAMESPACE"

Make the script executable:

chmod +x install-chart.sh

Complete Workflow

On a Machine with Internet Access

1. Package your Helm chart with all images:

    ./package-helm-chart.sh ./my-app-chart ./output
   

2. Create a final archive:

    cd output
    tar -czf my-app-package.tar.gz my-app-chart/
   

3. Copy the installation script into the package:

    cp ../install-chart.sh my-app-chart/
    tar -czf my-app-package.tar.gz my-app-chart/
   

4. Transfer to air-gapped environment:

    scp my-app-package.tar.gz airgapped-node.local:/tmp/
   

On the Air-Gapped Machine

1. Extract the package:

    cd /tmp
    tar -xzf my-app-package.tar.gz
    cd my-app-chart
   

2. Run the installation:

    ./install-chart.sh . registry.airgapped.local admin my-app-namespace
   

The script will:

• Load all Docker images
• Push them to the private registry
• Install or upgrade the Helm chart
• Configure the chart to use the local registry

Handling Upgrades

When you need to upgrade the application:

1. Package the new version using the same packaging script
2. Transfer to the air-gapped environment
3. Run the installation script - it will automatically detect the existing release and perform an upgrade

The old images remain in the registry for rollback scenarios. To clean up old images:

# List images in registry
curl -u admin:password https://registry.airgapped.local/v2/_catalog

# Delete specific image tag (requires registry with deletion enabled)
curl -u admin:password -X DELETE \
  https://registry.airgapped.local/v2/<image-name>/manifests/<tag>

Conclusion

Shipping Helm charts to air-gapped Kubernetes clusters requires careful planning and automation. The key components are:

• A private Docker registry running inside the Kubernetes cluster for high availability
• Automated packaging scripts that bundle Helm charts with all required Docker images
• Installation scripts that handle both fresh installations and upgrades intelligently
• Proper backup and rollback mechanisms for safe deployments

These days various Kubernetes distributions offer built-in support for air-gapped installations, making this process easier. For example, K3S provides a registry mirror, OpenShift also has an inbuilt registry.

References

• Helm Documentation
• Registry Documentation
• Kubernetes Documentation