The 'init' Package

In a traditional Kubernetes deployment, clusters pull resources (e.g. cluster images, OCI artifacts, Git repos) from external sources.

However, in an air-gapped environment, these external providers are not available, or exist at different locations to their references within Kubernetes manifests.

Zarf solves this problem with its ‘init’ package, a special Zarf Package (traditionally deployed first) that provides the necessary mechanisms to enable air-gapped Kubernetes, and deliver DevSecOps across air-gaps.

tldr;

Don’t care about the details and just want to get deploying as quickly as possible? Run the following after connecting to a cluster:

zarf tools download-init
zarf init --confirm

Want to see a guided zarf init? Check out the Zarf Init tutorial.

View all init options w/ zarf init --help.

Core Components

An ‘init’ package requires a series of specially named, and configured components to ensure the cluster is correctly initialized. These components are:

• zarf-injector
• zarf-seed-registry
• zarf-registry
• zarf-agent

zarf-injector and zarf-seed-registry

One of the most challenging aspects of deploying into an air-gapped environment is the initial bootstrapping of the cluster.

A cluster needs a registry to pull images from, but spinning up a registry requires an image to be pulled from a registry - chicken, meet egg.

To ensure that our approach is distro-agnostic, the Zarf team developed a unique solution to seed the container registry into the cluster, populate said registry, and redirect cluster resources to use the air-gapped registry.

Shoving random data into a cluster is generally a bad idea, and an antipattern overall to containerization. However in the case of Zarf, and air-gapped environments, certain liberties must be taken.

While there is no distro-agnostic method to inject images into a cluster, every cluster has support for configmaps. However, the size of a configmap is limited to 1MB (technically only limited by whatever is configured in etcd, the default is 1MB), and the registry:2 image is around 10MB (as of this writing). So we split the registry:2 image into chunks and inject them into the cluster as configmaps.

But then we have another problem of how to reassemble the image on the other side, as we don’t have any consistent image that exists in the cluster that would have such utilities. This is where the zarf-injector Rust binary comes in.

For compiling the zarf-injector binary, refer to its README.md.

The zarf-injector binary is statically compiled and injected into the cluster as a configmap along with the chunks of the registry:2 image. During the zarf-seed-registry’s deployment, the zarf-injector binary is run in a pod that mounts the configmaps and reassembles the registry:2 image. It then hosts a temporary, pull-only Docker registry implemented in Rust so that a real registry can be deployed into the cluster from the hosted registry:2 image.

While the zarf-injector component must be defined and deployed before the zarf-seed-registry component, the magic doesn’t start until zarf-seed-registry is deployed. The zarf-injector component for the most part just copies the zarf-injector binary to ###ZARF_TEMP###/zarf-injector.

When zarf init deploys the zarf-seed-registry component, the following happens:

1. Zarf injects the zarf-injector binary and the registry:2 image chunks into the cluster.
2. Zarf connects to the cluster and grabs a pod that is running an image that is already present in the cluster.
3. Zarf spins up a pod using the existing image, mounts the configmaps that contain the zarf-injector binary and the registry:2 image chunks and runs the zarf-injector binary.

Note

Doing this keeps Zarf cluster agnostic, however does require that the kubelet be able to reach out to a cluster NodePort service, which may require changes to firewall configurations like allowing UDP traffic between nodes if using something like VXLAN tunneling.

4. The zarf-injector binary reassembles the registry:2 image and hosts a temporary registry that the cluster can pull from.
5. The docker-registry chart in the zarf-seed-registry component is then deployed, with its image.repository set to the temporary registry that the zarf-injector binary is hosting (consumed as the ###ZARF_SEED_REGISTRY### built-in variable set at runtime).
6. Once the docker-registry chart is deployed, the zarf-seed-registry component is marked as complete and the zarf-injector pod is removed from the cluster.
7. Deployment proceeds to the zarf-registry component.

Note

The registry:2 image and the Zarf Agent image can be configured with a custom init package using the registry_image_* and agent_image_* templates defined in the Zarf repo’s zarf-config.toml. This allows you to swap them for enterprise provided / hardened versions if desired such as those provided by Iron Bank.

zarf-registry

The zarf-registry component is a long-lived container registry service that is deployed into the cluster.

It leverages the same docker-registry chart used in zarf-seed-registry but with a few key differences:

1. The image.repository is set to the value of the built-in variable ###ZARF_REGISTRY### which is set at runtime to the registry hosted by zarf-seed-registry.

2. A connect manifest for running zarf connect registry to tunnel to the Zarf Registry.
3. A configmap to satisfy KEP-1755

Tip

You can further customize how the registry behaves by setting variables such as REGISTRY_PVC_SIZE with a config file or --set on zarf init.

To see a full list of variables you can view the zarf.yaml that defines the registry.

Using External Registries

Zarf can be configured to use an already existing registry with the --registry-* flags when running zarf init.

This option skips the injector and seed process, and will not deploy a registry inside of the cluster. Instead, it pushes any images contained in the package to the externally configured registry.

Note

Given the registry is a core part of any Kubernetes deployment you MUST either specify an external registry with the --registry-* flags or use the injected registry.

Making the Registry Highly-Available

By default, the registry included in the init package creates a ReadWriteOnce PVC and is only scheduled to run on one node at a time.

This setup is usually enough for smaller and simpler deployments. However, for larger deployments or those where nodes are frequently restarted or updated, you may want to make the registry highly-available.

This approach requires certain prerequisites, such as a storage class that supports ReadWriteMany, or being in an environment that allows you to configure the registry to use an S3-compatible backend.

Additionally, you must provide custom configuration to the registry to ensure it is distributed across all nodes and has the appropriate number of replicas. Below is an example configuration file using a ReadWriteMany storage class:

zarf-config.yaml

package:
  deploy:
    set:
      REGISTRY_PVC_ENABLED: "true"
      REGISTRY_PVC_ACCESS_MODE: "ReadWriteMany"
      REGISTRY_HPA_AUTO_SIZE: "true"
      REGISTRY_AFFINITY_CUSTOM: |
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - docker-registry
                topologyKey: kubernetes.io/hostname

Notably, the REGISTRY_AFFINITY_CUSTOM variable overrides the default pod anti-affinity, and REGISTRY_HPA_AUTO_SIZE automatically adjusts the minimum and maximum replicas for the registry based on the number of nodes in the cluster. If you prefer to manually set the minimum and maximum replicas, you can use REGISTRY_HPA_MIN and REGISTRY_HPA_MAX to specify the desired values.

zarf-agent

The zarf-agent is a Kubernetes Mutating Webhook that intercepts requests to create resources and uses the zarf-state secret to mutate them to point to their air-gapped equivalents.

The zarf-agent is responsible for modifying Kubernetes PodSpec objects Image fields to point to the Zarf Registry. This allows the cluster to pull images from the Zarf Registry instead of the internet without having to modify the original image references.

The zarf-agent modifies the following flux resources: GitRepository, OCIRepository, & HelmRepository to point to the local Git Server or Zarf Registry. HelmRepositories are only modified if the type key is set to oci.

Support for mutating OCIRepository and HelmRepository objects is in alpha and should be tested on non-production clusters before being deployed to production clusters.

The zarf-agent modifies ArgoCD applications & ArgoCD Repositories objects to point to the local Git Server.

Support for mutating Application and Repository objects in ArgoCD is in beta and should be tested on non-production clusters before being deployed to production clusters.

Note

During the zarf init operation, the Zarf Agent will add the zarf.dev/agent: ignore label to prevent the Agent from modifying any resources in that namespace. This is done because there is no way to guarantee the images used by pods in existing namespaces are available in the Zarf Registry.

If you would like to adopt pre-existing resources into a Zarf deployment you can use the --adopt-existing-resources flag on zarf package deploy to adopt those resources into the Helm Releases that Zarf manages (including namespaces). This will add the requisite annotations and labels to those resources and drop the zarf.dev/agent: ignore label from any namespaces specified by those resources.

Image Mutation to Unique Hashed Tags

When the agent mutates an image that is not pinned to a digest, it appends a CRC32 hash to the tag. For example, if the original image is ghcr.io/stefanprodan/podinfo:6.4.0 the mutated image tag will be 6.4.0-zarf-298505108. The CRC32 hash 298505108 is generated using the original image name ghcr.io/stefanprodan/podinfo.

Without this unique hash, images from different registries with the same path—such as docker.io/stefanprodan/podinfo:6.4.0-would overwrite each other when pushed to the Zarf registry. Zarf pushes both the regular tag and the unique tag for non pinned images. This ensures no images are lost during a push, and the agent can always mutate to the correct image. To see which image a pod used before mutation, check the zarf.dev/original-image-<container-name> annotation.

Additionally, when Git repositories are pushed to the Zarf Git server their name is appended with a CRC32 hash to prevent similar collisions.

Excluding Resources from zarf-agent

Resources can be excluded at the namespace or resources level by adding the zarf.dev/agent: ignore label.

Zarf will refuse to adopt the Kubernetes initial namespaces (default, kube-*, etc…). This is because these namespaces are critical to the operation of the cluster and should not be managed by Zarf.

Additionally, when adopting resources, ensure that the namespaces specified are dedicated to Zarf, or add the zarf.dev/agent: ignore label to any non-Zarf managed resources in those namespaces (and ensure that updates to those resources do not strip that label) otherwise ImagePullBackOff errors may occur.

The Agent does not need to create any secrets in the cluster. Instead, during zarf init and zarf package deploy, secrets are automatically created in a Helm Postrender Hook for any namespaces Zarf sees. If you have resources managed by Flux that are not in a namespace managed by Zarf, you can either create the secrets manually or include a manifest to create the namespace in your package and let Zarf create the secrets for you.

Optional Components

The Zarf team maintains some optional components in the default ‘init’ package.

Components	Description
k3s	REQUIRES ROOT (not sudo). Installs a lightweight Kubernetes Cluster on the local host K3s and configures it to start up on boot.
git-server	Adds a GitOps-compatible source control service Gitea into the cluster.

There are two ways to deploy these optional components. First, you can provide a comma-separated list of components to the --components flag, such as zarf init --components k3s,git-server --confirm, or, you can choose to exclude the --components and --confirm flags and respond with a yes (y) or no (n) for each optional component when interactively prompted.

Caution

(Linux only) Deploying the ‘k3s’ component will require root access (not just sudo), as it modifies your host machine to install the cluster and by default only provides access to the cluster to the root user.

Tip

The k3s component included differs from the default k3s install in that it disables the installation of traefik out of the box. This was done so that people could more intentionally choose if they wanted traefik or another ingress provider (or no ingress at all) depending on their needs. If you would like to return k3s to its defaults, you can set the K3S_ARGS Zarf variable to an empty string:

root@machine ~ # zarf init --components k3s --set K3S_ARGS="" --confirm

Tip

You can further customize how the git-server behaves by setting variables such as GIT_SERVER_PVC_SIZE with a config file or --set on zarf init.

To see a full list of variables you can view the zarf.yaml that defines the git-server.

Putting it All Together

The package definition ‘init’ is similar to writing any other Zarf Package, but with a few key differences:

Starting with kind and metadata:

zarf.yaml

# kind must be ZarfInitConfig
kind: ZarfInitConfig
metadata:
  # name *can* be anything, but it is generally recommended to end with 'init'
  name: init
  # version should be empty as it will be set by the Zarf CLI
  # (this is ONLY for the 'init' package)
  # version: 0.1.0
...

In order for Zarf to operate correctly, the following components:

• must be defined, ordered, and named exactly as shown below
• must have the required field set to true

zarf.yaml

components:
  # components (like k3s) that spin up a cluster...

  - name: zarf-injector
    required: true
    ...

  - name: zarf-seed-registry
    required: true
    ...

  - name: zarf-registry
    required: true
    ...

  - name: zarf-agent
    required: true
    ...

  # optional components that need a cluster ...

Note

In order to reproduce / build the following example, you will need to have the Zarf repository cloned locally.

git clone https://github.com/zarf-dev/zarf.git
cd zarf
mv zarf.yaml zarf.yaml.bak

You can learn more about creating a custom init package in the Creating a Custom ‘init’ Package Tutorial.

A minimal zarf.yaml for the ‘init’ package looks something like:

zarf.yaml

kind: ZarfInitConfig
metadata:
  name: init

components:
  - name: zarf-injector
    required: true
    import:
      path: packages/zarf-registry

  - name: zarf-seed-registry
    required: true
    import:
      path: packages/zarf-registry

  - name: zarf-registry
    required: true
    import:
      path: packages/zarf-registry

  - name: zarf-agent
    required: true
    import:
      path: packages/zarf-agent