Skip to main content

Air-Gapped Environments

This guide covers how to deploy and operate ClearML in air-gapped environments—environments with restricted or no internet access. It includes steps for locally hosting all required resources and configuring ClearML components accordingly.

The guide covers:

  • Hosting dependencies for ClearML Applications
  • Configuring application containers with offline resources
  • Configuring Kubernetes deployments using private registries and image pull secrets

Preparing ClearML Applications for Air-Gapped Use​

Various application dependencies that are auto-downloaded from the internet can be locally hosted and configured for offline access.

Hosting Required Python Packages​

Ensure the following Python packages are locally hosted in your PyPI proxy or Python packages artifactory, and available using a local URL. Or, if you are going to use custom images, make sure they are installed.

jupyter
jupyterlab>4,<4.4
traitlets
mitmproxy<10.2
werkzeug>2,<3.0 ; python_version < '3.9'
clearml>=1.9
clearml_session==0.16.0
tqdm
boto3>=1.9
pylint
clearml-agent

If hosting the previous Python packages locally, make sure to set PIP_EXTRA_INDEX_URL=<LOCAL_REPO_URL> for containers running ClearML tasks. The following is an example in Kubernetes using the ClearML Agent helm values override:

agentk8sglue:
  queues:
    myQueue:
      templateOverrides:
        env:
          - name: PIP_EXTRA_INDEX_URL
            value: "<LOCAL_REPO_URL>"

Application environment variables (see below) can be set using any of the following:

  • ClearML Administrator Vault configuration: Set agent.extra_docker_arguments
  • Agent config on VMs or bare-metal: Set agent.extra_docker_arguments
  • Kubernetes ClearML Agent deployments: Set the basePodTemplate

Ensure that:

  • All containers and pods are configured to pull images from your private container registry
  • Custom images include Python 3

App-Specific Offline Resources​

VSCode Application​

For the ClearML VSCode Application to work offline, provide the following to all containers started by the ClearML Agent running GPU workloads:

  • VSCode Server debian package: Set CLEARML_SESSION_VSCODE_SERVER_DEB=<PATH_TO_DEB_FILE>. Package can be found here (version number can be updated, see https://github.com/coder/code-server/releases).
  • VSCode Python extension: Set CLEARML_SESSION_VSCODE_PY_EXT=<PATH_TO_EXTENSION_FILE> pointing to the Visual Studio marketplace. Package can be found here (version number can be updated, see the Visual Studio Marketplace).

Example in Kubernetes using the ClearML Agent helm values override:

agentk8sglue:
  queues:
    myQueue:
      templateOverrides:
        env:
          - name: CLEARML_SESSION_VSCODE_SERVER_DEB
            value: "<PATH_TO_DEB_FILE>"
          - name: CLEARML_SESSION_VSCODE_PY_EXT
            value: "<PATH_TO_EXTENSION_FILE>"

SSH Session Application​

For air-gapped SSH applications using the DropBear server (required for non-privileged containers):

  • Download and host the DropBear executable.
  • Set CLEARML_DROPBEAR_EXEC=<PATH_TO_EXECUTABLE> in all containers started by the ClearML Agent running GPU workloads

Convert Image Registry​

ClearML Application installation requires running the convert_image_registry.py script included in the package. Images that need to be mirrored will be listed in the script output of the same script. Mirror these images to your private registry before proceeding with the upload of application packages.

Kubernetes Environments​

Use a Custom imagePullSecret​

To ensure Kubernetes workloads (ClearML Agents, Server, and App Gateway) use your private registry, configure imagePullSecrets in the appropriate Helm override files.

  • To use a custom defined imagePullSecret for a ClearML Agent and the tasks Pods it creates, configure the following in your clearml-agent-values.override.yaml file:

    imageCredentials:
      extraImagePullSecrets:
        - name: "<IMAGE_PULL_SECRET_NAME>"

  • To use a custom defined imagePullSecret for the ClearML Server, configure the following in your clearml-values.override.yaml file:

    imageCredentials:
      existingImagePullSecrets:
        - name: "<IMAGE_PULL_SECRET_NAME>"

  • To use a custom defined imagePullSecret for the ClearML App Gateway, configure the following in your clearml-app-gateway-values.override.yaml file:

    imageCredentials:
      existingImagePullSecrets:
        - name: "<IMAGE_PULL_SECRET_NAME>"

Create a Custom imagePullSecret​

To create a registry secret in Kubernetes, use the following command example. The secret needs to be created in the namespace where it will be used.

kubectl create secret docker-registry -n <NAMESPACE> <SECRET_NAME> \
  --docker-server=<REPO_URL> \
  --docker-username=<USERNAME> \
  --docker-password=<PASSWORD> \
  --docker-email=<EMAIL_OR_EMPTY_STRING>

List Images Used in a ClearML Helm Chart​

To see all container images used by a ClearML Helm chart:

helm template <CHART_NAME> | yq '..|.image? | select(.)' | sort -u
note

This requires the helm and yq commands to be installed.

Webserver​

When using a private registry, this configuration will make the Webserver reference the correct extra index URL for Enterprise packages.

In Kubernetes:

clearml:
  extraIndexUrl: "<YOUR_REPO_URL>"