Shipping a 6 GB ISO to every pod (and then deleting the storage key)

A Windows 11 ISO is about 6 GB. Every computer pod needs one, plus the OEM provisioning scripts, plus the seven control servers that get installed into the guest.

Three options, and the first two are traps.

Bake it into the image. Now your container image is 6+ GB. Every push moves it, every node pulls it, and your build takes minutes to do nothing. Worse, the image now contains redistributable-restricted Microsoft installation media, which is a licensing conversation you do not want to have with your registry.

Mount it from a pre-existing volume. Now the chart has an external prerequisite that has to exist before anything works, created by a process outside the chart, in a state nobody can verify from the manifest.

Pull it at pod start, once. This is what we did.

The initContainer

      {{- if not .Values.dockur.useSnapshot }}
      initContainers:
        - name: my-init-container
          image: {{ .Values.initContainers.image }}
          env:
            - name: STORAGE_ACCOUNT_NAME
              value: "{{ .Values.initContainers.storage_account_name }}"
            - name: BASE_PATH
              value: "{{ .Values.initContainers.base_path }}"
            - name: ISO_NAME
              value: "{{ .Values.dockur.isoName }}"
          volumeMounts:
            - name: hdd
              mountPath: "{{ .Values.initContainers.base_path }}"
      {{- end }}

It runs before the main container, mounts the same PVC, and downloads three things from Azure Files: the OO servers, the Windows provisioning data, and the ISO.

The {{- if not .Values.dockur.useSnapshot }} is important and gets its own post — when a computer is cloned from a golden image, the disk already has everything, and running the initContainer would be pure waste.

Idempotency, twice

Two separate guards, because two different things are being downloaded.

For directories, “already there and non-empty” is good enough:

def download_all(dir_client: ShareDirectoryClient, local_path: str):
    if os.path.exists(local_path) and os.listdir(local_path):
        print(f"✅ Skipping already downloaded folder: {local_path}")
        return

For the ISO, an explicit file check:

if os.path.exists(local_iso_path):
    print(f"✅ Skipping ISO download, file already exists: {local_iso_path}")
else:
    print(f"⬇️ Downloading ISO file: {iso_file_path} -> {local_iso_path}")
    with open(local_iso_path, "wb") as f:
        stream = file_client.download_file()
        f.write(stream.readall())

Because the PVC persists across pod restarts, this means the expensive download happens once per computer, not once per restart. A pod that crash-loops does not re-pull 6 GB each time — which it absolutely did, once, before this check existed, and the storage egress graph for that afternoon was memorable.

One PVC, eleven mount points

The container side is where it gets interesting. A single PVC gets projected into the QEMU wrapper’s expected filesystem layout via subPath:

          volumeMounts:
            - name: hdd
              mountPath: /storage                    # the qcow2 disk lives here

            - name: hdd
              mountPath: /custom.iso                 # ← a FILE, not a directory
              subPath: downloads/windows-iso/{{ .Values.dockur.isoName }}

            - name: hdd
              mountPath: /oem
              subPath: downloads/windows-data/oem
            - name: hdd
              mountPath: /data/init
              subPath: downloads/windows-data/data/init

            - name: hdd
              mountPath: /data/server_computer_control
              subPath: downloads/oo-servers/server_computer_control
            - name: hdd
              mountPath: /data/mcp_server_computer_control
              subPath: downloads/oo-servers/mcp_server_computer_control
            - name: hdd
              mountPath: /data/server_browser_control
              subPath: downloads/oo-servers/server_browser_control
            - name: hdd
              mountPath: /data/server_evaluator
              subPath: downloads/oo-servers/server_evaluator
            - name: hdd
              mountPath: /data/server_network_proxy
              subPath: downloads/oo-servers/server_network_proxy
            - name: hdd
              mountPath: /data/server_teams_control
              subPath: downloads/oo-servers/server_teams_control
            - name: hdd
              mountPath: /data/server_appium
              subPath: downloads/oo-servers/server_appium

That block is the entire guest filesystem contract in thirty lines, and it is the Kubernetes translation of the bind mounts from the local compose file.

Two things worth knowing:

subPath can mount a single file. /custom.iso is a file mount, not a directory. The QEMU wrapper looks for a file at exactly that path, and subPath lets one PVC serve both the file and the directories around it.

subPath mounts do not receive updates. If the underlying content changes after the pod starts, a subPath mount will not see it — unlike a normal volume mount. For a ConfigMap that is a famous gotcha. Here it is fine, because the content is written once by the initContainer before the main container starts, and is then immutable for the pod’s life. Worth knowing before you copy the pattern somewhere it matters.

Getting the payload up there

The upload side is a small Python script in the app repo, run from a workstation whenever the servers change:

SHARE_NAME = "oo-servers"
ROOT_FOLDER = "../../servers"
SELECTED_FOLDERS = [
    "mcp_server_computer_control",
    "server_computer_control",
    "server_browser_control",
    "server_evaluator",
    "server_network_proxy",
    "server_appium",
    "server_teams_control"
]
IGNORE_NAMES = {".venv", "__pycache__", ".DS_Store"}

Three Azure File shares — oo-servers, windows-data, my-iso — populated by two scripts.

I want to be honest that this is the weakest link in the whole pipeline. Source code reaches production by a human running a script that copies files onto a file share. There is no versioning, no immutability, and no way to tell from a test result which revision of the servers it exercised. A colleague running the upload mid-benchmark changes what a running test is executing.

The right answer is to build the servers into a versioned artifact — a container image, or at minimum a tagged tarball — and have the initContainer fetch a specific version named in the chart values. We knew that. It stayed on the list. If you take one design idea from this post, take that one instead of the script.

Killing the keys, in three acts

The credential story took three months and is visible in the code as a stack of commented-out attempts:

# credential = DefaultAzureCredential()
# credential = DefaultAzureCredential(managed_identity_client_id=IDENTITY)
# credential = ManagedIdentityCredential(client_id=IDENTITY)
credential = WorkloadIdentityCredential(
    tenant_id=TENANT_ID,
    client_id=IDENTITY,
    token_file_path="/var/run/secrets/azure/tokens/azure-identity-token"
)

Act 1 — public container. Fastest possible path to a working pod. No credentials because no access control. Anyone who could guess the URL could download our provisioning payload.

Act 2 — a connection string in an env var. Better, and still a long-lived shared secret that is impossible to rotate without coordinating every consumer, and that lands in .env files, in shell history, and eventually in git.

Act 3 — workload identity. The pod gets a projected service-account token, exchanges it for an Azure AD token scoped to a user-assigned managed identity, and that identity has an RBAC role on the storage account. No secret exists anywhere. Rotation is automatic. Revocation is a role assignment.

The pod side is one label:

      labels:
        azure.workload.identity/use: "true"

That triggers the mutating webhook that projects the token file the credential above reads. Miss the label and you get a confusing authentication failure that looks like a permissions problem and is actually a “the token file does not exist” problem.

There is a commit in this arc called WIP security - does not work, which is the most accurate commit message I wrote all year.

The final act came later still. On 14 July a commit landed titled “Fix blob storage access not via public access anymore” — turning off public network access on the storage account entirely, so it is reachable only from the cluster’s network. Identity plus network isolation, rather than either alone.

The order I would do it in

We did access control last, and it cost more than doing it first would have. Not because the final design is complicated — a managed identity and a role assignment is a small amount of Terraform — but because every intermediate stage left artifacts. Connection strings in .env files that are still in git history. A public container that existed for six weeks. Scripts that assumed a key and had to be rewritten.

If I were starting again:

1. Identity first, on day one. The plumbing is the same amount of work whenever you do it; doing it later means also doing the cleanup.
2. Versioned artifacts, not file shares. Then “which code did this test run” has an answer.
3. Idempotency guards before the first cluster deploy. The egress bill from a crash-looping pod re-downloading 6 GB is a lesson you only need once.

Next: the pod now boots, provisions itself, and installs Windows. Forty minutes later it is ready. Let us make that ninety seconds.