Nested virtualisation on AKS: yes, you can run KVM in a pod

Everything so far ran on one laptop. One Windows VM, one agent, one result at a time — which, once we established that a single run is not a result, stopped being useful.

So: put the whole thing on Kubernetes. Which means running a hypervisor inside a pod, on a cloud VM that is itself a guest. Nested virtualisation, on managed Kubernetes, with a Windows install inside it.

It works. Here is the spec that makes it work, line by line.

The three lines that do everything

          volumeMounts:
            - name: dev-tun
              mountPath: /dev/net/tun
            - name: dev-kvm
              mountPath: /dev/kvm

          securityContext:
            privileged: true
            capabilities:
              add:
                - NET_ADMIN
      volumes:
        - name: dev-tun
          hostPath:
            path: /dev/net/tun
            type: CharDevice
        - name: dev-kvm
          hostPath:
            path: /dev/kvm
            type: CharDevice

This is the compose file from the first post, translated. Docker’s devices: has no direct Kubernetes equivalent — the Device Plugin API is the “proper” answer and requires writing and deploying a plugin — so the pragmatic route is a hostPath volume with type: CharDevice.

type: CharDevice matters. Without it, if /dev/kvm does not exist on the node, Kubernetes helpfully creates a directory at that path and mounts it. The pod starts. QEMU opens /dev/kvm, gets a directory, and fails in a way that has nothing to do with the actual problem. With CharDevice, the pod fails to schedule with a clear message. Fail loudly.

About privileged: true

I am not going to pretend this is fine.

privileged: true disables essentially all container isolation: full capability set, unrestricted device access, no seccomp confinement. A process in that container is, for practical purposes, root on the node.

We tried to avoid it. NET_ADMIN alone plus the device mounts is not enough for this workload — QEMU’s device setup and the TAP/bridge configuration need more than the capability grant, and narrowing it down to a minimal set was a research project we did not have room for.

What we did instead was contain the blast radius:

• Computers run in their own namespace (computer), never alongside the control plane.
• They run on a dedicated node pool (oousernp) via nodeSelector, so a compromised or misbehaving VM pod shares nodes only with other VM pods.
• The admin services — API, database, dashboard — are on a different pool entirely.

Picking a VM size, the hard way

Nested virtualisation on Azure is a size property, not a subscription setting. Some families expose it, some do not, and the failure is not a helpful error — it is /dev/kvm simply not existing on the node.

The scar tissue is in the chart, commented out:

            # AKS
            # - name: KVM
            #   value: "N"
            # - name: USER_PORTS
            #   value: "Y"

KVM: "N" tells the QEMU wrapper to fall back to software emulation. We tried it on a size without nested virt, confirmed that a Windows install takes roughly forty minutes instead of eight, and moved to a size that supports it.

Check for /dev/kvm on a node before building anything on top:

kubectl debug node/<node> -it --image=busybox -- ls -l /host/dev/kvm

Thirty seconds, and it saves a day.

Deployment → StatefulSet

The first version used a Deployment. It was wrong, and the migration commit is dated 8 April.

A Deployment models interchangeable replicas. Any pod can serve any request; if one dies, another takes over; storage, if any, is incidental. A virtual computer is none of those things:

• It has an identity. “The machine running test 47” is a specific machine.
• It has a disk that is the machine. The PVC holds the qcow2 image — installed Windows, installed software, a signed-in Teams session. Lose it and you lose forty minutes.
• Replacement is not recovery. A new pod on a fresh disk is not the same computer. It is an unprovisioned one.

StatefulSet gives you stable identity, stable per-replica storage via volumeClaimTemplates, and a headless Service for stable DNS:

spec:
  serviceName: "{{ .Values.computer_name }}-headless"
  ...
  volumeClaimTemplates:
  - metadata:
      name: "hdd"
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: "default"
      resources:
        requests:
          storage: {{ .Values.dockur.storageSize }}

The mental model shift: a VM is a pet with an API, not cattle. Kubernetes is built for cattle. StatefulSet is the pet accommodation.

The annotation that saves your install

      annotations:
        # Will prevent the cluster autoscaler from evicting this pod - ???
        cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

The ??? is mine, and it is honest — I was not certain it was sufficient when I wrote it.

The problem: the cluster autoscaler consolidates. It sees an under-utilised node, decides to drain and remove it, and evicts the pods. For a stateless web service that is a rescheduling event nobody notices. For a pod that is twenty-five minutes into a Windows installation, it is a total loss — and the replacement starts from zero.

Worse, it is intermittent and load-dependent, so it shows up as “sometimes provisioning takes twice as long” long before anyone connects it to the autoscaler.

safe-to-evict: "false" tells the autoscaler to leave the node alone while the pod is on it. The reason for the ???: this prevents autoscaler eviction. It does not prevent node upgrades, spot reclamation, or a manual drain. The complete answer also involves not using spot instances for computers and controlling when node image upgrades happen — but the annotation is the 90% fix and it is one line.

QEMU’s memory tax

This one produced a genuinely confusing incident.

raw_ram = ram
ram_bytes = parse_size(raw_ram)
ram_limit_bytes = int(ram_bytes * 1.15)  # Add 15%
ram_limit = format_size(ram_limit_bytes, binary=False).replace("B", "").replace(" ", "")

If the guest gets 24 GB, the pod needs more than 24 GB. QEMU itself has overhead: device emulation, the graphics buffer, its own heap. Set the pod limit equal to the guest allocation and the container gets OOM-killed — not at boot, but later, under memory pressure inside the guest, when QEMU actually touches all the pages it promised.

The symptom is the worst kind: a VM that runs fine for twenty minutes and then vanishes, mid-test, with OOMKilled in the pod status and nothing whatsoever in the guest logs, because the guest was not aware it was dying.

15% headroom, derived empirically. The chart reflects the same idea in its defaults:

dockur:
  min_cpu_cores: 3
  min_ram_size: 24G
  max_cpu_cores: 4
  max_ram_size: 30G
  storageSize: 100Gi

Request 24 GB, limit 30 GB. Requests are what the scheduler packs against; limits are the ceiling before the kernel intervenes. For a workload whose real usage varies with what the guest is doing, that gap is deliberate.

Labels as a query interface

Every computer carries its provenance:

  labels:
    os: "windows"
    benchmark-id: "{{ .Values.benchmark_id }}"
    test-id: "{{ .Values.test_id }}"
    computer-name: "{{ .Values.computer_name }}"
    os-version: "{{ .Values.dockur.version }}"
    cpu-cores: "{{ .Values.dockur.max_cpu_cores }}"
    ram-size: "{{ .Values.dockur.max_ram_size }}"
    storage-size: "{{ .Values.dockur.storageSize }}"

This looked like over-labelling. It was not. It makes the cluster queryable in the terms the platform thinks in:

kubectl get pods -n computer -l benchmark-id=abc123          # everything in one benchmark
kubectl delete statefulset -n computer -l test-id=xyz789     # clean up one test
kubectl get pods -n computer -l os=windows,ram-size=30G      # find the heavy ones

The control plane uses exactly these selectors to find, monitor and delete resources — no separate registry, no bookkeeping table that can drift from reality. The cluster is the source of truth about what exists, and labels are the index.

Putting resource sizing in labels is arguably a misuse — labels are for identity, not configuration. In practice being able to ask “which computers are running with 30 GB” without parsing pod specs paid for the impurity many times over.

What you have now

A Helm chart that provisions a real Windows 11 machine, with a persistent disk and a stable identity, as a Kubernetes workload. Deploy it N times and you have N desktops.

And it takes forty minutes each, because every one of them installs Windows from scratch.

That is the next problem, and solving it was worth more than everything else in this post combined. But first: how a 6 GB ISO gets to the pod in the first place.