Forty minutes to ninety seconds: VolumeSnapshots as golden images for VMs

Do the arithmetic on a benchmark run.

Twenty scenarios. Five repeats each, because a single run is not a result. One hundred Windows machines. At roughly forty minutes to install Windows and provision it, that is sixty-six hours of compute before a single agent action happens.

That is not a test suite. That is a weekend, and a bill.

The fix is the oldest idea in systems administration: install once, image it, clone the image. What is interesting is doing it in Kubernetes, where the disk is a PVC and the imaging primitive is a CSI VolumeSnapshot.

Turning snapshots on

AKS does not ship snapshot support enabled. The CRDs and controller are separate, and the VolumeSnapshotClass is yours to create:

# CRDs — once per cluster
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v6.2.1/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v6.2.1/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v6.2.1/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml

# Snapshot controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v6.2.1/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v6.2.1/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml

Then the class, which binds the generic API to the Azure Disk CSI driver:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: azuredisk-snapclass
driver: disk.csi.azure.com
deletionPolicy: Delete

Taking the image

Install Windows once, provision it, sign in to the applications that need a human, then snapshot the disk:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: windows-snapshot-3
  namespace: computer
spec:
  volumeSnapshotClassName: azuredisk-snapclass
  source:
    persistentVolumeClaimName: hdd-test-1-0

Note the source PVC name — hdd-test-1-0. That naming is not decorative, and it is about to cause a bug.

This is also where the manual step from the bootstrap post gets amortised. Signing in to a Microsoft account with MFA is deliberately not automatable. It does not need to be: a human does it once, into the machine that becomes the golden image, and every clone inherits the signed-in session.

One chart, two topologies

Here is the pattern I would take to another project unchanged. A single boolean switches the chart between two entirely different storage models.

Fresh install path — a per-replica PVC from volumeClaimTemplates, plus the initContainer that downloads the ISO:

      {{- if not .Values.dockur.useSnapshot }}
      initContainers:
        - name: my-init-container
          image: {{ .Values.initContainers.image }}
          ...
      {{- end }}

  {{- if not .Values.dockur.useSnapshot }}
  volumeClaimTemplates:
  - metadata:
      name: "hdd"
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: "default"
      resources:
        requests:
          storage: {{ .Values.dockur.storageSize }}
  {{- end }}

Snapshot path — an explicit PVC cloned from a snapshot, and no initContainer at all:

{{- if .Values.dockur.useSnapshot }}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: "hdd-{{ .Values.computer_name }}-from-snapshot"
  namespace: {{ .Values.namespace }}
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: default
  resources:
    requests:
      storage: {{ .Values.dockur.storageSize }}
  dataSource:
    name: {{ .Values.dockur.snapshotName }}
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
{{- end }}

dataSource is the whole trick. The CSI driver provisions the new disk from the snapshot, and Azure does it as a lazy copy — the PVC is usable almost immediately, with blocks faulted in as they are read, rather than after a full copy of 100 GB.

The result: the pod starts, the qcow2 is already there, Windows boots normally. No installation. No provisioning. No initContainer. Ninety seconds instead of forty minutes, and a machine that is byte-identical to every other clone.

Deploying is the same command either way — the caller sets one flag:

useSnapshot = hdd_snapshot_name != ''

helm_cmd = [
    "helm", "upgrade", "--install", release_name, chart_path,
    "--set", f"computer_name={computer_name}",
    "--set", f"dockur.useSnapshot={useSnapshot}",
    "--set", f"dockur.snapshotName={hdd_snapshot_name}",
    ...
]

Bug one: StatefulSet PVC naming

Both fixes landed on 22 April, two minutes apart. Here is the first.

A StatefulSet’s volumeClaimTemplates generates PVCs with a mandatory name format:

<template-name>-<statefulset-name>-<ordinal>

So a template called hdd on a StatefulSet called test-1 produces hdd-test-1-0. This is not a convention you can opt out of; it is how the controller finds a replica’s storage across rescheduling.

Our hand-written snapshot PVC was originally named:

name: "{{ .Values.computerName }}-pvc-from-snapshot"

Which reads perfectly well and is wrong in a way that has consequences beyond the pod. Fixed to match the convention:

name: "hdd-{{ .Values.computer_name }}-from-snapshot"

Aligning the name matters because everything else in the system — the snapshot tooling, the API’s lookup logic, the cleanup routines — reasons about disks through that hdd-<computer>-… prefix. One resource using a different scheme means every consumer needs a special case, and the ones that do not have it fail silently on that resource.

The general form: when a controller imposes a naming convention, adopt it for your hand-written resources too, even where nothing forces you to. The convention is an interface.

Bug two: the substring match

The second fix is one line, and it is my favourite kind of bug — completely correct-looking, and wrong in a way that only appears after you have been running for a few weeks.

The API lists the snapshots belonging to a computer:

for snap in snapshot_list.get("items", []):
    pvc_name = snap["spec"]["source"].get("persistentVolumeClaimName")
-   if pvc_name and name in pvc_name:
+   if pvc_name and pvc_name.startswith(f"hdd-{name}-"):

name in pvc_name is a substring test. Computers are named test-1, test-2, … test-10, test-11.

Ask for the snapshots of test-1 and you get the snapshots of test-1, test-10, test-11, test-12, and every other computer whose name happens to contain test-1 as a substring.

The consequences are worse than a messy list. The UI presented all of them as restore candidates, so it became possible to provision test-1 from a snapshot of test-11 — a machine in a completely different state, from a different scenario, with different applications open. The run would proceed and produce a result. A wrong result, with no error anywhere.

startswith(f"hdd-{name}-") anchors both ends: the hdd- prefix and the trailing - that terminates the name. test-1 no longer matches hdd-test-10-0.

What a golden image costs you

Snapshots are not free, and the costs are not the obvious one.

They go stale. The image contains the software versions, the certificates, the signed-in sessions and the OS patch level from the day it was taken. Two months on, that Teams build is old enough that you may be benchmarking against a UI nobody uses. You need a rebuild cadence, and you need to record which snapshot a result came from — otherwise “the agent got worse in June” is unattributable between the agent, the model, and the frozen application.

They contain secrets. This is the one to think about carefully. Our golden image has a signed-in Microsoft account in it — which means it contains refresh tokens on disk. The snapshot is therefore a credential. It needs the same access control as one, and a test account rather than anything real. We used dedicated test users; if you take this pattern, do that from the start, not after someone asks an uncomfortable question in a review.

Storage adds up. Each snapshot is the full logical size of a 100 GB disk, and “take a snapshot before each experiment” is a habit that becomes an invoice without a retention policy.

The result

	Fresh install	From snapshot
Time to usable desktop	~40 min	~90 s
initContainer	required, downloads 6 GB	not created
Provisioning	full chain, every time	already done
State	stock Windows	fully provisioned, signed in
Reproducibility	subject to installer variance	byte-identical every clone

That last row is worth as much as the time saving. A fresh install depends on package mirrors, download order, and whatever Windows Update decided to do that morning. Two “identical” machines are not identical, and when you are measuring a stochastic agent you cannot afford a second source of variance in the environment.

A snapshot clone is the same machine every time. That is what makes a benchmark a benchmark.

Next: the control plane that decides which snapshot to use — and streams helm upgrade to a browser while it does it.