7 minOpenOperatorkubernetesterraformazureobservability

Benchmarking AI agents: node pools, job queues, and cost per step

Five workload classes with five different scaling shapes, a bounded queue that fans tests into Kubernetes Jobs, and the reason your autoscaler makes terrible decisions about 30 GB virtual machines.

The whole platform exists to answer one question:

Did this change make the agent better?

Answering it means running N scenarios × M repeats, recording pass rate, cost and duration for each, and comparing runs. That is a benchmark, and building one that runs on Kubernetes surfaces a problem that does not exist at small scale: your workloads have wildly different scaling shapes, and a single node pool serves none of them well.

Five workload classes

The Terraform ended up with five pools plus the system one, and the interesting part is that no two share a scaling policy:

resource "azurerm_kubernetes_cluster_node_pool" "adminnp" {
  name       = "ooadminnp"
  node_count = var.admin_node_count      # fixed — the control plane
}

resource "azurerm_kubernetes_cluster_node_pool" "storagenp" {
  name                = "oostoragenp"
  min_count           = var.storage_min_node_count
  max_count           = var.storage_max_node_count
  enable_auto_scaling = true
  os_disk_size_gb     = 300
  node_labels = { "longhorn.io/storage" = "true" }
  node_taints = ["longhorn=true:NoSchedule"]
}

resource "azurerm_kubernetes_cluster_node_pool" "usernp" {
  name                = "oousernp"
  min_count           = var.user_min_node_count
  max_count           = var.user_max_node_count   # 20
  enable_auto_scaling = true
  max_pods            = 30
}
Pool Shape Why
ooadminnp fixed, small API, UI, Mongo, broker — always on, never bursty
oostoragenp autoscaling, 300 GB disks, tainted Longhorn replicas; sized for disk, not CPU
oobenchnp fixed benchmark orchestrator Jobs — one per run, tiny
ooagentnp fixed agent Jobs — CPU-light, they wait on LLM calls
oousernp autoscaling 1→20, huge VMs the virtual computers

The variables carry their own cost annotations, which I recommend as a habit:

variable "user_vm_size" {
  # For testing the deployment (allows nested virtualization)
  # default = "Standard_D2_v5"  # 2 cores, 8 GB   -> $70/month,  $2.3/day
  # default = "Standard_D4_v5"  # 4 cores, 16 GB  -> $140/month, $4.6/day
  # For real environment
  default   = "Standard_D16_v5" # 16 cores, 64 GB -> $560/month, $18.5/day
}

Twenty of those at full scale is roughly $370 a day. Having that number in the variable file, where you change the value, is worth more than any dashboard. It is also the number that makes min_count = 1 non-negotiable.

Why one pool does not work

The temptation is a single autoscaling pool. Three reasons it fails here.

Nested virtualisation is a size property. Only some VM families expose /dev/kvm to guests. The computers must land on those; nothing else needs to. Mixing them means either overpaying for every pod or discovering the forty-minute install on a node that cannot do KVM.

Autoscaler heuristics are tuned for stateless pods. It assumes a pod is cheap to move: evict, reschedule, done. A computer pod is 30 GB of RAM, a 100 GB disk, and possibly a Windows installation in progress. Consolidation, which is a good default, is destructive here — which is why every computer carries cluster-autoscaler.kubernetes.io/safe-to-evict: "false".

Blast radius. Computers run privileged with hostPath /dev/kvm. Keeping them on their own pool means they never share a node with the API, the database, or anything holding a credential.

There is also max_pods = 30 on the user pool, which is deliberate. The default is higher, and it does not matter how many pod slots a node advertises when each pod wants 30 GB of RAM — a 64 GB node fits two. Capping pod density stops the scheduler from making optimistic packing decisions it cannot honour.

The benchmark, as a queue

A benchmark is a Kubernetes Job that spawns test Jobs. The fan-out is a bounded worker pool over an asyncio.Queue:

class TestRunnerQueue:
    def __init__(self, jobs: list[TestJob], concurrency_limit: int, wsc: WSClient, ws_path: str):
        self.queue = asyncio.Queue()
        for job in jobs:
            self.queue.put_nowait(job)
        self.concurrency_limit = concurrency_limit

    async def run_all(self):
        workers = [asyncio.create_task(self.worker(i)) for i in range(self.concurrency_limit)]
        await asyncio.gather(*workers)

    async def worker(self, worker_id: int):
        while not self.queue.empty():
            job = await self.queue.get()
            try:
                await self.handle_test_job(...)
            except Exception as e:
                logger.exception(f"[Worker {worker_id}] Job {job.test_config_name} #{job.run_number} failed: {e}")
            finally:
                self.queue.task_done()

Fifty jobs, concurrency_limit workers, each pulling the next job when it finishes. Standard, and correct here for a specific reason: the concurrency limit is a money dial. Ten workers means up to ten Windows VMs at once, which is a known hourly rate. Unbounded fan-out means the autoscaler discovers twenty nodes and you discover the invoice.

The try/except/finally matters more than it looks. One test failing must not kill the worker — with fifty jobs in flight, an unhandled exception in worker 3 silently reduces your throughput and leaves jobs unprocessed with no error at the benchmark level. Catch, log, continue, and let the result record the failure.

Each test Job then does the full sequence:

from core.k8s import (deploy_computer, deploy_agent, is_computer_running,
                      wait_until_agent_job_completes, register_computer_with_guacamole,
                      delete_test)
from core.oo_servers import check_oo_servers
from core.functions import FunctionExecutor, evaluate
from core.storage import AzureBlobStorageClient

Provision a computer (from a snapshot), wait for it, verify the control servers answer, register it with the remote-desktop gateway, run the scenario’s setup hooks, deploy the agent, wait, run teardown and evaluation, upload artifacts, tear everything down.

check_oo_servers deserves a mention. Before the agent starts, the test Job health-checks all seven control servers inside the guest. Without that gate, a guest that booted but whose scheduled tasks did not fire produces a test failure rather than an infrastructure failure — and those get counted as agent regressions, which corrupts exactly the number you built the platform to measure.

Recording results

Every run produces four kinds of output:

  • Structured results in MongoDB — the test instance, its steps, per-step validation, cost, duration.
  • Video in Azure Blob Storage — the recording of the desktop.
  • LLM traces in Langfuse — every prompt, completion and token count.
  • Logs in Elasticsearch, with Grafana for cluster metrics.

sum_costs aggregates the per-step spend into a run total. As the evaluation post argues, cost is a result, not telemetry: a scenario that passes reliably at $4 per run and eleven minutes is a different finding from one that passes at $0.40 in ninety seconds, and only one of them is deployable.

Langfuse is the piece I would add earliest next time. When a run goes wrong, the question is always “what did the model actually see?” — and a trace showing the exact prompt, including the annotated screenshot, answers it in seconds. Video tells you the desktop was in a strange state; the trace tells you the model was shown something misleading.

Longhorn, and why not just Azure Disk

Storage got its own pool, its own labels and its own taint:

defaultSettings:
  systemManagedComponentsNodeSelector: longhorn.io/storage:true
  taintToleration: longhorn=true:NoSchedule

global:
  nodeSelector:
    longhorn.io/storage: "true"
  tolerations:
    - key: "longhorn"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

The taint is doing real work: longhorn=true:NoSchedule means only pods that explicitly tolerate it land on storage nodes. Storage nodes have 300 GB disks and exist to hold replicas — filling them with agent pods would be an expensive mistake.

The honest assessment of Longhorn: it gave us node-local volume performance and a storage layer we controlled, which matters when the volume is a VM disk being written to constantly. It also added a distributed storage system to operate, and VolumeSnapshots went through the Azure Disk CSI driver anyway. If I were starting over I would begin with the managed CSI driver and only reach for Longhorn when a specific measured problem demanded it. We reached for it slightly ahead of the evidence.

What a benchmark actually tells you

After all of this, a benchmark run produces a table:

Scenario Runs Passed Avg cost Avg duration
teams / chat_switch 5 4 $1.82 6m 40s
teams / summarize 5 2 $4.15 11m 20s
notepad / write_file 5 5 $0.31 1m 50s

Which is worth every hour spent on hypervisors in pods and byte-level protocol decoders, because it is the first artifact in the whole project that supports a decision. “Summarise is our weak scenario, it is also our most expensive, and the two are probably the same problem” is actionable. “The agent worked when I tried it” is not.

Next, and last: what five months of this actually taught me.