You cannot debug an agent from logs

For the first ten days, debugging the agent looked like this:

INFO  planner    - plan v1: [open Teams, find chat list, click first chat, ...]
INFO  executor   - step 1: open Teams
INFO  executor   - action: mouse_left_click(x=412, y=987)
INFO  executor   - action: mouse_left_click(x=412, y=987)
INFO  executor   - action: mouse_left_click(x=412, y=987)
INFO  executor   - step 1 not complete after 3 iterations
INFO  replanner  - replanning...

Three identical clicks. Why? Pick your theory:

The click landed on the taskbar but four pixels below the icon.
It landed correctly, Teams is launching, and it takes eleven seconds, so each screenshot still shows the same desktop.
Teams launched behind another window.
A Windows Update toast appeared over the taskbar between the screenshot and the click.
The click went to the right coordinates on the wrong monitor.

The logs distinguish between exactly none of these. I spent the better part of a day on one of them. Then we spent an afternoon wiring up a live view of the desktop and the answer was visible in four seconds: a “Your organisation requires a restart” dialog had focus, and every click was landing on a window that had no idea what to do with it.

The chain

Three pieces, none of them clever:

VNC server (in the guest) ──TCP 5900──▶ websockify ──WebSocket──▶ noVNC (browser)

VNC server in the guest exposes the framebuffer over RFB, an old, boring, extremely well-supported protocol.
websockify translates. RFB is raw TCP; browsers cannot open raw TCP sockets. websockify accepts a WebSocket connection, unwraps each frame, and pipes the bytes to the TCP socket — and back. That is its entire job, and it is the only reason browser-based remote desktop exists at all.
noVNC is an RFB client written in JavaScript that paints into a <canvas>.

The QEMU wrapper already exposes a web console on port 8006 for exactly this reason. We ran our own chain alongside it because we needed the stream embeddable in our own UI rather than as a standalone page — and, later, because we needed it to work for N simultaneous machines behind an ingress, which is a much harder problem than it looks.

What changed

Concretely, what became diagnosable in seconds instead of hours:

Focus theft. The single most common failure in desktop automation. Something — an update prompt, an antivirus balloon, a Teams “you’ve been added to a team” toast — takes focus between the screenshot and the action. The agent’s model of the world is now one frame stale and every subsequent action is wrong. Invisible in logs. Blindingly obvious on screen.

Off-by-a-few clicks. The parser returns a bounding box, we click its centre, and for most elements that is fine. For a few — tall list rows with the label at the top, controls with large invisible padding — the centre is not the hit target. You only ever notice by watching the cursor land.

Loading states. The agent screenshots at time t, the app finishes painting at t+400ms, and the agent concludes nothing happened. Watching it live turns this from a mystery into an obvious timing problem with an obvious fix.

“It worked, actually.” Several runs marked failed had done the task correctly and then failed the verification step. Without a live view we were debugging the executor when the bug was in the validator.

The real lesson

Standard observability practice says: instrument, emit structured logs, aggregate, query. We did all of that — Elasticsearch and Kibana went in early and stayed.

But logs are a projection. They contain what you thought to record. In an agent whose entire input is a rendered 2D image, the highest-bandwidth debugging signal is the image, and no amount of logger.info recovers it.

This is also why video recording ended up as a first-class artifact of every test run, uploaded to blob storage alongside the structured results. When a benchmark reports 3 of 5 runs failed, the only thing that ever explains it is watching the three failures. A trace tells you where; the video tells you why.

The part that did not survive

The naive version of this — one VNC server, one websockify, one noVNC page, per machine — worked beautifully for one machine and became a liability at twenty. Every desktop meant an extra proxy pod, an extra Service, and an extra ingress route, purely to translate one protocol into another.

Two months later we deleted all of it and replaced it with a single shared gateway that computers register themselves with when they are provisioned. That post is here — it is a good example of a design that is obviously right at n=1 and obviously wrong at n=20, with no warning in between.

But for the first two months, three processes in a row and a canvas element were worth more than every log line we wrote.