We tried every vision model we could get our hands on

In early March 2025 the honest state of the art for “look at a screenshot and tell me where to click” was: nobody knew. Every week brought a model claiming GUI grounding. So we built a models/ directory and worked through them.

models/
├── gemma3/            # local, Ollama
├── llama3.2-vision/   # local, Ollama
├── llama3.3/          # local, Ollama
├── magma/             # microsoft/Magma-8B, transformers, image + video
├── mistral/           # local, Ollama
├── openai/            # GPT-4o via API
└── phi4/              # local, Ollama

Each one is a tiny self-contained project — a README, a pyproject.toml, a script that loads the model and asks it about a screenshot of a Windows desktop. Deliberately minimal, because the point was to answer one question per model as cheaply as possible.

The question we were actually asking

Not “is this model good.” The question was narrower and more useful:

Given a 1920×1080 screenshot of a Windows desktop, can this model tell me where to click, precisely enough that a mouse event lands on the intended control?

That is a spatial grounding question, and it is much harder than description. Almost every model in that list can tell you “there is a Microsoft Teams icon in the taskbar.” Far fewer can tell you it is at roughly (412, 1041), and fewer still are consistent about it across a hundred screenshots.

Magma, and the video experiment

Magma-8B was the most interesting thing we tested, because it is explicitly built as a foundation model for multimodal agents rather than for captioning. The image path is unremarkable to drive:

convs = [
    {"role": "system", "content": "You are agent that can see, talk and act."},
    {"role": "user", "content": "<image_start><image><image_end>\nWhat time is it?"},
]
prompt = processor.tokenizer.apply_chat_template(convs, tokenize=False, add_generation_prompt=True)
inputs = processor(images=[image], texts=prompt, return_tensors="pt")

It answers The time displayed on the computer is 12:27 PM. correctly, from a screenshot of a Windows taskbar clock. Which is a genuinely good result for an 8B model reading small rendered text.

The part we spent longer on was the video path — feeding a screen recording instead of a still frame. The intuition is appealing: a computer-use agent’s hardest question is “did my last action do anything?”, and that is inherently a question about change over time. A single frame cannot tell you whether a menu is opening or closing.

What we found:

• For step-wise control it did not pay off. Deciding the next click from a video costs far more tokens and latency than deciding it from the current frame, and the extra temporal context rarely changed the answer. The information you need to click a button is in the frame with the button in it.
• For change detection it was genuinely better. “Did the window open?” “Is this still loading?” — those are exactly the questions a single frame is bad at and a short clip is good at.
• But we already had a cheaper answer. Two consecutive screenshots and a targeted “has anything relevant changed?” prompt captured most of the benefit at a fraction of the cost.

The video branch stayed in the repo as a reference implementation. It never made it into the production loop.

The local models

Gemma 3, Llama 3.2 Vision, Llama 3.3, Mistral and Phi-4 all went through Ollama, because for a bake-off you want the cheapest possible harness:

ollama pull llama3.2-vision

They are useful for the parts of the loop that are not spatial: summarising what a step accomplished, deciding whether a plan is finished, condensing a long action history. Those are text tasks wearing a vision costume, and a local 8B model does them acceptably at zero marginal cost.

For grounding a click on a dense enterprise UI in March 2025, none of the local models we tried were reliable enough to build on.

Where we landed

The conclusion held for the rest of the project, and it is slightly deflating:

A general frontier model plus a good set-of-mark parser beat every specialised GUI model we tested.

Instead of asking a model for pixel coordinates, we ran the screenshot through a detection and captioning pipeline that finds interactive elements, numbers them, and overlays the numbers on the image. Then the question to the LLM becomes:

Which numbered element should I click?

14 instead of (817, 442). A discrete choice from a visible list instead of a continuous regression over pixel space. That reframing did more for reliability than any model swap, and it makes the whole system robust to model choice — which turned out to matter more than picking the best model.

The abstraction that made this survivable

Because we expected to keep swapping, the LLM layer was a thin unified client from very early on:

agents/core/clients/llm/
├── azure_openai.py
├── anthropic.py
├── local_ollama.py
└── helpers.py

One interface, several backends, provider chosen by config. Boring, and it paid for itself repeatedly: re-running a scenario against a different provider was a config change, not a refactor. When Azure OpenAI later moved from API keys to managed identity, exactly one file changed.

What I would tell you if you are starting this today

1. Test grounding, not description. Any model can describe a screenshot. Ask for a click target and measure whether the click lands.
2. Reframe continuous into discrete. Numbered candidate elements beat raw coordinates by a wide margin, and the gap does not close as models improve — it just makes both options better.
3. Keep the provider layer thin and swap often. The best model for this changes every few months. Design so that finding out costs you an afternoon.
4. Video is not free context. It is a real capability with a real bill.

Next: the parser that made the discrete framing possible, and why it had to be a separate server rather than a library.