6 minOpenOperatoraiagentspython

Reading the screen: OmniParser as a service

Turning a screenshot into a numbered list of clickable things — YOLOv8 for detection, Florence-2 for captions, OCR for text. And why it had to be a server, not a library.

Hand a multimodal LLM a raw 1920×1080 screenshot and ask where to click, and you get an answer with the confidence of a satnav and the accuracy of a weather forecast. It knows the Send button exists. It is fuzzy about where.

The fix is not a better model. It is not asking the question.

Set-of-mark, in one paragraph

Instead of asking for coordinates, pre-process the screenshot: detect every interactive element, draw a numbered box around each one, and hand the model the annotated image plus a list of what each number is. Now the question is “which number should I click?” — a choice from maybe forty visible options, not a regression over two million pixels. The system resolves number → bounding box → centre point → mouse event, deterministically, in code the model never touches.

OmniParser does the annotation. Three stages:

  1. Detection — a YOLOv8 model trained on UI elements finds the interactive regions: buttons, inputs, icons, list rows, tabs.
  2. Captioning — Florence-2 describes each detected region, so the model gets “chat list item, Marie Doe, 14” instead of “box #23.”
  3. OCR — text extraction, because a great deal of what matters on a desktop is literally written on it.

The output is an annotated image and a structured list. The LLM sees both.

Why it became a server

The first version was a library import inside the agent. It lasted about a week, and it broke on three independent axes.

GPU affinity. Detection plus captioning wants a GPU. The agent — an orchestration loop making HTTP calls and LLM calls — wants nothing of the sort. Fusing them means every agent process needs a GPU, which is both expensive and often impossible on the machine where you actually want to run the agent.

Model load time. Loading YOLOv8 and Florence-2 into VRAM takes real seconds. As a library, every agent process pays that on every start. As a server, one process pays it once and amortises it across every request from every agent for the lifetime of the deployment. When your benchmark runs twenty agents in parallel, that is the difference between twenty model loads and one.

Development ergonomics. Once it is an HTTP service you can run the agent on your laptop and point OMNIPARSER_URL at a GPU box, or at the cluster, or at a colleague’s machine. That single environment variable removed a category of “it only works on the GPU machine” friction that was quietly killing iteration speed.

The interface is deliberately dull:

class ParseRequest(BaseModel):
    base64_image: str

@app.post('/parse')
async def parse(parse_request: ParseRequest):
    start = time.time()
    dino_labled_img, parsed_content_list = omniparser.parse(parse_request.base64_image)
    logger.info(f'time: {time.time() - start:.2f} seconds')
    ...

Base64 in, annotated image plus element list out. Latency logged on every call, which matters more than it sounds — parse time is the floor on your agent’s loop time, and if it drifts you want to see it drift.

The weights problem

The models are hundreds of megabytes. Three obvious options:

  • Bake them into the image. Slow builds, enormous image, and every tag change re-pushes the weights.
  • Mount them from a volume. Now the service has an external prerequisite and a new failure mode for anyone running it locally.
  • Fetch on first boot.

We took the third:

weights_dir = path.join(root_dir, 'weights')

if path.exists(weights_dir) == False:
    os.makedirs(weights_dir, exist_ok=True)
    logger.info('weights folder not found, downloading models...')
    download_omniparser(weights_dir)
    logger.info('models downloaded successfully!')

docker run and it works. First start is slow, subsequent starts are not, because the directory persists in the volume. The container image stays small enough to iterate on.

The trade-off is real and worth stating: first boot now depends on the network. In an air-gapped environment this is the wrong design, and in a cluster where pods are recreated frequently you want that weights/ directory on a persistent volume or you re-download constantly. It was the right call for our mix of local development and long-lived GPU deployment; it would be the wrong call for a fleet of short-lived pods.

Degrade, do not crash

'device': 'cuda' if cuda.is_available() else 'cpu',

One line, and it means the service starts on a laptop. Slowly — CPU inference on a 1080p screenshot is not fun — but it starts, it answers, and you can develop the agent against it without a GPU anywhere in the picture.

The alternative, asserting CUDA at startup, is defensible and I would argue against it here. The agent developer does not care about parse latency; they care that /parse returns a plausible element list so they can work on the planner. Being usable but slow on a laptop is worth more than being correct or absent.

The threshold that surprised me

'BOX_TRESHOLD': 0.05,

That is a very permissive detection confidence. My instinct was to tighten it — fewer false positives, cleaner annotations, less noise for the model.

That instinct was wrong, and it took a while to see why. The two failure modes are not symmetric:

  • False positive — a numbered box around something that is not interactive. The LLM reads the caption, sees it is irrelevant, ignores it. Cost: a few tokens.
  • False negative — the button the agent needs is not in the list at all. The LLM cannot select what it cannot see. It picks the closest wrong thing, the action fails, the step retries, the executor burns its iteration budget, and eventually the whole plan gets replanned around an obstacle that does not exist. Cost: minutes and dollars.

So: over-detect and let the language model filter. It is extremely good at filtering and completely unable to hallucinate a missing element into existence.

Where it sits in the loop

screenshot ──▶ /parse ──▶ annotated image + element list ──▶ LLM ──▶ "click 14"
                                                                       │
                              element[14].bbox ──▶ centre ──▶ mouse ◀───┘

The model never emits a coordinate. It emits an index. Everything spatial happens in ordinary, testable, deterministic Python.

That is the design principle worth extracting: give the model the decision, keep the arithmetic. Every place we let an LLM compute rather than choose, we regretted it.

Next: four generations of agent in four weeks — what happens between “which element should I click” and “the task is done.”