MCP for mouse and keyboard: a desktop as a tool server

In March 2025 the Model Context Protocol was young enough that most examples were “here is a filesystem server” and “here is a database server.” We built something weirder: an MCP server whose tools are mouse_move, mouse_left_click and keyboard_type, running inside a Windows VM, exposing the desktop itself as a tool surface.

The tools

Six, and they map to physical acts rather than intentions:

ALL_TOOLS = [
    types.Tool(
        name="mouse_move",
        description="Moves the mouse cursor to the specified screen coordinates.",
        inputSchema={
            "type": "object",
            "required": ["x", "y"],
            "properties": {
                "x": {"type": "integer", "description": "The x coordinate (absolute) to move to."},
                "y": {"type": "integer", "description": "The y coordinate (absolute) to move to."},
            },
        },
    ),
    types.Tool(name="mouse_left_click",   description="Performs a single left mouse click.",
               inputSchema={"type": "object", "properties": {}}),
    types.Tool(name="mouse_double_click", description="Performs a double left mouse click.",
               inputSchema={"type": "object", "properties": {}}),
    ...
]

Two things about this design that were arguments at the time.

Move and click are separate tools. There is no click_at(x, y). The model moves the cursor, then clicks. That is more round trips, and it is right: it matches how a GUI actually behaves. Hover states exist. Tooltips appear. Drag-select needs move-press-move-release. A fused click_at cannot express any of that, and the moment you need one of them you are bolting on a second, subtly different tool.

mouse_left_click takes no arguments. It clicks wherever the cursor is. The alternative — every click carrying coordinates — sounds more convenient and quietly encourages the model to teleport the pointer, which produces interaction sequences no application was ever tested against.

mouse_scroll is the outlier, and it is honest about the domain being messy:

inputSchema={
    "required": ["direction", "amount", "delay", "steps"],
    "properties": {
        "direction": {"type": "string", "description": "'up', 'down', 'left', 'right'."},
        "amount":    {"type": "integer"},
        "delay":     {"type": "number", "description": "Delay (in seconds) between consecutive scrolls."},
        "steps":     {"type": "integer", "description": "Number of times to apply the scroll for smoother motion."},
    },
}

delay and steps exist because a single large scroll event and twenty small ones are different things to a modern application. Virtualised lists — which is every chat client, including the one we were driving — render lazily. Jump 2000 pixels in one event and the app has nothing to paint; the agent screenshots an empty region and concludes the list ended. Twenty steps of 100 pixels with a small delay gives the renderer time to keep up.

That parameter exists because of a specific bug where an agent decided a chat history was three messages long.

The screenshot is a resource, not a tool

This is the decision I would defend hardest.

@app.list_resources()
async def list_resources() -> list[types.Resource]:
    return [
        types.Resource(uri="binary:///screenshot", name="Screenshot", mimeType="image/png"),
    ]

@app.read_resource()
async def handle_read_resource(uri: AnyUrl) -> list[ReadResourceContents]:
    if str(uri) == "binary:///screenshot":
        image_bytes = get_screenshot_with_cursor()
        return [ReadResourceContents(content=image_bytes, mime_type="image/png")]

MCP distinguishes tools (model-invoked, cause effects) from resources (application-managed context, read into the conversation). The obvious implementation makes get_screenshot a tool. We made it a resource, and the distinction turns out to matter:

• Tools change the world; resources describe it. Taking a screenshot has no side effects. Modelling it as an action muddies the model’s own reasoning about what it has done versus what it has seen.
• Context management belongs to the host, not the model. The application decides when to refresh the screenshot — before every decision, after every action, once per iteration. As a tool, that policy is delegated to the model, and models are erratic about it: some screenshot obsessively and blow the context window, some forget and reason about a stale frame for five turns.
• Images are expensive. A screenshot is a large number of tokens. You want a deterministic policy for how often that cost is paid.

The general principle: if the model calling it more or fewer times changes only cost and not correctness, it is a resource. If calling it twice does something different from calling it once, it is a tool.

The cursor is not in the screenshot

Here is the detail people will actually steal.

if user_platform == "Windows":
    cursor_path = os.path.join(os.path.dirname(__file__), "img", "cursor.png")
    screenshot = pyautogui.screenshot()
    cursor_x, cursor_y = pyautogui.position()
    cursor = Image.open(cursor_path)
    cursor = cursor.resize((int(cursor.width * 2), int(cursor.height * 2)))
    screenshot.paste(cursor, (cursor_x, cursor_y), cursor)
    screenshot.save(buffer, format="PNG")

Windows screen capture does not include the mouse cursor. The cursor is drawn by a separate compositing path — historically a hardware overlay — and standard capture APIs give you the framebuffer without it.

For a human taking a screenshot this is fine, usually desirable. For an agent it is a disaster:

• It cannot see where the pointer is, so it cannot verify that mouse_move worked.
• It sees hover effects — a highlighted row, an expanded tooltip — with no visible cause, and attributes them to something else.
• It has no feedback loop on its own most basic action.

So we paste a cursor image in at the current position. Note the 2× resize: at native size the cursor is roughly 32 pixels, which after the downscaling that happens before the image reaches the model is a handful of pixels — visible to you, invisible to it. Doubling it makes it survive the pipeline.

macOS needs none of this; screencapture -C includes the cursor natively. One line versus seven, which is a fair summary of cross-platform desktop work.

Transport and placement

The server runs on port 5055 inside the guest, over SSE via Starlette, using the low-level MCP server API rather than a higher-level wrapper — we wanted full control over the tool and resource listings.

Which means the topology is unusual: the MCP server is inside the sandbox and the client is outside it. Most MCP deployments are the reverse — server on your machine, model somewhere else. Here the server is the thing being controlled, and the sandbox boundary is a VM boundary, which is a genuinely strong one. The agent can move the mouse; it cannot escape.

Getting it reachable took a firewall rule, a pythonw.exe launcher and a scheduled task with the interactive flag set — all of which is its own miserable story.

The honest retrospective

We already had a plain REST control server on port 5050 doing the same six actions. We built the MCP server alongside it and ran both for months. So: what did MCP actually buy?

What it bought:

• Introspection. list_tools means a client discovers capabilities rather than being hard-coded against them. Adding a tool to the guest did not require updating the agent.
• A standard shape for the awkward parts. Structured schemas, mixed content types, resources — all things we would have invented worse versions of.
• Ecosystem access. Any MCP client could drive our desktop. Debugging interactively with an off-the-shelf client, against the same server the agent used, was worth the build on its own.

What it did not:

• Not fewer round trips. Same six calls, more envelope.
• Not better reliability. The failure modes are all in the guest — focus theft, timing, DPI — and the protocol is irrelevant to every one of them.
• Not less code. At the time, the low-level server API cost us more lines than the equivalent FastAPI routes.

If you have one agent and one desktop and control both ends, REST is fine and faster to build. MCP earns its keep the moment there is more than one client, or more than one server, or you want other tools in the ecosystem to reach your thing.

We kept both. The REST server stayed the fast path for the agent’s inner loop; MCP became the interface for everything else. Three and a half months later, the commit working MCP computer control, MCP browser control marked the point where the MCP path was good enough to drive a full session — by which time the protocol had matured considerably, and so had our understanding of what to put in it.

Next: everything so far ran on one laptop. Time to put a hypervisor inside a Kubernetes pod.