Computer MCP / Learn / Computer-use MCP servers
What is a computer-use MCP server?
How computer-use MCP servers actually work on macOS - the accessibility tree versus pixels versus a shell, five things that are harder than they look, and seven questions worth asking before you let one run on the machine you work on.
An MCP server is a small program that gives an AI assistant a set of tools. A computer-use MCP server gives it tools that operate the computer itself: read what is on screen, click, type, press buttons, choose menu items. The assistant stops being something you paste into and becomes something that can do the thing.
That is a short description of a large decision, so this page is the long version: how these servers actually work underneath, what separates them from each other, and the specific things worth checking before you let one run on the machine you work on. Most of the numbers below were measured while building one, and where something is unmeasured it says so.
What "computer use" means in practice
Three different mechanisms hide behind the same phrase, and they fail in different ways. Knowing which one a server uses tells you most of what you need to know about it.
| Mechanism | How it works | Where it breaks |
|---|---|---|
| Pixels | Screenshot, send the image to a model, let the model say "click at 840, 312". | The model is guessing from a picture. A screenshot is in pixels and a click is in points; get the scale wrong on a Retina display and you are 600 points off with no error message. It also costs an image per step. |
| The accessibility tree | macOS exposes every window as a tree of elements with roles, titles, values and frames - the same data a screen reader uses. The agent asks "where is the button called Export" and gets an answer. | Apps that draw their own controls (Electron, games, canvases) expose a thin or empty tree. And an app in true full-screen gets its own Space, which hides it from everything outside. |
| A shell | The server exposes "run this command" and the model does the rest. | This is remote access under a friendlier name. It is also the most capable option, which is exactly why it deserves a deliberate decision rather than an accident. |
Most real servers mix the first two. The third is a separate choice, and a server that has it should say so on the first screen rather than in a tool list you read later.
Five things that are harder than they look
These are not theoretical. Each one was measured while building a server, and each produced a wrong answer before it produced a right one.
1. A screenshot is one display, not "the screen"
The obvious implementation takes the first display the system hands back. On a machine with one screen that is correct. On a machine with three it means two thirds of the desktop is invisible, silently, with no error - and an agent that cannot find a window has no way to learn why.
Measured on a three-display Mac: the first display in the list was an
external monitor at (-1920, 27), while the built-in where the
person actually worked sat at (0, 0) and was never captured. Worse,
the order was not stable: index 0 pointed at two different monitors
within a single run, seconds apart. If a server lets you pick a display, check
whether it identifies them by position in a list or by a stable id.
2. Coordinates have an origin, and it is usually not zero
Click coordinates are global: one flat space across every display. A
screenshot of a secondary display has its own top-left corner, and that corner
may sit at x = -1920 on the desktop. An agent that divides by the
scale factor and clicks lands on a different monitor entirely. The answer a
server gives back should carry the origin, not just the scale.
3. Half of macOS has no button
File > Export, Edit > Find, Format > Font. A large share of what a person does on a Mac lives in the menu bar, where a vision model sees nothing and a click has nowhere to land. Measured: Chrome exposes 301 menu items. A server without menu access can show you those actions and not reach them.
4. The clipboard is a hole through every other protection
This is the sharpest one and it is easy to miss. Suppose a server blacks out password fields in screenshots - good. Now suppose it also offers "read the clipboard". A person copies their password out of a password manager, and one read hands it over in clear text, past every other guard. Any server that reads the clipboard should say what it does when the frontmost app is a password manager, and the honest answer may be "refuses".
5. Consent is not containment
A confirmation dialog decides whether an action happens. It does not limit what an approved action can reach. Once you say yes, the agent is driving your real machine with your real logged-in sessions, and nothing underneath constrains it. That is a reasonable trade for work on your own desktop; it is not a sandbox, and a server that implies otherwise is overselling. If you need a blast radius rather than a record, run the agent against a virtual machine.
What to check before you install one
Seven questions. They take about ten minutes and they are the same questions regardless of which server you end up with.
| # | Question | Why it matters |
|---|---|---|
| 1 | Does it have a read-only mode, and are the write tools hidden in it rather than merely refused? | A tool the model cannot see is a tool it cannot be talked into trying. Ask for the tool list in that mode and count. |
| 2 | What happens to password fields in a screenshot? | The useful answer names the mechanism: secure fields painted out on the bitmap before the file is written. If the unredacted image exists on disk at any point, it can leak from there. |
| 3 | What happens if nobody answers a consent dialog? | It must be a refusal. A prompt that resolves to yes because nobody saw it is not consent, and this is the one failure that is silent. |
| 4 | Is there a log, and what is in it? | "Everything is logged" is only good until the log is the thing worth stealing. A complete record of every keystroke in clear text is exactly the file an attacker wants. |
| 5 | Does it execute shell commands? | Not disqualifying - but it should be your decision, visible in your config, not a line item you find later. |
| 6 | Does it make network calls of its own? | Check rather than trust: clone it and grep for fetch,
http, axios. Screenshots of your desktop are the most
sensitive thing in the whole arrangement. |
| 7 | Can it work without taking over your screen? | Pressing an element through the accessibility API reaches a window sitting behind another one and never moves your pointer. Coordinate clicking cannot. The difference decides whether you can keep working while it runs. |
The field, as of September 2026
A dozen or so macOS computer-use MCP servers exist. Measured on GitHub on 18 September 2026: the largest has around 5,000 stars and is actively pushed; two of the next three had not been pushed in 5 and 15 months respectively; the rest sit in single digits. That shape matters more than the star counts - this is a young field where several of the obvious-looking options are no longer maintained, so last commit is worth more than stars when you are choosing.
They differ less in capability than you would expect. Screenshots, clicks, keystrokes and the accessibility tree are common to most. What differs is what happens by default, and whether the limits are written down.
Disclosure, since this page is on a vendor's site: we build one of them, Computer MCP. It is MIT, it runs on your machine, and its particular bet is that the guardrails are the product: redaction before the file is written, a consent gate on writes, a log that can only be added to, never edited that never stores typed text in clear. If that is not what you need, the honest advice is on when not to use it, which names the cases where something else is better - including our own browser tool for anything that lives in a web page.
Where this goes next
The interesting gap is not capability, it is unattended capability. Almost any of these servers can do a task while you watch. Far fewer are designed for the case where you start something and go to lunch, which is where the questions above stop being theoretical: what is in the log when you come back, what happened to the dialog nobody answered, and what was visible on the screen the whole time.
Our own notes on that case are in leaving an agent running overnight, including the part most pages leave out: the single most useful thing you can do before a long run has nothing to do with the software. Look at your own screen and close the terminal with the credentials in the scrollback. Redaction covers what macOS marks as secret. It cannot cover what you left sitting in a text window.
Against the published taxonomy
The seven questions above are ours. If you would rather work from a published list, OWASP has one: the OWASP MCP Top 10, answered with running code walks all ten categories and marks the four where the honest answer is "not our layer" or "this cannot be closed".
FAQ
What is a computer-use MCP server?
A small program that gives an AI assistant tools to operate a computer: read what is on screen, click, type, press buttons, choose menu items. It uses the standard that assistants use to find tools, so any of them - Claude Code, Cursor, Codex, VS Code - can use it. It runs on your machine, not in a data centre.
How is it different from browser automation?
A browser tool drives web pages and keeps the DOM, which is a far better handle than pixels. A computer-use server drives the operating system: native apps, menus, dialogs, the Finder. If the task lives in a web page, use a browser tool; driving a page through the accessibility tree means fighting a layer built for screen readers and losing the DOM.
Is it safe to let an AI agent control my Mac?
It depends on what the server does by default and what you have on screen. The useful questions are: does it have a read-only mode, what happens to password fields in a screenshot, what happens if nobody answers a consent dialog, and what is in the log. Note that consent is not containment: once you approve an action the agent drives your real machine with no isolation underneath. For a real blast radius, use a virtual machine.
Does a computer-use MCP server need an API key?
It should not. The server runs locally and talks to your assistant directly, the way one program talks to another on the same machine; the client talks to whichever model provider you already chose. If a server wants its own key or account, ask what it sends and to whom.
Why can an agent not find a window that is clearly open?
Two common causes, both measurable. An app in true full-screen mode gets its own Space, and from outside that Space neither the accessibility API nor a screenshot sees it. And on a machine with several displays, a screenshot captures one of them - often not the one you are looking at. Ask the server which display it captured and how many exist.
Can it work in the background while I keep using my Mac?
Partly, and it depends on the tools. Pressing an element through its own accessibility action reaches a window behind another one and never moves your pointer; so does writing into a field through the same API. Coordinate clicking and typing use the real cursor and keyboard, so those you will see. Anything with no accessibility element behind it - dragging on a canvas - needs the real pointer.
What runs on Windows or Linux?
Not the macOS servers. Their guarantees are built on macOS APIs: the Accessibility API for knowing that a field is a password field, and ScreenCaptureKit for the image. A port would be a different product with different guarantees, not the same one recompiled.
How do I install one?
Register it with your MCP client, then grant the operating system permissions it needs - on macOS, Accessibility and Screen Recording, which only a person can click. A server that claims to be installed before those are granted is not finished installing.