Driving macOS from an AI agent: what actually works

18 September 2026

Five findings from building a macOS computer-use server. Each one cost real time to learn, and each one looks like a bug in your code when it is not.

1. The accessibility tree beats the screenshot, every time

The intuitive design is: screenshot, let the model look, let it say "click at 812, 460". It works in a demo and degrades immediately, because a coordinate is only true until something moves. A window shifts, a notification pushes a toolbar down, the display scale differs - and now the agent is clicking something else, confidently, without any signal that it missed.

macOS exposes a structured tree of every control: role, title, description, value, and frame. Query it and you get "the Log in button is at this rectangle" instead of a guess. Two advantages that matter more than accuracy:

• Failure becomes visible. "No element named Log in" is information. A click that lands two pixels off is not.
• The log becomes readable. "Pressed Log in" can be audited by a human. "Clicked 812, 460" cannot.

Use the screenshot for what it is good at - letting the model understand the situation - and the tree for acting on it.

2. A full-screen app hides the entire rest of the machine

This one cost us half a day, and the symptom is the most misleading in the whole category: everything returns empty and nothing errors.

A full-screen window on macOS gets its own Space. Both the accessibility API and the screen-capture API see only the Space that is currently in front. So if your editor is full-screen - which, for a developer, it usually is - then every other app reports zero windows. Safari, Chrome, your test app, all of them. It looks exactly like a broken lookup, and we rewrote our window enumeration twice before checking with a second API and finding it had been right all along.

The second-order lesson is worth more than the first: a test that runs against the live screen can pass while measuring nothing. Ours did. A security test that goes green when it cannot see anything is worse than no test, so the ones that matter should build their own input instead of trusting the desktop.

3. In Electron apps, names are not unique - and it is not close

Electron apps - VS Code, Slack, Discord, Notion, Spotify - expose good, pressable accessibility elements. They also duplicate names heavily, because a composer, a message list and a sidebar each contribute their own copy of the same control.

Measured on one Electron app with a chat panel open:

73 buttons found, 73 named, 47 unique names

  "Show command menu (/)"   x10
  "Send message"            x8
  "Copy code to clipboard"  x6

Twenty-six of seventy-three buttons share a name with another button. So "press the Send message button" is ambiguous far more often than it looks, and taking the first match means picking one of eight at random. That is the worst kind of bug: it works in testing, and it does something almost right in production.

One more Electron detail: labels often carry a trailing hint in the accessible name. Show command menu (/) is one string. An exact-title match on Show command menu finds nothing at all.

4. Web password fields are marked differently from native ones

If you are redacting screenshots, or avoiding reading credential values, the marking differs by where the field lives:

Check the role only and you handle native apps and miss every browser password box - which is the case that actually occurs. Check both. It is one extra line and it is the difference between a guarantee and a nearly-guarantee.

5. An error message that lies costs more than no error message

Ours said "the application is not running." The application was running perfectly well - it just had no window on the visible Space, so the capture API did not list it. That message sent us looking for something that was standing right in front of us.

The fix is not clever: when the capture list is empty, ask the workspace whether the process exists, and say which of the two situations you are in.

not in capture list, and not running   -> "not running"
not in capture list, but running       -> "running, but no windows on the
                                          Space that is in front"

Worth noticing where this was found: not by rereading the code, but by following a review protocol that insisted on walking the install path in a fresh clone. The same pass found a test that measured the machine instead of the code, and a binary we had accidentally committed. None of the three came from staring harder at the source.

What we would tell someone starting today

1. Build on the accessibility tree first. Add pixel clicking as the fallback for canvases and maps, not as the primary path.
2. Assume the desktop is hostile to your tests. Anything load-bearing should construct its own input.
3. Refuse ambiguity rather than resolving it silently. A refusal the caller can act on beats a guess they cannot see.
4. Treat error messages as a user interface. A wrong one is more expensive than a missing one.
5. Decide early what you will never do. We have no shell tool, no arbitrary file reads and no URL fetching, and each of those absences has been worth more than the feature would have been.