Desktop control

Desktop control lets an agent drive native apps. Finder, Photos, System Settings, third-party apps with no APIs. It's powerful and the least mature layer of agent tooling. Use carefully.

How it works

The agent takes a screenshot, reasons about what's on screen, and emits actions: click at (x, y), type text, press keys, scroll. It's pixel-based vision, not DOM-aware.

When to reach for it

• Native desktop apps with no API (many older pro apps)
• Cross-app workflows (Finder → Photos → Mail)
• System Settings, Preferences
• Apps you paid for and trust (your own workstation)

When NOT to use it

• If an API or MCP exists. API is faster, more reliable.
• If the web version works. Browser automation beats desktop on reliability.
• For anything with irreversible consequences on shared systems. Desktop control is "fast and blunt."

Permission tiers

Anthropic's Computer Use (and similar systems) have tiered permissions:

• read, can screenshot only (observe, don't interact)
• click, can click, but not type or use keyboard shortcuts
• full, can click, type, use modifier keys, drag, etc.

Default: browsers are tier "read" (observe, but use Chrome extension for actual control). Terminals/IDEs are tier "click" (can click buttons, can't type into the editor). Everything else is tier "full."

This prevents common mistakes like: agent reads an email, sees a malicious link, and clicks it via desktop control (instead of opening it in Chrome where link safety rules apply).

The loop

1. Take screenshot
2. Reason: what is on screen? What's the next step toward the goal?
3. Emit action (click, type, key)
4. Wait briefly for UI to update
5. Take screenshot again
6. Repeat

Batch actions

Each screenshot + reason cycle is slow. If you know the next 5 actions, batch them. Computer Use supports a computer_batch tool that executes a sequence without screenshotting between each.

Example: open Spotlight, type "Maps," press Enter, wait. That's 4 actions as one batch.

Reliability patterns

Visual anchors over coordinates

Don't click (450, 230). Click "the search icon near the top-right." The model re-locates it each time, robust to window movement.

State verification

After each action, screenshot and verify. "Did the dialog close? Did the text appear? Did the state change?" If no, retry or replan.

Timeouts

Apps can hang. Set a max-turns budget for any desktop task. If the goal isn't reached, stop.

Safety

• Never drive through login screens. You type the password, not the agent.
• Never enter payment information. Ever.
• Avoid destructive dialogs. "Are you sure you want to delete?" is the agent's fault if it clicks Yes. Design prompts that say "if a destructive confirmation appears, stop and ask."
• Scope the agent's app access. Don't give it access to apps it doesn't need. Request access explicitly for each.

Cost

Screenshots are images. Images are expensive (both in tokens and in inference time). A 50-step desktop task can cost 10× a comparable DOM-based task.

Mitigations:

• Batch actions wherever possible
• Use compressed screenshots when high fidelity isn't needed
• Cache previous screenshots if you can reason "nothing changed since last shot"

The last-resort principle

Desktop control is the agent tool of last resort. Exhaust the alternatives:

1. Can an API or MCP do this? Use that.
2. Can a browser (via Chrome extension) do this? Use that.
3. Can a shell command do this? Use that.
4. Only then, desktop control.

The reason: every tier above is faster, cheaper, and more reliable. Desktop is the "can do anything" option, which is also the "can break anything" option.