Setting up Vercel Agent Desktop with Claude Code on a Mac, and where background mode stops working
Agent Desktop (vercel-labs/agent-desktop, Apache 2.0) lets Claude Code use desktop apps by reading their accessibility tree instead of screenshots. Default mode runs in the background, so no mouse movement and no focus stealing.
Setup on a Mac:
npm install -g agent-desktop
agent-desktop permissions --guide
npx skills add vercel-labs/agent-desktop --skill agent-desktop
claude --allowedTools "Bash(agent-desktop:*)"
The Accessibility permission goes to whatever app launches Claude Code (Terminal, iTerm2, VS Code). Screen Recording is only needed for the screenshot command.
Gotchas from setting it up:
- that allowedTools rule doesn’t cover commands piped into grep or head, and some of those got denied
- snapshots are JSON and get big fast. A File menu task used about 550k input tokens (mostly cached). --app, -i --compact and --root keep them down
- anything on screen in that app ends up in Claude’s context, so name the app in your prompt and keep sensitive ones closed
Where background mode worked and where it didn’t (Claude Code 2.1.316, Opus 5.5, one run per task):
- reading an app and pressing buttons: worked. Calculator 128 × 46 came back as 5,888
- filling a text field: worked, it added a line in TextEdit
- saving: not reliable. cmd+s couldn’t be confirmed, and Save was greyed out in the File menu while the window was in the background
- selecting or styling text: needs --headed, which takes over your real mouse and keyboard
Claude also handled the failures well. When an action couldn’t be confirmed or a click was refused, it said so instead of claiming it worked.
Cleanup when you’re done: npm uninstall -g agent-desktop, delete the skill folder and ~/.agent-desktop, and turn off the Accessibility permission you gave your terminal.
aiparabellum.com
3 points · 1 comment
Empowering AI Agents with agent-desktop
A New Approach to the Problem
The agent-desktop repository addresses the challenge of enabling AI agents to effectively interact with desktop applications across different operating systems. By leveraging the OS accessibility tree, it allows agents to read and operate applications without pixel matching, which is often unreliable.
What is it and How it Works
agent-desktop is a CLI and C library developed for macOS, Windows, and Linux. It creates snapshots of the desktop environment, returning JSON representations of all elements an agent can interact with. The agent executes actions on these elements and verifies changes without requiring pixel recognition. Each CLI call operates in its own process, ensuring that headless calls do not interfere with the foreground application.
Key Features:
- Cross-Platform Compatibility: Works on macOS, Windows, and Linux.
- Snapshot Functionality: Returns a JSON object with references for every interactive element, allowing agents to perform actions and verify their results.
- No Pixel Matching Required: The system uses the accessibility API, which is more reliable than pixel-based methods.
- Privacy Considerations: Snapshots and outputs may contain sensitive data; users should manage this information carefully.
Key Capabilities & Developer Onboarding
Installation is straightforward via npm, Bun, Yarn, or pnpm:
npm install -g agent-desktop
agent-desktop version
For direct execution without a JavaScript launcher, users can download the native archive from GitHub Releases and keep its helper beside the executable.
Here’s a quick start example showing how to interact with the Finder app on macOS:
agent-desktop snapshot --app Finder -i # interactive elements, each with a ref
agent-desktop click @s8f3k2p9:e8 # act on a ref from that snapshot
agent-desktop snapshot --app Finder -i # observe again to verify
The JSON response includes details about the application state, making it easier for developers to understand the environment.
Who Should Use It & Integration Scenarios
agent-desktop is ideal for developers building AI agents that need to automate desktop tasks across multiple operating systems. It is particularly useful for applications requiring interaction with GUI elements without direct user input. However, it may not be suitable for scenarios demanding real-time visual feedback or where pixel precision is critical.
Limitations & Trade-offs:
- Platform-Specific Permissions: On macOS, accessibility permissions are required, which can add complexity to deployment.
- Headless Operation Constraints: Although the system is headless by default, it requires a graphical session on Linux, which can limit usability in certain environments.
- No Foreground Activation: Actions performed do not activate the foreground application, which may not align with all automation needs.
Community Pulse: What Developers Are Saying
Community discussions indicate a growing interest in the agent-desktop project, especially following the recent release of version v1.0.2 on October 30, 2026, which introduced various documentation improvements and clarified installation permissions. Users appreciate the simplicity of the API and the potential for integrating AI-driven workflows into desktop environments.
Our Read
Overall, agent-desktop presents a compelling solution for AI agent interactions with desktop applications, emphasizing cross-platform compatibility and privacy considerations. While it offers significant capabilities, potential users should be mindful of its limitations, particularly regarding permissions and headless operation constraints.
---
Originally published on GitTrending — Explore daily architectural deep-dives and quality scores for trending open-source GitHub projects.
AI HAS TAKEN OVER MY PC. IT’S TERRIFYING AS HELL...
VERCEL JUST SHIPPED COMPUTER USE FOR ANY AI AGENT. I SCANNED IT BEFORE I INSTALLED IT.
agent-desktop by Vercel Labs: ⭐ 8.8K, +2,243 today, #1 on GitHub Trending.
vercel-labs/agent-desktop
It reads any app's real UI through the OS accessibility tree, then clicks, types and scrolls by reference. No pixel matching. Version 1.0 landed on Oct 28.
What the scan found:
→ npm install runs a postinstall script. It uses the binary bundled in the package, or downloads its own GitHub release and checks the SHA-256 before installing it
→ 10 prebuilt binaries in the npm package: its own CLI and helpers for macOS, Windows and Linux
→ 6 agent skills + 2 dev skills, 0 auto-run hooks, 0 MCP servers
→ 2 other repos already reuse its description. vercel-labs/agent-desktop is the original
Verdict: CLEAR. The 12 flags were its own prebuilt binaries and that installer.
If you install it: on macOS it needs Accessibility permission, and its snapshots and screenshots can hold whatever is on your screen. Keep them out of public logs.
Repo and install line in the reply. Save this before you give an agent your desktop.
12 likes · 1 repost · 462 views
Vercel agent-desktop: computer use for coding agents through the accessibility tree
Vercel Labs released agent-desktop this week, a free, open source tool that gives AI coding agents computer use on macOS, Windows and Linux. Chris Tate, who built Vercel's agent-browser, announced it on October 31 and described it as "agent-browser, for your whole computer." The interesting part is not that an agent can click buttons. It is how it sees the screen: through the operating system's accessibility tree, with no screenshots and no pixel matching.
What actually shipped
agent-desktop is a Rust CLI plus a C library, on npm as version 1.0.2 under Apache-2.0, with native binaries for x64 and ARM64. There is no language model inside it. Any agent that can run shell commands can drive it, so Claude Code, Codex, Cursor's CLI or a plain bash script all qualify. The command set covers observation, clicks and typing, window management, notifications, the clipboard and sessions, about 56 commands in total. Development started in February, led by Lahfir. Version 1.0 with Linux support shipped on October 28, and 1.0.2 followed on October 30.
Refs, not pixels
The core loop is observe, act, observe again:
agent-desktop snapshot --app Finder -i # interactive elements, each with a ref
agent-desktop click @s8f3k2p9:e8 # act on a ref from that snapshot
agent-desktop snapshot --app Finder -i # observe again to verify
A snapshot returns JSON with a ref for every element the agent can act on. The agent acts on a ref, then takes a new snapshot to check the result. Nothing is inferred from pixels. The tool reads the AX API on macOS, UI Automation on Windows and AT-SPI on Linux, so it sees the same UI structure a screen reader sees, with roles and states the OS already maintains.
Errors an agent can actually handle
Refs belong to the snapshot that produced them. When the UI shifts before an action lands, you do not get a wrong click, you get a machine-readable error: STALE_REF or AMBIGUOUS_TARGET. If several windows match, the error lists the candidate window ids in error.details, and you pass one back with --window-id. A bare @e8 without its snapshot prefix is rejected with INVALID_ARGS.
This is the part I would underline for anyone wiring agents to GUIs. Vision-based computer use fails silently: the model clicks the wrong coordinate and you find out downstream, if you find out at all. Here every failure mode has a code and a defined recovery: take a new snapshot, use its refs. Actions are safe to retry because refs stay stable, which is exactly the property you want in an unattended loop.
Dense apps: skeleton first, then drill down
A full accessibility tree of an Electron app is big. The skeleton traversal work merged into the project measured it: a Slack skeleton snapshot is about 3.6KB versus roughly 17.3KB for the full tree, a 78-96% token saving on dense Electron apps. The pattern is to map first, then drill in:
agent-desktop snapshot --skeleton --app Slack -i --compact
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact
The skeleton gives a shallow overview with child counts on truncated containers. --root starts a traversal from a previously discovered ref, so you only pay tokens for the region you are working in. For an agent on a token budget, this is the difference between a tool you can leave running and one you babysit.
Wiring it to an existing agent
Install is npm install -g agent-desktop. The binary carries its own docs, matched to its version:
agent-desktop skills get desktop # the loop, refs, verification and retry rules
agent-desktop skills get platform # what differs on the OS you run on
You can install the bundled skill into a harness with npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop. Two details worth knowing before you script it. Actions run headless by default: they go through the accessibility API without moving the pointer or stealing focus, and --headed opts into real input. And on PowerShell, quote every ref, because a bare @token is the splatting operator.
Platform reality check
The platform table in the repo readme is honest about gaps. macOS needs an Accessibility grant, and agent-desktop permissions --guide walks you through it. Windows needs no grant for apps at your terminal's integrity level. Linux needs at-spi2-core in your graphical session; mouse input is X11 only or --headed, the Wayland clipboard needs wl-clipboard, and notifications go through GNOME Shell. Apps without an accessibility tree are invisible to it, and it does not automate web pages, that is what agent-browser is for.
Take the data exposure seriously
Snapshots, screenshots and clipboard reads can contain private data, and the project's own docs warn against routing them into public logs. An optional helper called JEV sends screen descriptions to a configured model provider. Sit with that for a second: this tool reads your screen the way a screen reader does, and it hands what it reads to whatever agent invoked it. An agent with agent-desktop plus shell access effectively holds your user account.
I wrote about where that boundary sits when Nvidia moved agent execution into the kernel, and about the guardrail details to check before letting Copilot click your desktop. The same logic applies here: run it in a dedicated environment where the blast radius is a test account, not your daily driver.
When I would reach for it
Not for anything with an API. If the thing you are driving exposes an MCP server, a CLI or an HTTP endpoint, use that instead, it will be cheaper and more reliable. agent-desktop earns its keep on apps with no automation surface at all: installers, vendor tools, that one dialog box your CI cannot get past. The observe-act-observe loop with structured errors is the same verification discipline I want in every agent workflow, and now it extends to GUIs. That is the real news here, not the clicking.
any vibe coded desktop app can now add computer use as a feature.
repo: vercel-labs/agent-desktop
x.com
33 likes · 2 reposts · 4,541 views
GitHub - vercel-labs/agent-desktop: Agent Desktop gives any agent reliable computer use on the deskt...
要約:
Agent Desktopは、AIエージェントがデスクトップアプリケーションを操作するためのツールです。macOS、Windows、Linuxで利用可能で、OSのアクセシビリティツリーを通じてアプリケーションのUI構造を読み取り、操作します。CLIとCライブラリを提供し、エージェントが安全に再試行可能な操作を行うための安定した参照を使用します。インストール方法やコマンドの使...
1 like
Vercel's agent-desktop Lets Coding Agents Click Native Apps Without Moving Your Mouse
- Book: AI Agents Pocket Guide
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
---
Your coding agent can now open Finder, read every button in the window as JSON, and click one by its ID, while your own pointer stays exactly where you left it.
That is the pitch of agent-desktop, which Vercel Labs released as version 1.0.2 on October 30. The README calls it "Computer use for AI agents: read and operate desktop apps through the OS accessibility tree." It is a Rust CLI plus a C library, Apache-2.0, and it runs on macOS, Windows and Linux. It contains no model. It is pitched as computer use for any coding agent, and the agent drives it by running shell commands.
What shipped, and when
The builder's launch post went up on X at 23:48 UTC on October 30, according to CellCog. Vercel's Chris Tate then described it as "agent-browser, for your whole computer," a nod to Vercel's agent-browser, which does the same thing for web pages. Vercel CEO Guillermo Rauch shared it with "An agent on every desktop."
The project is older than the launch: it started in February and later moved under the Vercel Labs organization, per ShiftDelete. Windows 10 1809+ support arrived in v0.10.0 on October 18, and v1.0.2 on October 30 added a docs site and made AGENTS.md the single instructions file for agents. MarkTechPost and AlexTech covered it the next day. The repo had about 6,900 stars on November 2, per its GitHub page.
The docs list the floors: macOS 13 or later (Intel or Apple Silicon) through the AX API, Windows 10 1809 or later through UI Automation, and Ubuntu 24.04 or later through AT-SPI, on x64 and ARM64.
How the loop works
The tool does one step per call. Each command prints one JSON envelope and exits. The observe, act, verify loop lives in your agent. Adapted from the README:
# setup
npm install -g agent-desktop
agent-desktop status
agent-desktop skills get desktop
# look: interactive elements, each with a ref
agent-desktop snapshot --app Finder -i
# act on a ref from that snapshot
agent-desktop click @s8f3k2p9:e8
# look again to check it worked
agent-desktop snapshot --app Finder -i
The ref values above are the README's examples. Yours come from your own snapshot. Refs are tied to the snapshot that produced them: v0.10.0 removed bare refs like @e8, and the README says they now fail with INVALID_ARGS. If the UI changed, the call returns STALE_REF and the agent should snapshot again. If several windows match, AMBIGUOUS_TARGET lists candidate window IDs to pass with --window-id. Every error carries a disposition, and the README says to retry only when error.disposition.retry is safe. Put that rule in your agent's AGENTS.md or system prompt.
Calls are headless by default. Per the README, ref actions go through the accessibility API "without moving the pointer or taking focus." A global --headed flag allows physical mouse and keyboard input when an app needs it. With several agents at once, each subagent gets its own colored overlay cursor, ShiftDelete reports, and actions that need the one physical pointer take turns.
!An editorial illustration of a branching tree of interface nodes rising out of a desktop window, a small robot hand tapping one glowing orange node while a real mouse pointer rests untouched to the side
The agent reads the app's accessibility tree and acts on one node by ref. Your pointer isn't part of it.
The token numbers are the builder's own
The project's headline argument against screenshot-based computer use is cost. Every figure for that so far comes from the project itself.
The builder's posts report 273 tokens per look at Apple Maps when you snapshot one region, against 3,811 for the whole window, and a simulated 52-step task dropping from 198k tokens to 31k, as summarised by CellCog. AlexTech puts the skeleton mode's saving at 78% to 96% against full-tree dumps in Slack, VS Code and Notion, and calls these "self-reported figures not verified by third parties." CellCog also points out that there is no shared yardstick for these numbers yet.
The technique behind the saving is in the README. You map a dense app shallowly first, then drill into the part you care about:
agent-desktop snapshot --skeleton --app Slack -i --compact
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact
I haven't run agent-desktop myself, so I can only relay these numbers. If you try it, count tokens on your own app. Slack and Notion are not your internal Electron tool.
What it can't see
The docs are blunt about the limits. It reads accessibility trees and does not match pixels, so a UI that publishes no accessibility tree is invisible to it. How well it works depends on how well each app's interface is built.
It also does not automate web pages. For those, use agent-browser. For web content inside a Chromium or Electron app, agent-desktop opens a DevTools port that agent-browser can connect to.
The CLI has 60 command names, the same on every platform. Four of them (key-down, key-up, mouse-down, mouse-up) fail closed everywhere. And a successful action does not prove the UI changed, which is why the docs recommend a follow-up read such as wait --element ... --predicate value.
!An editorial illustration of a magnifying glass over a desktop window split in two: the left half shows a neat tree of nodes, the right half is a blank black panel with no structure for the lens to find
No accessibility tree, no view. Pixel-only UIs stay out of reach.
Before you hand it to an agent
A few things in the docs are worth settling before the first run.
Who gets the permission. On macOS the Accessibility permission goes to the app that runs agent-desktop. If your agent runs in a terminal, that terminal is the app asking, so decide on purpose which one gets it. agent-desktop permissions --guide walks you through it, and the README also requires TMPDIR to be owned by you with mode 0700. On Windows no grant is needed for apps at your terminal's integrity level.
What ends up in transcripts. The README warns that "Snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content." The FAQ goes further: "redaction does not make a screenshot safe to publish." Your agent's transcript now holds whatever was on screen, so treat it like a log with secrets in it.
What leaves the machine. The FAQ says the native CLI "does not require a remote model service." The optional JEV helper is different: it sends goals and screen descriptions to the model provider you configure.
What git won't show you. A click in Finder leaves no diff. Keep desktop actions on a short leash: name the app, name the task, and use --headed only when headless fails. I build Hermes IDE, so weigh this with that in mind. Hermes runs each agent in a real terminal, so an agent there can call agent-desktop like any other CLI. But Hermes, by its README, "observes what agents report on purpose and what git shows." Its Review Desk can show you every file an agent changed, turn by turn. It cannot show you what the agent clicked in another app.
Would you give it your desktop?
A cheap first test: point your agent at one app, let it run snapshot only, and read what comes back before it gets to click anything.
Which native app would you trust an agent to drive first, and which one would you never hand over? Tell me in the comments.
---
If this was useful
agent-desktop is a tool, and the interesting design work sits in the loop around it: when to look again, when a retry is safe, when to stop and ask. That loop is what the AI Agents Pocket Guide is about, from tool calling to guardrails for agents that act on their own.
If you run more than one coding agent, Hermes IDE is free, source-available and on macOS, Windows and Linux, and the code is on GitHub.
1 reaction
Vercel Releases Free, Open-Source Tool That Lets AI Agents Use Desktop Apps Without Taking Over Your Mouse
Vercel Labs has released agent-desktop, a free, open-source (Apache-2.0) CLI and C library that lets AI agents read and operate native apps on macOS, Windows and Linux. Version 1.0.2 shipped on October 30, 2026.
Vercel Labs' Chris Tate announced it on X as "agent-browser, for your whole computer," a nod to Vercel's earlier browser-automation CLI for agents.
How it's different from screenshot-based computer use
Most computer-use setups send the model a screenshot and get back pixel coordinates to click. agent-desktop doesn't look at pixels. It reads the OS accessibility tree, the same structured UI data screen readers rely on:
- macOS → AX API
- Windows → UI Automation
- Linux → AT-SPI
A snapshot returns JSON with a ref for every element the agent can act on, such as @s8f3k2p9:e8. The agent clicks or types on that ref, then snapshots again to check the result.
The "doesn't take over your mouse" part
Actions are headless by default. According to the project's README, they go through the accessibility API "without moving the pointer or taking focus," so you can keep using your machine while the agent works. A global --headed flag turns on real mouse and keyboard input when an app needs it.
It's a tool, not an agent
There's no LLM inside. Each command prints one JSON envelope and exits, and the observe → act → verify loop stays in whatever agent you already use. Anything that can run shell commands can drive it.
Token cost
Anthropic's computer-use docs put each screenshot at roughly 1,000–1,800 input tokens, and its computer toolset adds about 4,500 input tokens per request.
MarkTechPost reports that agent-desktop's builder measured 78–96% fewer tokens than full-tree dumps on Slack, VS Code and Notion, using a --skeleton mode that reads only 3 levels deep. These are self-reported numbers, not an independent benchmark.
Limits
- It can't see apps that don't expose an accessibility tree.
- It doesn't automate web pages. Vercel points you to its agent-browser tool for those.
- Snapshots, clipboard reads and screenshots can contain private data, and the README says to keep them out of public logs.
Install
npm install -g agent-desktop
On macOS, grant Accessibility permission with agent-desktop permissions --guide.
Sources
- GitHub: vercel-labs/agent-desktop
- Chris Tate's announcement on X
- MarkTechPost: Vercel Labs Open-Sources agent-desktop for Agent Computer Use
3 points
vercel-labs / agent-desktop: Agent Desktop gives any agent reliable computer use on the desktop. Built with Rust, it sees any app's real UI structure through OS accessibility trees and operates it — refs stay stable and actions stay safe to re ... ★5513 vercel-labs/agent-desktop
2 likes
Vercel LabsのAgent Desktopが正式版になりました。スクリーンショットから座標を推測するのではなく、OSのアクセシビリティツリーからボタンや入力欄を特定して操作する仕組みです。API / MCPがないアプリの自動化が捗りますね！
vercel-labs/agent-desktop
21 likes · 1 repost · 537 views
Mentions
Setting up Vercel Agent Desktop with Claude Code on a Mac, and where background mode stops working Agent Desktop (vercel-labs/agent-desktop, Apache 2.0) lets Claude Code use desktop apps by reading their accessibility tree instead of screenshots. Default mode runs in the background, so no mouse movement and no focus stealing. Setup on a Mac: npm install -g agent-desktop agent-desktop permissions --guide npx skills add vercel-labs/agent-desktop --skill agent-desktop claude --allowedTools "Bash(agent-desktop:*)" The Accessibility permission goes to whatever app launches Claude Code (Terminal, iTerm2, VS Code). Screen Recording is only needed for the screenshot command. Gotchas from setting it up: - that allowedTools rule doesn’t cover commands piped into grep or head, and some of those got denied - snapshots are JSON and get big fast. A File menu task used about 550k input tokens (mostly cached). --app, -i --compact and --root keep them down - anything on screen in that app ends up in Claude’s context, so name the app in your prompt and keep sensitive ones closed Where background mode worked and where it didn’t (Claude Code 2.1.316, Opus 5.5, one run per task): - reading an app and pressing buttons: worked. Calculator 128 × 46 came back as 5,888 - filling a text field: worked, it added a line in TextEdit - saving: not reliable. cmd+s couldn’t be confirmed, and Save was greyed out in the File menu while the window was in the background - selecting or styling text: needs --headed, which takes over your real mouse and keyboard Claude also handled the failures well. When an action couldn’t be confirmed or a click was refused, it said so instead of claiming it worked. Cleanup when you’re done: npm uninstall -g agent-desktop, delete the skill folder and ~/.agent-desktop, and turn off the Accessibility permission you gave your terminal. aiparabellum.com
Empowering AI Agents with agent-desktop A New Approach to the Problem The agent-desktop repository addresses the challenge of enabling AI agents to effectively interact with desktop applications across different operating systems. By leveraging the OS accessibility tree, it allows agents to read and operate applications without pixel matching, which is often unreliable. What is it and How it Works agent-desktop is a CLI and C library developed for macOS, Windows, and Linux. It creates snapshots of the desktop environment, returning JSON representations of all elements an agent can interact with. The agent executes actions on these elements and verifies changes without requiring pixel recognition. Each CLI call operates in its own process, ensuring that headless calls do not interfere with the foreground application. Key Features: - Cross-Platform Compatibility: Works on macOS, Windows, and Linux. - Snapshot Functionality: Returns a JSON object with references for every interactive element, allowing agents to perform actions and verify their results. - No Pixel Matching Required: The system uses the accessibility API, which is more reliable than pixel-based methods. - Privacy Considerations: Snapshots and outputs may contain sensitive data; users should manage this information carefully. Key Capabilities & Developer Onboarding Installation is straightforward via npm, Bun, Yarn, or pnpm: npm install -g agent-desktop agent-desktop version For direct execution without a JavaScript launcher, users can download the native archive from GitHub Releases and keep its helper beside the executable. Here’s a quick start example showing how to interact with the Finder app on macOS: agent-desktop snapshot --app Finder -i # interactive elements, each with a ref agent-desktop click @s8f3k2p9:e8 # act on a ref from that snapshot agent-desktop snapshot --app Finder -i # observe again to verify The JSON response includes details about the application state, making it easier for developers to understand the environment. Who Should Use It & Integration Scenarios agent-desktop is ideal for developers building AI agents that need to automate desktop tasks across multiple operating systems. It is particularly useful for applications requiring interaction with GUI elements without direct user input. However, it may not be suitable for scenarios demanding real-time visual feedback or where pixel precision is critical. Limitations & Trade-offs: - Platform-Specific Permissions: On macOS, accessibility permissions are required, which can add complexity to deployment. - Headless Operation Constraints: Although the system is headless by default, it requires a graphical session on Linux, which can limit usability in certain environments. - No Foreground Activation: Actions performed do not activate the foreground application, which may not align with all automation needs. Community Pulse: What Developers Are Saying Community discussions indicate a growing interest in the agent-desktop project, especially following the recent release of version v1.0.2 on October 30, 2026, which introduced various documentation improvements and clarified installation permissions. Users appreciate the simplicity of the API and the potential for integrating AI-driven workflows into desktop environments. Our Read Overall, agent-desktop presents a compelling solution for AI agent interactions with desktop applications, emphasizing cross-platform compatibility and privacy considerations. While it offers significant capabilities, potential users should be mindful of its limitations, particularly regarding permissions and headless operation constraints. --- Originally published on GitTrending — Explore daily architectural deep-dives and quality scores for trending open-source GitHub projects.
AI HAS TAKEN OVER MY PC. IT’S TERRIFYING AS HELL... VERCEL JUST SHIPPED COMPUTER USE FOR ANY AI AGENT. I SCANNED IT BEFORE I INSTALLED IT. agent-desktop by Vercel Labs: ⭐ 8.8K, +2,243 today, #1 on GitHub Trending. vercel-labs/agent-desktop It reads any app's real UI through the OS accessibility tree, then clicks, types and scrolls by reference. No pixel matching. Version 1.0 landed on Oct 28. What the scan found: → npm install runs a postinstall script. It uses the binary bundled in the package, or downloads its own GitHub release and checks the SHA-256 before installing it → 10 prebuilt binaries in the npm package: its own CLI and helpers for macOS, Windows and Linux → 6 agent skills + 2 dev skills, 0 auto-run hooks, 0 MCP servers → 2 other repos already reuse its description. vercel-labs/agent-desktop is the original Verdict: CLEAR. The 12 flags were its own prebuilt binaries and that installer. If you install it: on macOS it needs Accessibility permission, and its snapshots and screenshots can hold whatever is on your screen. Keep them out of public logs. Repo and install line in the reply. Save this before you give an agent your desktop.
Vercel agent-desktop: computer use for coding agents through the accessibility tree Vercel Labs released agent-desktop this week, a free, open source tool that gives AI coding agents computer use on macOS, Windows and Linux. Chris Tate, who built Vercel's agent-browser, announced it on October 31 and described it as "agent-browser, for your whole computer." The interesting part is not that an agent can click buttons. It is how it sees the screen: through the operating system's accessibility tree, with no screenshots and no pixel matching. What actually shipped agent-desktop is a Rust CLI plus a C library, on npm as version 1.0.2 under Apache-2.0, with native binaries for x64 and ARM64. There is no language model inside it. Any agent that can run shell commands can drive it, so Claude Code, Codex, Cursor's CLI or a plain bash script all qualify. The command set covers observation, clicks and typing, window management, notifications, the clipboard and sessions, about 56 commands in total. Development started in February, led by Lahfir. Version 1.0 with Linux support shipped on October 28, and 1.0.2 followed on October 30. Refs, not pixels The core loop is observe, act, observe again: agent-desktop snapshot --app Finder -i # interactive elements, each with a ref agent-desktop click @s8f3k2p9:e8 # act on a ref from that snapshot agent-desktop snapshot --app Finder -i # observe again to verify A snapshot returns JSON with a ref for every element the agent can act on. The agent acts on a ref, then takes a new snapshot to check the result. Nothing is inferred from pixels. The tool reads the AX API on macOS, UI Automation on Windows and AT-SPI on Linux, so it sees the same UI structure a screen reader sees, with roles and states the OS already maintains. Errors an agent can actually handle Refs belong to the snapshot that produced them. When the UI shifts before an action lands, you do not get a wrong click, you get a machine-readable error: STALE_REF or AMBIGUOUS_TARGET. If several windows match, the error lists the candidate window ids in error.details, and you pass one back with --window-id. A bare @e8 without its snapshot prefix is rejected with INVALID_ARGS. This is the part I would underline for anyone wiring agents to GUIs. Vision-based computer use fails silently: the model clicks the wrong coordinate and you find out downstream, if you find out at all. Here every failure mode has a code and a defined recovery: take a new snapshot, use its refs. Actions are safe to retry because refs stay stable, which is exactly the property you want in an unattended loop. Dense apps: skeleton first, then drill down A full accessibility tree of an Electron app is big. The skeleton traversal work merged into the project measured it: a Slack skeleton snapshot is about 3.6KB versus roughly 17.3KB for the full tree, a 78-96% token saving on dense Electron apps. The pattern is to map first, then drill in: agent-desktop snapshot --skeleton --app Slack -i --compact agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact The skeleton gives a shallow overview with child counts on truncated containers. --root starts a traversal from a previously discovered ref, so you only pay tokens for the region you are working in. For an agent on a token budget, this is the difference between a tool you can leave running and one you babysit. Wiring it to an existing agent Install is npm install -g agent-desktop. The binary carries its own docs, matched to its version: agent-desktop skills get desktop # the loop, refs, verification and retry rules agent-desktop skills get platform # what differs on the OS you run on You can install the bundled skill into a harness with npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop. Two details worth knowing before you script it. Actions run headless by default: they go through the accessibility API without moving the pointer or stealing focus, and --headed opts into real input. And on PowerShell, quote every ref, because a bare @token is the splatting operator. Platform reality check The platform table in the repo readme is honest about gaps. macOS needs an Accessibility grant, and agent-desktop permissions --guide walks you through it. Windows needs no grant for apps at your terminal's integrity level. Linux needs at-spi2-core in your graphical session; mouse input is X11 only or --headed, the Wayland clipboard needs wl-clipboard, and notifications go through GNOME Shell. Apps without an accessibility tree are invisible to it, and it does not automate web pages, that is what agent-browser is for. Take the data exposure seriously Snapshots, screenshots and clipboard reads can contain private data, and the project's own docs warn against routing them into public logs. An optional helper called JEV sends screen descriptions to a configured model provider. Sit with that for a second: this tool reads your screen the way a screen reader does, and it hands what it reads to whatever agent invoked it. An agent with agent-desktop plus shell access effectively holds your user account. I wrote about where that boundary sits when Nvidia moved agent execution into the kernel, and about the guardrail details to check before letting Copilot click your desktop. The same logic applies here: run it in a dedicated environment where the blast radius is a test account, not your daily driver. When I would reach for it Not for anything with an API. If the thing you are driving exposes an MCP server, a CLI or an HTTP endpoint, use that instead, it will be cheaper and more reliable. agent-desktop earns its keep on apps with no automation surface at all: installers, vendor tools, that one dialog box your CI cannot get past. The observe-act-observe loop with structured errors is the same verification discipline I want in every agent workflow, and now it extends to GUIs. That is the real news here, not the clicking.
any vibe coded desktop app can now add computer use as a feature. repo: vercel-labs/agent-desktop x.com
GitHub - vercel-labs/agent-desktop: Agent Desktop gives any agent reliable computer use on the deskt... 要約: Agent Desktopは、AIエージェントがデスクトップアプリケーションを操作するためのツールです。macOS、Windows、Linuxで利用可能で、OSのアクセシビリティツリーを通じてアプリケーションのUI構造を読み取り、操作します。CLIとCライブラリを提供し、エージェントが安全に再試行可能な操作を行うための安定した参照を使用します。インストール方法やコマンドの使...
Vercel's agent-desktop Lets Coding Agents Click Native Apps Without Moving Your Mouse - Book: AI Agents Pocket Guide - Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- Your coding agent can now open Finder, read every button in the window as JSON, and click one by its ID, while your own pointer stays exactly where you left it. That is the pitch of agent-desktop, which Vercel Labs released as version 1.0.2 on October 30. The README calls it "Computer use for AI agents: read and operate desktop apps through the OS accessibility tree." It is a Rust CLI plus a C library, Apache-2.0, and it runs on macOS, Windows and Linux. It contains no model. It is pitched as computer use for any coding agent, and the agent drives it by running shell commands. What shipped, and when The builder's launch post went up on X at 23:48 UTC on October 30, according to CellCog. Vercel's Chris Tate then described it as "agent-browser, for your whole computer," a nod to Vercel's agent-browser, which does the same thing for web pages. Vercel CEO Guillermo Rauch shared it with "An agent on every desktop." The project is older than the launch: it started in February and later moved under the Vercel Labs organization, per ShiftDelete. Windows 10 1809+ support arrived in v0.10.0 on October 18, and v1.0.2 on October 30 added a docs site and made AGENTS.md the single instructions file for agents. MarkTechPost and AlexTech covered it the next day. The repo had about 6,900 stars on November 2, per its GitHub page. The docs list the floors: macOS 13 or later (Intel or Apple Silicon) through the AX API, Windows 10 1809 or later through UI Automation, and Ubuntu 24.04 or later through AT-SPI, on x64 and ARM64. How the loop works The tool does one step per call. Each command prints one JSON envelope and exits. The observe, act, verify loop lives in your agent. Adapted from the README: # setup npm install -g agent-desktop agent-desktop status agent-desktop skills get desktop # look: interactive elements, each with a ref agent-desktop snapshot --app Finder -i # act on a ref from that snapshot agent-desktop click @s8f3k2p9:e8 # look again to check it worked agent-desktop snapshot --app Finder -i The ref values above are the README's examples. Yours come from your own snapshot. Refs are tied to the snapshot that produced them: v0.10.0 removed bare refs like @e8, and the README says they now fail with INVALID_ARGS. If the UI changed, the call returns STALE_REF and the agent should snapshot again. If several windows match, AMBIGUOUS_TARGET lists candidate window IDs to pass with --window-id. Every error carries a disposition, and the README says to retry only when error.disposition.retry is safe. Put that rule in your agent's AGENTS.md or system prompt. Calls are headless by default. Per the README, ref actions go through the accessibility API "without moving the pointer or taking focus." A global --headed flag allows physical mouse and keyboard input when an app needs it. With several agents at once, each subagent gets its own colored overlay cursor, ShiftDelete reports, and actions that need the one physical pointer take turns. !An editorial illustration of a branching tree of interface nodes rising out of a desktop window, a small robot hand tapping one glowing orange node while a real mouse pointer rests untouched to the side The agent reads the app's accessibility tree and acts on one node by ref. Your pointer isn't part of it. The token numbers are the builder's own The project's headline argument against screenshot-based computer use is cost. Every figure for that so far comes from the project itself. The builder's posts report 273 tokens per look at Apple Maps when you snapshot one region, against 3,811 for the whole window, and a simulated 52-step task dropping from 198k tokens to 31k, as summarised by CellCog. AlexTech puts the skeleton mode's saving at 78% to 96% against full-tree dumps in Slack, VS Code and Notion, and calls these "self-reported figures not verified by third parties." CellCog also points out that there is no shared yardstick for these numbers yet. The technique behind the saving is in the README. You map a dense app shallowly first, then drill into the part you care about: agent-desktop snapshot --skeleton --app Slack -i --compact agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact I haven't run agent-desktop myself, so I can only relay these numbers. If you try it, count tokens on your own app. Slack and Notion are not your internal Electron tool. What it can't see The docs are blunt about the limits. It reads accessibility trees and does not match pixels, so a UI that publishes no accessibility tree is invisible to it. How well it works depends on how well each app's interface is built. It also does not automate web pages. For those, use agent-browser. For web content inside a Chromium or Electron app, agent-desktop opens a DevTools port that agent-browser can connect to. The CLI has 60 command names, the same on every platform. Four of them (key-down, key-up, mouse-down, mouse-up) fail closed everywhere. And a successful action does not prove the UI changed, which is why the docs recommend a follow-up read such as wait --element ... --predicate value. !An editorial illustration of a magnifying glass over a desktop window split in two: the left half shows a neat tree of nodes, the right half is a blank black panel with no structure for the lens to find No accessibility tree, no view. Pixel-only UIs stay out of reach. Before you hand it to an agent A few things in the docs are worth settling before the first run. Who gets the permission. On macOS the Accessibility permission goes to the app that runs agent-desktop. If your agent runs in a terminal, that terminal is the app asking, so decide on purpose which one gets it. agent-desktop permissions --guide walks you through it, and the README also requires TMPDIR to be owned by you with mode 0700. On Windows no grant is needed for apps at your terminal's integrity level. What ends up in transcripts. The README warns that "Snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content." The FAQ goes further: "redaction does not make a screenshot safe to publish." Your agent's transcript now holds whatever was on screen, so treat it like a log with secrets in it. What leaves the machine. The FAQ says the native CLI "does not require a remote model service." The optional JEV helper is different: it sends goals and screen descriptions to the model provider you configure. What git won't show you. A click in Finder leaves no diff. Keep desktop actions on a short leash: name the app, name the task, and use --headed only when headless fails. I build Hermes IDE, so weigh this with that in mind. Hermes runs each agent in a real terminal, so an agent there can call agent-desktop like any other CLI. But Hermes, by its README, "observes what agents report on purpose and what git shows." Its Review Desk can show you every file an agent changed, turn by turn. It cannot show you what the agent clicked in another app. Would you give it your desktop? A cheap first test: point your agent at one app, let it run snapshot only, and read what comes back before it gets to click anything. Which native app would you trust an agent to drive first, and which one would you never hand over? Tell me in the comments. --- If this was useful agent-desktop is a tool, and the interesting design work sits in the loop around it: when to look again, when a retry is safe, when to stop and ask. That loop is what the AI Agents Pocket Guide is about, from tool calling to guardrails for agents that act on their own. If you run more than one coding agent, Hermes IDE is free, source-available and on macOS, Windows and Linux, and the code is on GitHub. 
Vercel Releases Free, Open-Source Tool That Lets AI Agents Use Desktop Apps Without Taking Over Your Mouse Vercel Labs has released agent-desktop, a free, open-source (Apache-2.0) CLI and C library that lets AI agents read and operate native apps on macOS, Windows and Linux. Version 1.0.2 shipped on October 30, 2026. Vercel Labs' Chris Tate announced it on X as "agent-browser, for your whole computer," a nod to Vercel's earlier browser-automation CLI for agents. How it's different from screenshot-based computer use Most computer-use setups send the model a screenshot and get back pixel coordinates to click. agent-desktop doesn't look at pixels. It reads the OS accessibility tree, the same structured UI data screen readers rely on: - macOS → AX API - Windows → UI Automation - Linux → AT-SPI A snapshot returns JSON with a ref for every element the agent can act on, such as @s8f3k2p9:e8. The agent clicks or types on that ref, then snapshots again to check the result. The "doesn't take over your mouse" part Actions are headless by default. According to the project's README, they go through the accessibility API "without moving the pointer or taking focus," so you can keep using your machine while the agent works. A global --headed flag turns on real mouse and keyboard input when an app needs it. It's a tool, not an agent There's no LLM inside. Each command prints one JSON envelope and exits, and the observe → act → verify loop stays in whatever agent you already use. Anything that can run shell commands can drive it. Token cost Anthropic's computer-use docs put each screenshot at roughly 1,000–1,800 input tokens, and its computer toolset adds about 4,500 input tokens per request. MarkTechPost reports that agent-desktop's builder measured 78–96% fewer tokens than full-tree dumps on Slack, VS Code and Notion, using a --skeleton mode that reads only 3 levels deep. These are self-reported numbers, not an independent benchmark. Limits - It can't see apps that don't expose an accessibility tree. - It doesn't automate web pages. Vercel points you to its agent-browser tool for those. - Snapshots, clipboard reads and screenshots can contain private data, and the README says to keep them out of public logs. Install npm install -g agent-desktop On macOS, grant Accessibility permission with agent-desktop permissions --guide. Sources - GitHub: vercel-labs/agent-desktop - Chris Tate's announcement on X - MarkTechPost: Vercel Labs Open-Sources agent-desktop for Agent Computer Use
vercel-labs / agent-desktop: Agent Desktop gives any agent reliable computer use on the desktop. Built with Rust, it sees any app's real UI structure through OS accessibility trees and operates it — refs stay stable and actions stay safe to re ... ★5513 vercel-labs/agent-desktop
Vercel LabsのAgent Desktopが正式版になりました。スクリーンショットから座標を推測するのではなく、OSのアクセシビリティツリーからボタンや入力欄を特定して操作する仕組みです。API / MCPがないアプリの自動化が捗りますね！ vercel-labs/agent-desktop