Vercel Labs agent-desktop: Desktop Automation via Accessibility Trees

Vercel Labs' agent-desktop is an open-source Rust tool that lets AI agents read and operate desktop applications through operating system accessibility trees instead of pixel matching. It provides a CLI and C library (libagent_desktop_ffi) for macOS, Windows, and Linux.

Image source: Chris Tate (@ctatedev)

Instead of matching pixels, agent-desktop reads an application's UI structure through operating system accessibility trees. Snapshots return structured JSON and references for elements an agent can act on.

Snapshot Trees and Qualified Element References (@snapshot_id:eN)

The documented workflow is to observe, act on a reference, and observe again. A snapshot returns structured JSON with references for elements an agent can act on.

Every actionable element receives a snapshot-qualified identifier formatted as @<snapshot_id>:eN. For example, the documented snapshot addresses an item as @s8f3k2p9:e8. Bare references are rejected with INVALID_ARGS; when the UI changes, an action can return STALE_REF or AMBIGUOUS_TARGET, requiring a new snapshot and reference.

Headless by Default and Background Agent Parallelism

agent-desktop uses headless execution by default. Reference actions use the accessibility API without moving the pointer or taking focus.

• Background Workflows: The announcement says the tool runs in the background while the user keeps working.
• Parallel Agents: The announcement says agents can use the computer in parallel, each with its own cursor.
• Optional Headed Mode: The global --headed flag enables physical input.

Cross-Platform Surface and Installation

The tool supports accessibility trees on macOS, Windows, and Linux, with Linux support provided through AT-SPI.

The npm package includes native binaries and platform helpers for x64 and ARM64. The tool ships 60 command names, including 56 operational commands, and each release includes libagent_desktop_ffi so programs can call it in process instead of starting the CLI for each command. The project uses the Apache-2.0 license.

Sources

• Vercel Labs GitHub Repository: vercel-labs/agent-desktop
• Vercel Labs Announcement: Chris Tate X (@ctatedev)