Vercel Labs Open‑Sources agent‑desktop, a Rust CLI for AI agents to control desktops

It is an open‑source CLI licensed under Apache‑2.0 and written in Rust.

The tool lets any AI agent interact with desktop applications by reading the operating system’s accessibility tree instead of using pixel detection. It returns JSON snapshots with references that agents can click or manipulate safely.

agent‑desktop runs on macOS, Windows and Linux. On macOS it uses a private command host that retains native accessibility objects and exits after 15 minutes of idle time. Windows needs no extra grant, while Linux requires a graphical session with at‑spi2‑core.

Installation is possible via npm (global install) for x64 and ARM64 binaries, or by downloading native archives from GitHub Releases. Homebrew and Scoop packages are being prepared but are not yet in default catalogs.

Users must grant Accessibility permission on macOS and keep temporary directories private. An optional JEV helper can send screen descriptions to a model provider, but snapshots may contain private content and should not be logged publicly.