FreeLLMVercel Labs Open‑Sources agent‑desktop, a Rust CLI for AI agents to control desktops
The new Apache‑2.0 tool lets AI agents read OS accessibility trees and act on stable references on macOS, Windows and Linux.
It is an open‑source CLI licensed under Apache‑2.0 and written in Rust.
The tool lets any AI agent interact with desktop applications by reading the operating system’s accessibility tree instead of using pixel detection. It returns JSON snapshots with references that agents can click or manipulate safely.
agent‑desktop runs on macOS, Windows and Linux. On macOS it uses a private command host that retains native accessibility objects and exits after 15 minutes of idle time. Windows needs no extra grant, while Linux requires a graphical session with at‑spi2‑core.
Installation is possible via npm (global install) for x64 and ARM64 binaries, or by downloading native archives from GitHub Releases. Homebrew and Scoop packages are being prepared but are not yet in default catalogs.
Users must grant Accessibility permission on macOS and keep temporary directories private. An optional JEV helper can send screen descriptions to a model provider, but snapshots may contain private content and should not be logged publicly.
What it means for you
For developers building AI agents, agent‑desktop provides a reliable way to automate GUI tasks without fragile pixel matching, simplifying error handling. Its open‑source, cross‑platform design helps hobbyists and researchers experiment with free or locally hosted models, complementing the free chat models highlighted on FreeLLMArena.
Sources
- Vercel Labs Open-Sources agent-desktop for Agent Computer Use, MarkTechPost, 31 Oct 2026
- GitHub - vercel-labs/agent-desktop: Agent Desktop gives any agent reliable computer use on the desktop. Built with Rust, it sees any app's real UI structure through OS accessibility trees and operates, github.com (official), 31 Oct 2026
Written automatically from the sources above, then checked by our code: every number and name must appear in them, and no sentence may copy a source. Spotted a mistake? Report it.
Which AI can you use for free today?
Every free AI model, ranked by blind human votes, with the memory and speed it needs on your own GPU.