Agent Desktop is a Rust-based utility designed to enable reliable computer-use automation for AI agents by interacting with desktop applications through native operating system accessibility trees rather than relying on pixel-based screen analysis. By leveraging the accessibility APIs of macOS, Windows, and Linux, the tool provides a structured, deterministic interface for agents to perceive and manipulate desktop UIs.

Core Methodology and Technical Architecture

The system operates on an "observe-act-verify" loop. Unlike traditional computer-use models that interpret screenshots, Agent Desktop generates a JSON-based snapshot of the application’s UI hierarchy. Each interactive element is assigned a unique, stable reference (ref_id), which serves as a pointer for subsequent operations.

The methodology is governed by several technical constraints:

1. Accessibility Integration: The tool utilizes at-spi2-core for Linux, and respective native accessibility APIs for Windows and macOS to construct a tree of objects. This ensures that the agent receives programmatic information about UI elements (e.g., roles, names, and state) rather than raw visual data.
2. Snapshot-Driven Interaction: Commands are strictly bound to snapshots. A reference is valid only within the context of the snapshot it originated from. If an application's state changes, the system returns an STALE_REF or AMBIGUOUS_TARGET error, mandating that the agent capture a new snapshot. This design prevents the "silently wrong" behavior where an action might be performed on the wrong element due to UI layout shifts.
3. Execution Model: The CLI functions as a coordinator for native binaries. On macOS, it maintains a persistent private command host to ensure efficient access to native accessibility objects and to facilitate stateful interaction. The tool supports both a CLI-driven approach and a C-compatible Foreign Function Interface (libagent_desktop_ffi) for in-process integration, allowing developers to avoid the overhead of spawning a new process for every interaction.
4. Safety and Retries: The tool exposes a standardized JSON envelope for every command. Each envelope contains a disposition field that indicates whether an error is transient or fatal. An agent is instructed to retry only when error.disposition.retry is explicitly marked as safe, ensuring that operations remain predictable.

Capabilities and Platform Support

Agent Desktop supports a wide range of operational tasks, categorized into:

• Observation: Querying state via snapshot, find, and list-surfaces.
• Manipulation: Performing semantic actions such as click, type, set-value, and scroll.
• System Management: Controlling window positioning, application launching, clipboard operations, and notification management.

The system is platform-agnostic, handling the nuances of different OS architectures—such as the requirement for specific permissions on macOS or the reliance on at-spi2 and X11/Wayland protocols on Linux—while exposing a unified API. By avoiding pixel-guessing, Agent Desktop achieves higher reliability, as actions are tied to the internal programmatic identifiers of the OS, rendering the automation resilient to changes in theme, resolution, or window scaling.