View original
GitHub - trycua/cua: Open-source infrastructure for Computer-Use Agents. Sandboxes, SDKs, and benchmarks to train and evaluate AI agents that can control full desktops (macOS, Linux, Windows).
Service

GitHub - trycua/cua: Open-source infrastructure for Computer-Use Agents. Sandboxes, SDKs, and benchmarks to train and evaluate AI agents that can control full desktops (macOS, Linux, Windows).

trycua
2026.05.16
·GitHub·by 임근석/부산/NLP
#AI Agents#Automation#Computer Control#macOS#Sandboxing

Key Points

  • 1Cua is a comprehensive platform designed for building and deploying AI agents that autonomously interact with computers across various operating systems, including background automation on macOS.
  • 2It provides a unified API for creating agent-ready sandboxes on Linux, macOS, Windows, and Android, enabling agents to see screens, click, type, and perform gestures within isolated environments.
  • 3The suite further includes CuaBot for collaborative agent workflows, Cua-Bench for benchmarking and training computer-use agents, and Lume for high-performance macOS virtualization on Apple Silicon.

Cua is a comprehensive framework designed for building, benchmarking, and deploying AI agents that interact with computer systems. It addresses the challenge of enabling agents to perceive and control graphical user interfaces (GUIs) across diverse operating systems and environments, from local machines to cloud infrastructures.

The framework comprises several interconnected components, each serving a distinct purpose:

  1. Cua Driver: This component focuses on background computer-use on macOS. Its core methodology involves enabling agents to programmatically click, type, and verify UI elements in native macOS applications without disrupting the user's cursor, focus, or active Space. A key technical feature is its ability to interact with non-Accessibility (AX) surfaces, such as Chromium web content and canvas-based tools (e.g., Blender, Figma, DAWs, game engines), which are traditionally challenging for standard accessibility APIs. Interactions are recorded as replayable trajectories for later analysis or training. Installation is via a bash script that curls from GitHub.
  1. Cua (Agent-Ready Sandboxes): This is the central SDK for creating and controlling sandboxed computer environments. The core methodology provides a unified Python API for interacting with various operating systems regardless of their underlying runtime or host. The API allows for programmatic control of the sandbox, including shell command execution, screenshot capture, mouse input, keyboard input, and multi-touch gestures (for mobile environments).
The Sandbox class offers an ephemeral context manager for temporary environments. The Image class specifies the desired OS type, supporting:
  • Operating Systems: Linux, macOS, Windows, Android.
  • Runtimes/Environments: Linux containers, Linux VMs, macOS VMs, Windows VMs, Android VMs.
  • Deployment: Local (QEMU), BYOI (Bring Your Own Image) via .qcow2 or .iso files, and future cloud integration (cua.ai).
Example API usage:
pythonfrom cua import Sandbox, Image async with Sandbox.ephemeral(Image.linux()) as sb: result = await sb.shell.run("echo hello") screenshot = await sb.screenshot() await sb.mouse.click(100, 200) await sb.keyboard.type("Hello from Cua!") await sb.mobile.gesture((100, 500), (100, 200))

  1. CuaBot (Co-op computer-use): This CLI tool facilitates seamless integration of coding agents with Cua sandboxes. It enables agents to interact with individual sandbox windows that appear natively on the user's desktop, featuring H.265 video streaming for efficient visual feedback, shared clipboard functionality, and audio transmission. It provides built-in support for agents interacting with browser environments (cuabot chromium) and mobile devices, and can be used to run specific agents (cuabot claude, cuabot openclaw) within the sandbox or execute direct GUI actions (cuabot --screenshot, cuabot --type, cuabot --click).
  1. Cua-Bench (Benchmarks & RL Environments): Dedicated to evaluating computer-use agents, Cua-Bench provides a framework for running benchmarks on established datasets like OSWorld, ScreenSpot, and Windows Arena, as well as custom tasks. Its methodology supports the creation of base images (cb image create) and running evaluations with specified agents (cb run dataset ... --agent ...). Crucially, it enables the export of agent trajectories, which are vital for training new agent models and for reinforcement learning (RL) environments.
  1. Lume (macOS Virtualization): Lume is a utility for creating and managing macOS and Linux virtual machines, specifically leveraging Apple's Virtualization.Framework on Apple Silicon. This ensures near-native performance for virtualized environments. Lume's role within the Cua ecosystem is to provide high-performance VM infrastructure, particularly for macOS sandboxes required by Cua. An associated tool, lumier, offers a Docker-compatible interface for Lume VMs.

The cua ecosystem is structured into several Python packages: cuabot (multi-agent sandbox CLI), cua-agent (AI agent framework), cua-sandbox (SDK for sandboxes), cua-computer-server (driver for UI interactions), cua-bench (benchmarking), and lume (macOS/Linux VM management). The project operates under an MIT License.