The paper introduces a functional demonstration of an AI agent, referred to as "Codex," capable of "Learning-by-Watching." This paradigm shifts the interaction model from explicit prompting to observational learning, allowing the model to acquire complex, multi-step workflows by mirroring human actions within a software environment.

Core Methodology

The methodology relies on an agentic framework that leverages "computer use" and "browser use" capabilities. The process follows a two-phase architecture:

1. Observational Learning Phase:

The user performs a repetitive task—such as publishing a YouTube video—while Codex monitors the session. During this phase, the model maps the user’s sequence of actions (mouse clicks, keystrokes, navigation) to specific data sources, such as external spreadsheets containing metadata, and asset repositories (thumbnails, caption files). The model identifies the causal relationships between input assets and target fields within a UI, effectively building a persistent "skill" template.

2. Autonomous Execution Phase:

Upon being provided with a new, distinct "package" of data, Codex retrieves the learned skill. It autonomously performs the task by:

• Data Orchestration: Integrating with external tools (plugins) or file systems to map new inputs to the learned schema: $D_{new} \rightarrow S_{target}$, where $S$ represents the required field in the UI.
• Agentic Navigation: Using computer-use capabilities to interact with the target software's interface (e.g., YouTube Studio). It navigates through menus, performs text entry, and executes file uploads without the need for API-level integration, mimicking the human's exact workflow.
• Verification Loop: The agent implements a post-action validation step to ensure that the persistent state $S_p$ matches the intended state $S_i$: $\forall s \in S_i, s_{actual} = s_{intended}$.

Technical Implementation

The system abstracts the complexity of UI interaction into a series of action primitives. Instead of requiring developers to script every step, the model learns the "topology" of a workflow. By observing the user, it defines a set of constraints and preferences that govern future executions. If $P$ represents the process and $C$ represents the context, the model functions as a function $f(C) \rightarrow A$ where $A$ is the sequence of actions that reproduces $P$. This allows the model to handle tasks ranging from video publishing to formatting pull requests and scheduling calendar invites by maintaining context across different sessions and applications.