Agent Lightning is a novel, flexible, and extensible framework designed to enable Reinforcement Learning (RL)-based training of Large Language Models (LLMs) for any AI agent. It addresses the significant challenges in applying RL to complex, dynamic agent behaviors, which current methods struggle with due to their focus on static, single-call tasks. The core innovation lies in its complete decoupling of agent execution from RL training, allowing seamless integration with existing agents (built with frameworks like LangChain, OpenAI Agents SDK, AutoGen, or from scratch) with minimal code modifications.

The framework's methodology is grounded in a rigorous formulation of agent execution as a Markov Decision Process (MDP). This allows for a unified data interface that abstracts away the underlying orchestration logic and agent framework specifics.

Unified Data Interface:
Agent execution, similar to software execution, can be conceptualized as a directed acyclic graph (DAG) of component invocations. Agent Lightning simplifies this by focusing on key state changes relevant for RL optimization.

1. State and Call: A "state" at any timestep $t$ for the $k$-th execution of a task $x$, denoted $state_t(x, k) = \{variable_{x,k,t}^i\}_{i=1}^V$, represents a snapshot of the agent's execution, comprising "semantic variables" that encapsulate critical program intents and evolve over time. Changes to these semantic variables occur through "Component Invocations" or "calls." A call $call_{x,k}^i = (meta_{x,k}^i, input_{x,k}^i, output_{x,k}^i)$ involves a component $C_i \in M \cup T$ (where $M$ is the set of LLMs and $T$ is the set of tools) producing $output_{x,k}^i$ from $input_{x,k}^i$. Both $input_{x,k}^i$ and $output_{x,k}^i$ are semantic variables at specific timesteps, representing the visible and modified parts of the state during an invocation.
2. Reward and Dataset: Each agent execution is augmented with scalar reward signals $\{r_1, \dots, r_N\}$, where $r_i \in \mathbb{R}$ evaluates the quality of the $i$-th invocation. An execution with rewards is represented as $execution_R(x, k) = \{(call_{x,k}^i, r_{x,k}^i)\}_{i=1}^N$. Rewards can be intermediate (e.g., successful tool use) or terminal (overall task success).

Markov Decision Process (MDP) Formulation:
For an LLM to be optimized as a policy model, its decision-making is modeled as a Partially Observable Markov Decision Process (POMDP) $M = (S, O, A, P, R)$:

• $S$: The space of states, i.e., $state_t \in S$.
• $O$: The observation space, corresponding to inputs visible to the policy LLM, i.e., $input_t \in O$.
• $A$: The action space, where an action $a_t$ is the entire token sequence generated by a single LLM invocation, i.e., $output_t \in A$.
• $P$: The (unknown) transition dynamics $T(s'|s,a)$.
• $R$: The reward function $R(s,a)$.

The LLM observes $input_t$, generates $output_t = (y_{t,1}, \dots, y_{t,N_t})$, which is treated as a single action $a_t$. After executing $a_t$, the agent transitions to $state_{t+1}$, receiving reward $r_t = R(s_t, a_t)$. The objective is to maximize the total return $\mathcal{R} = \sum_{t=1}^T r_t$.
For RL training, the system extracts $execution_{RL}(x, k) = \{(input_{x,k}^t, output_{x,k}^t, r_{x,k}^t)\}_{t=1}^T$, where $output_{x,k}^t = \pi_\theta(input_{x,k}^t)$ for the policy LLM $\pi_\theta$. This approach focuses solely on the LLM's input and output, abstracting away complex agent logic, enabling RL application to diverse agents. This formulation flexibly extends to single-LLM multi-agent scenarios (where a single LLM adopts different roles based on prompts) and can potentially be extended to multi-LLM settings using Multi-Agent RL.

LightningRL Algorithm:
Building on the MDP formulation, Agent Lightning introduces LightningRL, a hierarchical RL algorithm designed specifically for agent training. While the detailed algorithm is not fully described in the provided text, it features a crucial "credit assignment module." This module attributes trajectory-level returns to individual responses generated by each LLM call, enabling the optimization of the policy. LightningRL is designed to be fully compatible with existing single-turn RL methods for LLMs, allowing efficient and effective training by avoiding issues like excessively long sequences and supporting flexible context construction.

Training-Agent Disaggregation Architecture:
To implement this, Agent Lightning employs a Training-Agent Disaggregation (TA Disaggregation) architecture, which cleanly separates RL training from agent execution:

• Lightning Server: Acts as the controller for the RL training system, managing the training process and exposing an OpenAI-like API for the updated model to clients.
• Lightning Client: Comprises two parts:
  1. Communication Component: Handles data transmission and reception with the Lightning Server.
  2. Agent Runtime: Runs the agent and performs data collection. Crucially, it transparently manages agent execution and trajectory collection without requiring code modifications to the agent itself. This design leverages existing observability frameworks (e.g., OpenTelemetry) for trajectory collection, connecting monitoring data directly to RL training. The agent runtime also facilitates an Automatic Intermediate Rewarding (AIR) mechanism, which assigns intermediate rewards based on system monitoring signals (e.g., tool call success/failure), effectively mitigating the sparse reward problem common in RL.

Agent Lightning's key advancements include: (1) full decoupling of agents and RL training, enabling application to any AI agent with near-zero code changes; (2) a unified data interface and MDP formulation, transforming diverse agent execution data into training trajectories; (3) the hierarchical LightningRL algorithm with credit assignment; and (4) the TA Disaggregation architecture, which provides a standardized training service, integrates observability, and supports automatic intermediate rewarding. Experimental results across text-to-SQL, RAG, and math tool-use tasks demonstrate stable and continuous performance improvements, showcasing its practical potential for real-world agent training and deployment.