AgentOCR is an innovative framework designed to mitigate the computational and context-window bottlenecks associated with the rapidly expanding textual histories generated during multi-turn LLM agent training. By shifting from purely textual observation-action representations to compact, rendered images, the system leverages the superior information density of visual tokens to achieve efficient history management.

Core Methodology

The methodology is built upon three foundational pillars that transition agent history from token-heavy strings to optimized visual representations:

1. Visual Token Representation: Instead of storing raw textual logs, AgentOCR renders the observation-action history into compact image frames. This approach allows the LLM to process visual tokens, which are significantly more information-dense than standard textual tokens. The framework demonstrates a token consumption reduction of more than 50% while maintaining agent performance at over 95% of the original baseline.

2. Segment Optical Caching: To overcome the overhead of real-time image rendering, the framework implements a segment decomposition strategy. The history is broken down into hashable segments that are stored in a visual cache. By utilizing this caching mechanism, the system achieves a 20× speedup in the rendering process, ensuring that the latency introduced by image generation does not impede the reinforcement learning (RL) training loop.

3. Agentic Self-Compression: The system introduces an adaptive compression mechanism wherein the agent is trained to emit dynamic compression rates. Through compression-aware reward training, the agent learns to balance the trade-off between the preservation of critical state information and the degree of visual condensation. The objective is to maximize task performance $P$ given a constraint on token budget $T_b$:

$$\max_{\theta} \mathbb{E}_{\tau \sim \pi_{\theta}} \left[ \sum_{t=0}^{T} \gamma^t R_t \right] \quad \text{subject to} \quad \sum_{t=0}^{T} \text{tokens}(\text{OCR}(o_t, a_t)) \le T_b$$
where $o_t$ and $a_t$ represent observations and actions, and $\text{OCR}(\cdot)$ denotes the learned rendering/compression function.

Infrastructure and Environment

AgentOCR is integrated with the veRL infrastructure to facilitate scalable multi-turn agent training. It supports two distinct task environments:

• ALFWorld: Focused on embodied AI, utilizing PDDL and game files alongside MaskRCNN detectors to process state information.
• Search-based QA (Search-R1): Utilizes a dedicated retrieval server (built on Faiss-GPU and E5 embeddings) to handle information-intensive search tasks, necessitating a high-performance environment with significant GPU memory allocation (approximately 6GB per GPU for the retrieval server).

By combining visual tokenization with specialized caching and reward-driven compression, AgentOCR provides a robust solution for long-horizon agent tasks, effectively reconciling the competing demands of context-length constraints and the need for comprehensive historical state tracking.