The paper "You don't need long context" investigates the operational and economic inefficiencies of maintaining large input contexts in Large Language Models (LLMs). It demonstrates that the computational cost of generating tokens scales linearly with context depth, leading to significant increases in latency and expenses as context windows grow. The authors argue that frontier models can maintain task accuracy while utilizing significantly shorter contexts, provided that appropriate "compaction" strategies are employed.

Core Problem and Motivation

The researchers observe that popular LLMs have evolved to support increasingly large context windows (up to $1$M tokens), which results in considerable performance degradation. Specifically, the authors show that for models like Qwen 3 Coder Next, operating at $256$K context versus $16$K increases the monetary cost of token generation by up to $7\times$. Furthermore, inter-token generation latency can double at longer contexts. The paper challenges the necessity of these long-context regimes, identifying that most "long-horizon" (sequential action) and "long-input" (large-scale document) tasks do not require the entire history to be present at every step.

Research Questions

The study centers on three questions:

1. Can long-horizon tasks be completed without long contexts?
2. Can long-input tasks be completed without long contexts?
3. Can long-horizon tasks be completed taking no more than one action per context?

Methodology: Context-Management Regimes

The authors introduce and compare three context-management regimes to test the efficacy of keeping contexts short:

• Full-History Accumulation (FHA): The baseline approach where every action and environment observation is appended to the conversation history until task completion.
• End-of-Conversation Compaction (EoCC): A strategy where the model, upon reaching a predefined token threshold, is instructed to generate a condensed, task-relevant state representation. This representation is passed to a fresh conversation, effectively "resetting" the context while preserving necessary information.
• Single-Action Compaction (SAC): An extreme regime where the model performs exactly one action per conversation. The semantic state is updated and passed forward, maintaining the shortest possible active conversational context.

To handle "long-input" tasks (e.g., searching across multiple financial reports), the authors propose converting them into "long-horizon" tasks. Instead of loading full documents into the context, documents are stored in the filesystem and accessed via selective input reading (using tools like sed or awk), which triggers as an explicit agent action.

Key Findings

• Accuracy Preservation: The authors report that frequent compaction (EoCC) allows models to maintain task accuracy on long-horizon benchmarks (such as SWE-bench and FRAMES) even when the context is restricted to small fractions of the model's total capacity (e.g., $16$K–$50$K tokens).
• Efficiency Gains: By reducing the context window size from $256$K to $16$K, the authors demonstrate massive throughput improvements—speedups of $3.6\times$ for GLM 5.2 and $7\times$ for Qwen 3 Coder Next.
• Semantic State Space Models (SSSMs): The success of the SAC approach, where models perform complex software engineering tasks with a single action per context, suggests a paradigm shift toward SSSMs. This implies that language models can function as reactive agents that maintain only a distilled semantic state rather than a verbatim history of their inputs and prior actions.

The study concludes that the industry's focus on expanding context window sizes is largely unnecessary for standard agentic workflows. The findings suggest that by replacing "context bloating" with semantic state management, developers can achieve significant cost savings and latency improvements without compromising the model's ability to solve complex, multi-step problems.