The paper "Agentic Reasoning for Large Language Models" introduces agentic reasoning as a paradigm shift where Large Language Models (LLMs) are reframed as autonomous agents that plan, act, and learn through continual interaction, unifying reasoning with acting. This concept moves beyond traditional LLM reasoning, which is typically a static, one-shot prediction task, to an interactive, dynamic, and stateful process that enables planning, adaptation, and collaboration.

The survey systematically organizes agentic reasoning along three complementary dimensions, which characterize environmental dynamics and agent capabilities:

1. Foundational Agentic Reasoning: Establishes core single-agent capabilities like planning, tool use, and search in stable environments. Agents decompose goals, invoke external tools, and verify results.
2. Self-Evolving Agentic Reasoning: Focuses on how agents continually improve through cumulative experience, integrating feedback and memory-driven adaptation in evolving settings. This involves persistent updates of internal states and policies without full retraining.
3. Collective Multi-Agent Reasoning: Extends intelligence to collaborative scenarios where multiple agents coordinate roles, share knowledge, and pursue shared goals through communication and shared memory systems.

Across these dimensions, the paper distinguishes two complementary optimization settings:

4. In-context Reasoning: Scales inference-time interaction through structured orchestration, search-based planning, and adaptive workflow design, without modifying model parameters. It focuses on how agents navigate complex problem spaces dynamically during deployment.
5. Post-training Reasoning: Targets capability internalization by optimizing behaviors through reinforcement learning (RL) and supervised fine-tuning (SFT), consolidating successful reasoning patterns or tool-use strategies into the model's weights.

Core Methodology and Formalization:
The paper formalizes agentic reasoning as operating within a Partially Observable Markov Decision Process (POMDP) framework, defined by a tuple $\langle \mathcal{X}, \mathcal{O}, \mathcal{A}, \mathcal{Z}, \mathcal{M}, T, \Omega, R, \gamma \rangle$. Here, $\mathcal{X}$ is the latent environment state, $\mathcal{O}$ is the observation space, $\mathcal{A}$ is the external action space, $\mathcal{Z}$ is a reasoning trace space (e.g., latent plans, chain-of-thought), and $\mathcal{M}$ is the agent's internal memory/context. $T$ and $\Omega$ are transition and observation kernels, $R$ is the reward, and $\gamma$ is the discount factor.

The agent's policy is factorized into an internal thought process and an external action execution:
$\pi_{\theta}(z_t, a_t | h_t) = \pi_{\text{reason}}(z_t | h_t) \cdot \pi_{\text{exec}}(a_t | h_t, z_t)$
where $h_t = (o_{\le t}, z_{<t}, a_{<t})$ is the history up to timestep $t$. This decomposition highlights the core shift from traditional LLMs by making the "think-act" structure explicit. The objective is to maximize the expected return $J(\theta) = \mathbb{E}_{\tau} \left[ \sum_{t \ge 0} \gamma^t r_t \right]$.

Technical Details of Optimization Modes:

• In-Context Reasoning (Inference-Time Search): With frozen model parameters $\theta$, the agent optimizes the reasoning trajectory by searching over $\mathcal{Z}$ to maximize a heuristic value function $\hat{v}(h_t, z)$. Methods like ReAct perform greedy decoding over alternating thoughts ($z$) and actions ($a$). Tree-of-Thoughts (ToT) and similar Monte Carlo Tree Search (MCTS)-style approaches treat partial thoughts as nodes $u \in \mathcal{U}$ (e.g., a representation from $(h_t, z_t)$) and search for an optimal path $\tau^{\star} \in \arg \max_{\tau} \sum_t \hat{v}_{\phi}(u_t)$, where $\hat{v}_{\phi}$ is a heuristic evaluator or verifier. This corresponds to planning in $\mathcal{Z}$ without updating policy parameters.

• Post-Training (Policy Optimization): This paradigm optimizes $\theta$ to align the policy with long-horizon rewards $r_t$. While Proximal Policy Optimization (PPO) is common, Group Relative Policy Optimization (GRPO)-based methods are widely used. For a group of $G$ sampled outputs $\{y_i\}_{i=1}^G$ from the same prompt $q$, the GRPO objective is:

$L_{\text{GRPO}}(\theta) = \mathbb{E}_{q \sim P(Q)} \left[ \frac{1}{G} \sum_{i=1}^G \left( \min(\rho_i \hat{A}_i, \text{clip}(\rho_i, 1 - \epsilon, 1 + \epsilon) \hat{A}_i) - \beta D_{\text{KL}}(\pi_{\theta} \| \pi_{\text{ref}}) \right) \right]$
where $\rho_i = \frac{\pi_{\theta}(y_i|q)}{\pi_{\theta_{\text{old}}}(y_i|q)}$ and the group-normalized advantage is $\hat{A}_i = \frac{r_i - \mu}{\sigma + \delta}$, with $\mu = \frac{1}{G} \sum_{j=1}^G r_j$ and $\sigma = \sqrt{\frac{1}{G} \sum_{j=1}^G (r_j - \mu)^2}$. Advanced methods like ARPO and DAPO extend this for sparse rewards and stability.

• Collective Intelligence (Multi-Agent Reasoning): This extends the single-agent formulation to a decentralized partially observable multi-agent setting (Dec-POMDP). Each agent's observation includes a communication channel $\mathcal{C}$. For $N$ agents, the joint policy $\pi$ is composed of individual policies $\pi_i$. Communication acts as an extension of reasoning, where one agent's action can prompt another's internal reasoning. Centralized-Training/Decentralized-Execution (CTDE) paradigms are used to stabilize cooperative behaviors.

• Self-Evolving Agents (The Meta-Learning Loop): These agents optimize the agent system itself across episodes. Let $S_k$ denote the evolvable system state (e.g., explicit memories, tool libraries, code). A generic meta-update rule is $S_{k+1} \leftarrow U(S_k, \tau_k, F_k)$, where $F_k$ represents environmental feedback (rewards, execution errors). Types of evolution include:
  • Verbal Evolution: $S$ consists of textual reflections or guidelines (e.g., Reflexion updates $S$ by synthesizing error logs into linguistic cues).
  • Procedural Evolution: $S$ consists of executable tools or skills (e.g., Voyager synthesizes new code-based skills).
  • Structural Evolution: $S$ consists of the agent's source code or architecture, where an LLM acts as a mutation operator to search for superior reasoning algorithms.

The paper outlines its contributions as providing a conceptual framing of agentic reasoning, a systematic review across single-agent, adaptive, and multi-agent systems, a survey of real-world applications and benchmarks, and an identification of future challenges. It structures the survey into foundational, self-evolving, and collective reasoning, followed by applications, benchmarks, and open problems.