The paper introduces rStar2-Agent, a 14B math reasoning model trained with agentic reinforcement learning (RL) to achieve frontier-level performance. It highlights limitations of current long Chain-of-Thought (CoT) models in handling complex problems prone to subtle errors or requiring creative shifts, advocating for "smarter thinking" via autonomous tool utilization, validation, and learning from feedback. The focus is on Python coding tools and interpreter as the environment for agentic RL.

Challenges in Agentic Reinforcement Learning:

1. Inherent Environment Noises: The complexity of coding tools introduces noise. Syntactically or logically incorrect code leads to error messages that can mislead the model, causing it to waste tokens on corrections rather than advancing reasoning.
2. Impact of Outcome-only Reward: Current outcome-only rewards (binary accuracy of the final answer) do not penalize undesirable intermediate behaviors. This means trajectories with incorrect intermediate tool calls can still receive positive rewards if the final answer is correct, reinforcing low-quality reasoning with tool errors (observed as ~10-15% tool-related errors in correctly answered trajectories even after training).

Core Innovations of rStar2-Agent:
rStar2-Agent proposes three key innovations to address these challenges and enable effective agentic RL at scale:

3. Efficient RL Infrastructure: A reliable Python code environment capable of high-throughput execution (45K concurrent tool calls, 0.3s average feedback) and a load-balanced rollout scheduler that dynamically allocates rollout requests based on KV cache capacity to maximize GPU utilization. This enables training on limited GPU resources (64 MI300X GPUs).
4. GRPO-RoC Algorithm: Group Relative Policy Optimization with Resampling on Correct (GRPO-RoC), an agentic RL algorithm that specifically addresses environment noise.
5. Efficient Agent Training Recipe: A multi-stage RL training strategy starting with non-reasoning Supervised Fine-Tuning (SFT) and progressing through RL stages.

Smarter Reasoning in a Code Environment (Methodology):
The model performs multi-turn rollouts, where it interacts iteratively with the code environment.

• Process: An initial system prompt and question are given. The model generates reasoning and a tool_call. If a tool call is present, the code block is extracted, executed by an environment service, and the output (tool_response) is appended to the trajectory under the user role. The model then takes this updated context to continue reasoning under the assistant role. This repeats until a final answer or max turns ($T$) is reached.
• Tool Call Format: A structured JSON format is used: $<tool_call>{"name": "execute_python_code_with_standard_io", "arguments": {"code": "...", "input": "..."}}</tool_call>$. The tool_response wraps standard output, IPython output, execution errors (with tracebacks), or timeouts. This API-like interface separates reasoning from execution and generalizes across tools.
• Prompt Template: The prompt instructs the model to generate reasoning within $<reason>...</reason>$ and the final answer within $<answer>...</answer>$, with the numeric result boxed by $\boxed{}$.

End-to-End Agentic Reinforcement Learning (Technical Details):

Preliminary: GRPO
rStar2-Agent builds upon the Group Relative Policy Optimization (GRPO) algorithm. For each question $q$ and ground-truth answer $a$, GRPO samples a group of $G$ rollout trajectories $\{o_i\}_{i=1}^G$ from the old policy $\pi_{\theta_{old}}$. The policy $\pi_\theta$ is optimized by maximizing the objective:
$$J_{\text{GRPO}}(\theta) = \mathbb{E}_{\substack{(q,a)\sim\mathcal{D}, \\ \{o_i\}_{i=1}^G \sim \pi_{\theta_{old}}(\cdot|q)}} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \min \left( \frac{\pi_\theta(o_{i,t}|q, o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}|q, o_{i,<t})} A_{i,t}, \text{clip}\left(\frac{\pi_\theta(o_{i,t}|q, o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}|q, o_{i,<t})}, 1-\epsilon, 1+\epsilon\right) A_{i,t} \right) - \beta D_{KL}(\pi_\theta \| \pi_{ref}) \right]$$
where $A_{i,t}$ is the estimated advantage, computed as:
$$A_{i,t} = \frac{r_i - \text{mean}(\{r_1, r_2, \ldots, r_G\})}{\text{std}(\{r_1, r_2, \ldots, r_G\})}$$
The reward $r_i \in \{0,1\}$ is outcome-only, based on whether the final answer (extracted from $\boxed{}$) matches the ground truth.
Modifications from standard GRPO include: removal of KL divergence penalty ($D_{KL}$) and entropy loss, and an increased $\epsilon_{high}$ to 0.28 (Clip-Higher) to encourage exploration.

GRPO-RoC: Group Relative Policy Optimization with Resampling on Correct
To address environment noise and improve trajectory quality while retaining outcome-only rewards, GRPO-RoC introduces the Resample on Correct (RoC) rollout strategy.

6. Oversampling: Instead of $G$ rollouts, $2G$ trajectories $\{o_i\}_{i=1}^{2G}$ are initially sampled.
7. Asymmetric Selection: These $2G$ rollouts are then downsampled to $G$ for policy updates, applying different selection strategies for negative and positive samples:
   • Negative Samples ($O_{neg}$): For zero-reward rollouts, $\lfloor \frac{G}{2} \rfloor$ trajectories are uniformly sampled from $O_{neg}$ to preserve diversity of failure modes.
   • Positive Samples ($O_{pos}$): For successful rollouts (reward=1), $\lfloor \frac{G}{2} \rfloor$ trajectories are sampled from $O_{pos}$ with probability *inversely proportional* to a ptotal penalty score, prioritizing higher-quality traces.
     • ptotal considers two types of intermediate issues:
       • Tool Call Errors (perr):

$$perr = \begin{cases} 0.5 & \text{if no tool calls} \\ \frac{\text{num of error tool calls}}{\text{num of all tool calls}} & \text{otherwise} \end{cases}$$
(A default $perr=0.5$ for no tool calls encourages tool usage.)

• Format Violations (pformat): Punishes undesirable formats (e.g., redundant $<reason>$ blocks or incorrect $<answer>$ tag counts).

$$pformat = \begin{cases} 1 & \text{if no <answer> tags} \\ \frac{\min(1, \text{num of <answer> tags}-1)}{\text{num of turns}} & \text{otherwise} \end{cases}$$
The total penalty is $ptotal = perr + pformat$.

This asymmetric sampling guides the model towards cleaner, higher-quality positive trajectories with correct tool usage and formatting, while still exposing it to diverse failure modes. The final objective is similar to GRPO but applied to the selected $G$ rollouts $\hat{O} = \{ \hat{o}_i \}_{i=1}^G$.

Large-Scale Agentic RL Infrastructure:

• Reliable High-Throughput Code Environment: Handles up to 45,000 concurrent tool calls with average feedback latency of 0.3 seconds.
• Load-Balanced Rollout Scheduler: Optimizes computational utilization by dynamically allocating rollout requests based on available KV cache capacity across GPUs.

Training Recipe:

• Non-Reasoning Cold Start for SFT: The training begins with SFT focused solely on general instruction following, coding tool usage, and formatting, without explicitly enhancing reasoning. This aims to avoid SFT overfitting and keeps initial average responses short, allowing RL to cultivate reasoning effectively.
• Multi-Stage RL Training: GRPO-RoC is applied in multiple stages, gradually increasing task difficulty and maximum training length. Each stage uses shorter rollout lengths (8K-12K) to encourage efficient reasoning strategies, unlike prior methods that scale rollouts to 16K-48K.

Results:
rStar2-Agent-14B achieves state-of-the-art math reasoning performance in only 510 RL steps within one week. On AIME24, it scores 80.6%, surpassing DeepSeek-R1 (671B), OpenAI o3-mini (medium), and Claude-Opus-4.0. It also shows strong generalization to scientific reasoning and agentic tool-use tasks beyond mathematics.