This paper introduces Reinforcement Learning from Internal Feedback (RLIF), a novel paradigm for enhancing Large Language Model (LLM) reasoning capabilities by leveraging intrinsic, self-generated signals without external rewards or labeled data. The motivation stems from the limitations of existing reinforcement learning approaches for LLMs: Reinforcement Learning from Human Feedback (RLHF) is costly and prone to bias due to reliance on human annotation, while Reinforcement Learning with Verifiable Rewards (RLVR) demands domain-specific verifiers and gold-standard solutions, limiting its applicability and generalizability.

The core objective of RLIF is to optimize a policy $\pi_\theta$ based on an intrinsic signal $u(q, o)$, formalized as:
$$\max_{\pi_\theta} E_{o \sim \pi_\theta(q)} [u(q, o) - \beta KL[\pi_\theta(o|q)\|\pi_{ref}(o|q)]]$$
where $q$ is an input query, $o$ is the generated output, $\pi_{ref}$ is a reference policy, and $\beta$ controls the KL divergence penalty.

Under the RLIF paradigm, the paper proposes INTUITOR, a method that utilizes the model's own confidence, termed self-certainty, as the sole intrinsic reward signal $u(q, o)$. Self-certainty is defined as the average KL divergence between a uniform distribution $U$ over the vocabulary $\mathcal{V}$ and the model’s next-token distribution:
$$\text{Self-certainty}(o|q) := \frac{1}{|o|} \sum_{i=1}^{|o|} KL(U \| p_{\pi_\theta}(\cdot|q, o_{<i})) = - \frac{1}{|o| \cdot |\mathcal{V}|} \sum_{i=1}^{|o|} \sum_{j=1}^{|\mathcal{V}|} \log (|\mathcal{V}| \cdot p_{\pi_\theta}(j|q, o_{<i}))$$
Here, $o_{<i}$ refers to previously generated tokens, and $p_{\pi_\theta}(j|q, o_{<i})$ is the model's predicted probability for token $j$ at step $i$. Higher self-certainty indicates greater confidence. This metric is chosen for its mode-seeking property and reported robustness against length biases, aiming to encourage the model to generate responses it deems more convincing and coherent.

INTUITOR integrates this self-certainty reward into the Group Relative Policy Optimization (GRPO) framework, which is commonly used in RLVR. GRPO's objective for policy $\pi_\theta$ is typically:
$$J_{GRPO}(\theta) = E_{q \sim P(Q),\{o_i\}_{i=1}^G \sim \pi_{\theta_{old}}(O|q)} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \min \left( c_{i,t}(\theta) \hat{A}_{i,t}, \text{clip}(c_{i,t}(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}_{i,t} \right) - \beta D_{KL}(\pi_\theta\|\pi_{ref}) \right]$$
where $c_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t} | q, o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t} | q, o_{i,<t})}$ is the importance sampling ratio, and $\hat{A}_{i,t}$ is the advantage estimate. In INTUITOR, the external verifiable reward is replaced by the self-certainty score ($u_i = \text{Self-certainty}(o_i|q)$), and the advantage $\hat{A}_{i,t}$ is computed by normalizing these scores within a sampled group of $G$ outputs:
$$\hat{A}_{i,t} = \frac{u_i - \text{mean}(\{u_1, u_2, \ldots, u_G\})}{\text{std}(\{u_1, u_2, \ldots, u_G\})}$$
This mechanism allows the model to learn by reinforcing outputs it assesses as highly confident, creating a self-improving loop without external human or algorithmic supervision.

Experiments using Qwen2.5-1.5B and Qwen2.5-3B models on the MATH dataset demonstrate that INTUITOR matches the performance of GRPO (which uses gold answers) on in-domain mathematical benchmarks (GSM8K, MATH500). Crucially, INTUITOR exhibits superior generalization to out-of-domain tasks like code generation (LiveCodeBench, CRUXEval), achieving a 65% relative improvement on LiveCodeBench for the Qwen2.5-3B model compared to no improvement for GRPO. Furthermore, INTUITOR enabled a Qwen2.5-1.5B model, initially producing repetitive and incoherent output, to generate structured reasoning chains and well-formed code. The paper highlights that INTUITOR leads to faster initial learning, improved instruction-following, and the emergence of long-form structured reasoning, where models learn to generate explicit reasoning steps before providing final answers, particularly evident in code generation tasks. This suggests that optimizing for intrinsic confidence encourages the model to generate more self-explanatory traces, leading to better understanding and performance.