The paper introduces Reinforcement Learning with Self-Verifiable Rewards (RLSVR), a training paradigm designed to extend the scalability and rigor of Reinforcement Learning with Verifiable Rewards (RLVR) to open-ended, non-verifiable tasks. While traditional RLVR relies on deterministic verifiers ($V(x, y) \in \{0, 1\}$) such as unit tests or math checkers, RLSVR utilizes task transformation ($\Phi$) to map open-ended objectives to proxy environments where rewards are generated automatically from internal rules or interaction outcomes.

Core Methodology: RLSVR and SpyRL

RLSVR functions by injecting a latent variable $z$ (ground truth) into a proxy environment. This $z$ remains hidden from the policy $\pi_{\theta}$, but allows the environment to pose questions about the task outputs that are deterministically verifiable against $z$.

The authors instantiate this framework with SpyRL, a multi-agent social deduction game. In each training epoch, $n$ agents perform a task (e.g., summarization, creative writing):

1. Asymmetric Information Allocation: $n-1$ civilian agents receive full information $x$, while one spy agent receives a degraded version $g(x)$.
2. Performing Stage: Agents produce outputs $y_i \sim \pi_{\theta}^{P}(\cdot | o_i, \tau)$. To avoid detection, the spy is incentivized to minimize the quality gap between their output and the civilians', thereby optimizing task performance.
3. Detection Stage: Agents vote on the identity of the spy using $\pi_{\phi}^{D}(\cdot | (o_i, Y))$. The detection reward $r_D$ is based on the correctness of the vote against the ground-truth index $u$ assigned by the environment: $r_D = I[v_i = u]$.

Mathematical Formulation

The detection advantage is calculated using group-based normalization to stabilize training:
$$AD_i = \frac{r_D^i - \mu_D}{\sigma_D + \epsilon}$$
where $\mu_D$ and $\sigma_D$ are the mean and standard deviation of rewards within a group. The performing reward $r_P$ is zero-sum, defined by the discrepancy between the votes received by the spy and the average votes received by civilians ($\bar{m}_c$):
$$r_u^P = -\beta(m_u - \bar{m}_c), \quad r_{c_j}^P = \frac{\beta}{n_c}(m_u - \bar{m}_c) - \lambda(m_{c_j} - \bar{m}_c)$$
This objective is optimized using a clipped surrogate loss function with KL regularization:
$$L^P(\theta) = -\mathbb{E}\left[\frac{1}{n} \sum_{k \in \{u\} \cup C} \min(\rho_{k,t} A_k^P, \text{clip}(\rho_{k,t}, 1-\epsilon, 1+\epsilon) A_k^P)\right] + \beta_P KL(\pi_{\theta}^P \| \pi_{\text{ref}}^P)$$

Key Contributions and Results

• Dynamic Equilibrium: The model employs an alternating optimization strategy between the performer and the detector. This prevents stagnation by ensuring that as the performer improves (becoming harder to detect), the detector is forced to sharpen its discriminative capability.
• Performance: SpyRL demonstrated significant improvements over existing self-improvement methods. On Qwen3-8B, it achieved a 75.4% and 77.3% win rate on summarization and creative writing, respectively. It also provided consistent gains of 8.97% and 6.16% on mathematical reasoning benchmarks compared to baseline models.
• Conclusion: The authors conclude that by transforming tasks into environments with latent, verifiable ground truths, models can undergo large-scale self-improvement on subjective tasks without the bias or costs associated with human-in-the-loop or LLM-as-a-judge approaches.