The paper introduces Group reward-Decoupled Normalization Policy Optimization (GDPO), a novel policy optimization method for multi-reward Reinforcement Learning (RL) that addresses limitations of the widely adopted Group Relative Policy Optimization (GRPO).

The core problem identified with GRPO in multi-reward settings is its "reward signal collapse." When optimizing multiple objectives, GRPO typically sums all individual reward components ($r^{(i,j)}_{\text{sum}} = r^{(i,j)}_1 + \dots + r^{(i,j)}_n$) for the $j$-th response to question $i$. It then calculates a group-relative advantage by normalizing these aggregated rewards across all $G$ responses in a group: $A^{(i,j)}_{\text{sum}} = \frac{r^{(i,j)}_{\text{sum}} - \text{mean}\{r^{(i,1)}_{\text{sum}}, \dots, r^{(i,G)}_{\text{sum}}\}}{\text{std}\{r^{(i,1)}_{\text{sum}}, \dots, r^{(i,G)}_{\text{sum}}\}}$. This approach, formalized in the multi-reward GRPO objective $\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{(q_i, o_j) \sim \mathcal{D}, \{o_j\}^G_{j=1} \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \left[ \frac{1}{G} \sum_{j=1}^G \frac{1}{|o_j|} \sum_{t=1}^{|o_j|} \min \left( s_{i,t}(\theta) A^{(i,j)}_{\text{sum}}, \text{clip}(s_{i,t}(\theta), 1-\epsilon, 1+\epsilon) A^{(i,j)}_{\text{sum}} \right) \right]$ where $s_t(\theta) = \frac{\pi_\theta(o_t | q, o_{<t})}{\pi_{\theta_{\text{old}}}(o_t | q, o_{<t})}$, compresses distinct reward combinations into identical advantage values. For instance, in a two-binary-reward, two-rollout scenario, GRPO maps combinations like (0,1), (0,2), and (1,2) (total rewards of rollouts) into the same normalized advantage of $(-0.7071, 0.7071)$, and (0,0), (1,1), (2,2) into $(0,0)$. This loss of information reduces the resolution of the training signal, leading to suboptimal convergence and potential training failures. The paper also demonstrates that removing the standard deviation normalization term (GRPO w/o std) only slightly increases the distinct advantage groups and can lead to training instability.

GDPO resolves this by decoupling the normalization process. Instead of normalizing the sum of rewards, GDPO first performs group-wise normalization for *each individual reward* separately. For the $k$-th reward of the $j$-th rollout for question $i$, the normalized advantage is computed as:
$A^{(i,j)}_k = \frac{r^{(i,j)}_k - \text{mean}\{r^{(i,1)}_k, \dots, r^{(i,G)}_k\}}{\text{std}\{r^{(i,1)}_k, \dots, r^{(i,G)}_k\}}$.
This step ensures that the relative differences within each reward dimension are preserved.
After individual reward normalization, these normalized advantages are aggregated, potentially with specific weights $w_k$ to reflect priorities: $A^{(i,j)}_{\text{sum}} = \sum_{k=1}^n w_k A^{(i,j)}_k$.
Finally, GDPO applies a batch-wise normalization to this sum of multi-reward advantages:
$\hat{A}^{(i,j)}_{\text{sum}} = \frac{A^{(i,j)}_{\text{sum}} - \text{mean}\{A^{(i',j')}_{\text{sum}} | i' \in \mathcal{D}_{\text{Batch}}, j'=1,\dots,G\}}{\text{std}\{A^{(i',j')}_{\text{sum}} | i' \in \mathcal{D}_{\text{Batch}}, j'=1,\dots,G\} + \epsilon}$.
This batch-wise normalization ensures the magnitude of the advantage remains stable regardless of the number of rewards, improving training stability. GDPO's decoupled normalization preserves fine-grained distinctions, e.g., mapping (0,1) to $(-0.7071, 0.7071)$ and (0,2) to $(-1.4142, 1.4142)$, providing a more expressive training signal. GDPO consistently yields a substantially larger number of distinct advantage groups compared to GRPO and GRPO w/o std as the number of rollouts or rewards increases.

The paper also provides insights into effectively incorporating priority variations among objectives. While assigning different weights ($w_k$) to normalized advantages ($A^{(i,j)}_{\text{sum}} = w_1 A^{(i,j)}_1 + \dots + w_n A^{(i,j)}_n$) is common, its effectiveness is limited when objectives differ significantly in difficulty. Easier rewards might dominate optimization despite lower weights. A more robust approach is conditioning rewards, where an easier reward $r_k$ is only awarded if a more difficult, prioritized reward $r_l$ meets a threshold $t$: $r_k = \begin{cases} r_k, & \text{if } r_l \ge t \\ 0, & \text{otherwise} \end{cases}$. This forces the model to prioritize the more challenging objective.

GDPO's effectiveness is demonstrated across three tasks:

1. Tool Calling: Optimizing correctness ($\mathcal{R}_{\text{correct}} \in [-3,3]$) and format compliance ($\mathcal{R}_{\text{format}} \in \{0,1\}$). GDPO consistently achieves higher correctness and format reward scores, outperforming GRPO and GRPO w/o std (which failed to learn format compliance) on the BFCL-v3 benchmark for Qwen2.5-Instruct (1.5B and 3B models).
2. Math Reasoning: Balancing accuracy and adherence to a length constraint. GDPO showed better convergence and prevented training collapse observed with GRPO, leading to higher accuracy while maintaining response shortness for DeepSeek-R1 and Qwen3-4B-Instruct models.
3. Coding Reasoning: Jointly optimizing code-generation accuracy, length constraints, and bug ratio. GDPO's superior performance across these diverse multi-reward settings underscores its generalizability and effectiveness over GRPO.