"Progress Reward Modeling for Robotic Learning: A Comprehensive Survey" (arXiv:2607.21655) provides a systematic analysis of the emerging field of progress rewards in robotic reinforcement learning. It addresses the critical limitation of terminal success signals, which fail to provide dense, actionable feedback during complex, long-horizon manipulation tasks. The authors establish a unified framework for categorizing and evaluating progress models through a three-tiered taxonomy.

1. Interface of Progress Models

The interface defines the input-output mapping of the model. The model receives a state observation $s_t$ (or a sequence of observations $\tau = \{s_0, ..., s_t\}$) and, depending on the architecture, a goal specification $g$ or a task description. The output is a scalar reward signal $r_t = \phi(s_t, g; \theta)$ or a progress score that estimates the proximity to completion. The survey categorizes these interfaces based on whether the model is task-agnostic, goal-conditioned, or language-guided.

2. Methodological Construction

The internal mechanism of progress models is analyzed based on the assumptions of "progress" and how the reward is generated:

• Representation-based methods: These utilize latent spaces (e.g., VAEs, contrastive learning) to map observations to a manifold where distance $d(z_t, z_g)$ correlates with task progress.
• Reasoning-based methods: These employ large language models (LLMs) or visual-language models (VLMs) to explicitly parse task steps and evaluate state transitions against a predefined reasoning chain.
• Failure-aware modeling: This involves modeling the potential for regression or "undoing" of progress, where the reward function is designed as $\Delta P = P(s_t) - P(s_{t-1})$, often constrained to handle non-monotonicity.
• Supervision Sources: The survey distinguishes between self-supervised methods (using temporal distance or state-sequence ordering), human-in-the-loop preference learning, and simulated future alignment.

3. Benchmarks and Evaluation

The authors argue that the lack of standardized evaluation hinders progress in the field. They classify evaluation protocols into:

• Policy Performance: Measuring the success rate and efficiency of an RL agent trained using the specific progress reward function.
• Correlation Analysis: Quantifying the alignment between the learned progress signal and ground-truth metrics (e.g., Euclidean distance to goal, completion percentage in simulation).
• Robustness Benchmarking: Assessing the model’s performance under distributional shifts and variations in object/environment configuration.

Core Challenges and Future Directions

The paper identifies several unresolved issues:

• Non-monotonicity: The tendency for progress models to misinterpret exploration or necessary re-positioning as a decrease in progress.
• Sample Efficiency: The high data requirements for training reliable reward models, particularly in real-world deployments.
• Generalization: The inability of current models to transfer progress estimation across tasks with varying semantics.

The survey concludes by proposing a shift toward unified benchmarking environments and calls for community-driven development via repositories like "Awesome-Progress-Models" to standardize how the research community defines and rewards incremental success in robotic agents.