Beyond Monotonic Progress: Retry-Supervised Value Learning for Robot Imitation
ReTVL leverages sparse retry event supervision to capture local mistake-recovery dynamics, outperforming progress-based models in fine-grained value estimation.
Key Findings
Methodology
ReTVL integrates global task progress calibration with local pairwise preference learning, utilizing sparse annotations of retry keypoints as supervision. The approach constructs state pairs around retry events, where the value is expected to decrease before correction and rebound afterward, reflecting mistake and recovery dynamics. The training combines absolute progress labels with preference-based loss, employing a categorical distributional value function for stability. The model uses soft window weighting to mitigate boundary ambiguity, ensuring robust learning of local value drops and rebounds. Experiments on real robot manipulation tasks demonstrate that ReTVL produces more nuanced value estimates than progress-only baselines, effectively identifying mistake segments and improving downstream policy learning.
Key Results
- Across four real-world tasks, ReTVL achieves a VOC score of 0.987 and a success-fail detection score of 1.000, closely matching or surpassing state-of-the-art methods like Robometer and RECAP-Value. It significantly improves local metrics such as Pre > Retry (0.740) and Drop AUC (0.797), indicating enhanced sensitivity to subtle errors.
- In policy learning experiments, ReTVL-based weighted behavior cloning boosts success rates from 41.25% (standard BC) to over 80%, outperforming progress-based and other value models. This demonstrates its practical advantage in real-world imitation tasks.
- Ablation studies confirm that preference loss and soft window mechanisms are essential for capturing mistake-recovery structures, with removal leading to substantial drops in local error sensitivity.
Significance
This work advances the field of robot imitation learning by moving beyond the monotonic progress assumption, enabling models to recognize and leverage local errors and corrections. Such fine-grained understanding is crucial for developing robust, autonomous robots capable of learning from imperfect demonstrations. The approach addresses a long-standing challenge in the domain—how to effectively utilize noisy, correction-rich data—thus opening new avenues for scalable, real-world robot training. Its implications extend to safety-critical applications like surgical robots or assistive devices, where precise error detection and correction are vital. Overall, ReTVL enhances both the theoretical understanding and practical capabilities of mistake-sensitive value learning, promising a more resilient and adaptable robot learning paradigm.
Technical Contribution
ReTVL introduces a novel combination of global progress calibration with local pairwise preference learning, utilizing sparse retry annotations as supervision signals. The core innovation lies in constructing preference pairs around retry keypoints, enforcing a local value drop before correction and rebound after, captured via a preference loss with soft window weighting. This design enables the value function to reflect local mistake-recovery dynamics, a significant departure from traditional monotonic progress models. The use of a categorical distributional value function further stabilizes estimates. The training strategy effectively integrates sparse local supervision with global task progress, resulting in a model that is both globally consistent and locally sensitive. This framework sets a new standard for mistake-aware value estimation in robot imitation learning.
Novelty
The fundamental novelty of this work is the systematic exploitation of retry events as sparse, local supervision signals for value learning. Unlike prior approaches that rely solely on coarse, monotonic progress labels, ReTVL explicitly models local value drops and rebounds around correction points, capturing the nuanced dynamics of imperfect demonstrations. This approach is the first to leverage sparse annotations of correction start points to learn mistake-sensitive value functions, providing a new perspective on how to utilize imperfect data effectively. Its integration of preference learning with global progress calibration offers a unique, multi-scale view of task execution, setting it apart from existing methods that treat errors as noise or discard them altogether.
Limitations
- The assumption that value around retry events follows a local degradation-and-recovery pattern may not hold in all real-world scenarios, such as exploratory retries or multi-stage corrections, limiting its generality.
- Dependence on sparse manual annotations of retry keypoints introduces labeling costs and potential inconsistencies, which could affect scalability.
- Current model focuses on relatively simple manipulation tasks; extending to complex, multi-modal, or long-horizon scenarios remains an open challenge.
- Real-time online adaptation and continual learning capabilities are not addressed, which are critical for deploying in dynamic environments.
Future Work
Future research could explore automatic detection and annotation of retry events, reducing manual labeling efforts. Integrating multi-modal sensory data, such as vision and tactile feedback, could further improve local error detection. Extending the framework to multi-task and multi-robot systems would enhance scalability and robustness. Additionally, developing online learning mechanisms to adapt to evolving environments and error types remains an important direction. Investigating the applicability of ReTVL in more complex, unstructured settings, and exploring its integration with hierarchical or meta-learning approaches, could significantly broaden its impact.
AI Executive Summary
Robotic imitation learning has long grappled with the challenge of imperfect demonstration data. Traditional models often assume that task progress is monotonic and smooth, ignoring the reality that human demonstrations frequently include mistakes, corrections, and retries. These imperfections, rather than being mere noise, carry valuable information about the task dynamics, error points, and recovery strategies. Recognizing this, recent research has sought to improve value and reward models to better reflect the true nature of task execution.
However, most existing approaches rely heavily on coarse, global success signals or stage-based progress labels, which fail to capture the subtle, local dynamics of errors and corrections. This limitation hampers the robot's ability to learn nuanced behaviors, especially in complex manipulation tasks where minor mistakes can significantly impact success. To address this, the authors propose ReTVL, a novel framework that leverages sparse annotations of retry events as local supervision signals. By focusing on the start points of corrective behaviors, ReTVL constructs a set of preference pairs that encode the expected value drops before correction and rebounds after. This approach enables the model to learn mistake-sensitive value functions that reflect the true local dynamics of task execution.
The core technical innovation lies in combining global progress calibration with local pairwise preference learning, using a soft window mechanism to mitigate boundary ambiguity. The value function is modeled as a distribution over discretized progress bins, providing stable and reliable estimates. During training, the model optimizes a mixed loss that integrates absolute progress labels with local preference signals, resulting in a value function that is both globally consistent and locally sensitive.
Extensive experiments on four real-world robot manipulation tasks demonstrate the effectiveness of ReTVL. The results show that it achieves near-ideal global value estimation, with a VOC score of 0.987, and significantly improves local mistake detection metrics such as Pre > Retry and Drop AUC. Importantly, when used to guide behavior cloning, ReTVL substantially boosts task success rates, outperforming baseline models by large margins. Ablation studies confirm that the preference loss and soft window mechanisms are critical for capturing local error-recovery structures.
This work marks a significant step forward in robot imitation learning, emphasizing the importance of mistake-aware value functions. By turning the often-overlooked correction behaviors into valuable supervision signals, ReTVL opens new avenues for developing more robust, adaptive robots capable of learning from imperfect demonstrations. Its implications extend beyond manipulation tasks, potentially impacting autonomous driving, service robots, and assistive systems. Despite some limitations, such as reliance on manual annotations and assumptions about local value patterns, the framework provides a versatile foundation for future research aimed at making robots more resilient and autonomous in complex, real-world environments.
Deep Analysis
Background
机器人学习领域近年来经历了快速发展，尤其是在模仿学习和强化学习的推动下，机器人逐渐具备自主完成复杂任务的能力。早期工作如行为克隆（Behavior Cloning）和逆强化学习（Inverse Reinforcement Learning）为机器人提供了模仿人类操作的基础，但在实际应用中，示范数据常常包含噪声、错误甚至重复修正行为。随着深度学习的发展，研究者开始利用视觉和语言信息结合的模型（如Vision-Language-Action模型）提升机器人理解和操作能力，代表性工作包括VLM（Vision-Language Models）和基于任务阶段的奖励建模方法。然而，这些方法大多依赖于全局成功指标或粗略的任务阶段划分，难以捕捉局部的错误和修正动态。近年来，基于任务进展的价值函数学习逐渐成为主流，诸如TOPReward、Robometer和RECAP-Value等模型通过估算任务的整体进展或成功概率，为机器人提供了密集的监督信号。然而，这些模型在面对复杂的、包含大量错误和修正的示范数据时，表现出对局部错误识别能力不足的问题。尤其是在实际操控任务中，细微的操作失误和修正行为对最终任务成功至关重要，但现有方法难以有效捕获这些细节，限制了机器人自主学习的鲁棒性和适应性。
Core Problem
核心问题在于，现有的价值和奖励模型多假设任务进展是单调递增的，忽视了示范中的局部错误和修正行为。这导致模型在面对带有错误的示范时，无法准确反映任务的真实状态变化，影响策略的学习效果。例如，在机器人操作中，操作者可能会出现偏差或失误，但随后通过修正行为恢复任务，这些动态变化未被充分捕捉。传统模型在训练过程中，通常只利用全局成功或失败标签，忽略了错误发生前后的局部信息，导致价值估计过于平滑，不能反映实际操作中的微妙变化。解决这一问题的关键在于引入对局部错误和修正行为的敏感性，使模型能够识别出潜在的错误段落，并在修正后反弹，从而提升模仿学习的效果和鲁棒性。
Innovation
ReTVL的创新点在于将稀疏标注的重试事件作为局部偏好监督信号，结合全局任务进展校准，构建一个既考虑整体任务状态又关注局部错误的价值模型。具体创新包括：
- �� 利用重试关键点的稀疏标注，定义偏好对，学习价值的局部下降与反弹关系。
- �� 引入偏好对比损失（Preference Loss），通过状态对的偏好关系，强化模型对潜在错误段的敏感性。
- �� 设计软窗口机制，缓解边界模糊带来的噪声，提高偏好学习的鲁棒性。
- �� 采用类别离散化的分布式价值预测，增强模型稳定性。
- �� 将局部偏好学习与全局进展校准结合，形成一个多尺度、多层次的价值学习框架。这些创新共同作用，使得模型在捕获细微错误和修正行为方面表现出显著优势，超越了传统的单调进展假设。
Methodology
- �� 数据准备：收集机器人示范数据，标注任务成功/失败标签，并在出现重试行为时标注重试关键点。每个轨迹包含多个状态，部分状态位于错误段落，部分在修正后。
- �� 全局进展校准：利用成功轨迹的时间比例作为目标值，训练模型预测离散化的任务进展类别，提供全局任务状态参考。
- �� 局部偏好构建：在重试关键点附近定义三个区域（前置、邻近、后续），采样状态对（h+, h−），假设在错误发生前价值较高，错误段下降，修正后反弹。
- �� 损失设计：引入偏好对比损失（Lpref），结合软窗口权重（w(h+, h−)）调整偏好强度，确保模型学习到局部价值的下降与反弹关系。
- �� 训练策略：同时优化绝对进展标签和偏好关系，使用混合损失（Lvalue = λabs * Labs + λpref * Lpref），增强模型对全局和局部信息的感知。
- �� 行为克隆：利用学习到的价值模型对示范片段进行加权，强调潜在有用的修正行为，抑制错误段落，从而提升策略性能。
Experiments
- �� 数据集：在四个真实机器人操控任务（如堆叠块、折叠毛巾、打开抽屉、拾取汤匙）中采集示范数据，每个任务包含30条标注轨迹用于训练，20条用于验证，另外有200条混合质量的示范用于策略训练。
- �� 比较基线：包括TOPReward、Robometer和RECAP-Value，覆盖全局任务进展、成功概率和局部偏好学习。
- �� 评价指标：全局指标包括VOC（值序相关性）和成功检测指标（S/F Det.），局部指标包括Drop AUC、Pre > Retry和Post > Retry。
- �� 超参数：偏好对比温度Tpref、软窗口宽度Δnear、重试范围的超参数设置，确保偏好学习的鲁棒性。
- �� 消融实验：去除偏好损失、软窗口或全局校准，验证各部分对性能的贡献。
Results
- �� ReTVL在全局指标上表现优异，VOC达0.987，成功检测指标达到1.000，显示其在整体任务进展估计上的准确性。
- �� 在局部指标方面，Pre > Retry（0.740）和Drop AUC（0.797）远超传统模型，表明其对错误段的识别能力显著增强。
- �� 在策略学习中，ReTVL导出的价值加权行为克隆成功率平均提升至80%以上，远高于标准行为克隆（41.25%）和仅依赖全局指标的模型（62.50%），验证其在实际任务中的实用性。
- �� 消融分析显示偏好损失和软窗口机制是性能提升的关键因素，缺失任何一部分都导致局部误差识别能力下降。
Applications
- �� 立即应用：该方法可以用于工业机器人中的高精度装配任务，提升机器人在复杂环境中的自主修正能力，减少人工干预。
- �� 长远愿景：未来可结合多模态感知技术，实现自主错误检测与修正的端到端系统，推动机器人在家庭、医疗、服务等多领域的广泛应用，甚至实现自主学习与适应。
Limitations & Outlook
- �� 该方法假设价值在重试事件附近呈现局部退化与修正的模式，可能无法适应所有类型的修正策略，例如探索性修正或多阶段修正。
- �� 依赖稀疏标注的重试关键点，标注成本较高，且在大规模应用中存在数据获取难题。
- �� 当前模型主要针对有限任务场景，复杂环境中的多模态信息融合和长时序依赖仍需深入研究。
Plain Language Accessible to non-experts
想象你在厨房里做饭。每次你尝试做一道菜，可能会不小心放错调料或者火候不对。这时，你会重新调整，尝试修正错误。虽然这些修正行为看起来像是失败，但其实它们是学习的宝贵经验。ReTVL就像是一个聪明的厨师助手，它能记住你每次修正的瞬间，知道你在什么地方出了错，又在什么地方成功了。它通过观察你每次的“重试”或“修正”，学习到哪里容易出错，哪里需要改进。这样，下一次你做菜时，它能提前告诉你哪些步骤可能出错，帮你避免失败。这个方法让机器人也能像人一样，学会在操作中不断修正，变得越来越聪明，最终能自己完成复杂的任务，就像一个经验丰富的厨师一样。
ELI14 Explained like you're 14
想象你在玩一款游戏，刚开始你总是会犯错，比如走错路或者按错按钮，但你会不断尝试修正。每次你失败后，都会学到一些新东西，知道哪里需要改进。ReTVL就像是一个聪明的朋友，它能观察你每次失败和修正的瞬间，记住哪些地方容易出错，哪些修正有效。它会用这些信息帮你下一次更快地做对。比如，你在拼积木时，可能会把一块放错地方，然后重新调整。ReTVL会学习到，错误的地方会让价值变低，而修正后价值会反弹。这样，机器人也可以通过学习这些“错误-修正”的模式，变得更聪明，学会自己修正错误，完成任务。它不仅能看到大局，还能注意到每个细节，帮助机器人变得更像人一样聪明。
Glossary
Value Function (价值函数)
在机器人学习中，用于估算当前状态或动作的任务完成程度的函数。它帮助机器人判断某个行为是否朝目标前进。
本文中，价值函数用于衡量机器人在某一状态下完成任务的潜力。
Progress Regression (进展回归)
一种通过连续或离散值估计任务完成程度的方法，假设任务进展是单调递增的。
传统方法如TOPReward使用此技术估算任务进度。
偏好学习 (Preference Learning)
通过比较两个状态的优劣，学习它们之间的相对价值关系。
ReTVL利用偏好学习捕获错误与修正的局部动态。
重试关键点 (Retry Keypoints)
示范中标注的，代表修正行为开始的关键时刻。
作为局部监督信号，指导模型学习价值变化。
偏好对比损失 (Preference Loss)
一种损失函数，用于优化状态对的偏好关系，强化模型对局部价值变化的敏感性。
ReTVL采用此损失学习价值的局部下降与反弹。
软窗口机制 (Soft Windowing)
在定义偏好关系时引入加权策略，缓解边界模糊带来的噪声。
增强模型对重试区域边界的鲁棒性。
类别离散化 (Categorical Discretization)
将连续的任务进展划分为多个类别，进行分布式预测。
提升价值估计的稳定性。
行为克隆 (Behavior Cloning)
模仿示范行为，训练策略复制示范动作。
本文中，用价值模型对示范片段加权，提升策略效果。
全局价值校准 (Global Progress Calibration)
通过全局标签调整模型输出，确保整体任务进展的合理性。
结合偏好学习，实现多尺度价值建模。
局部误差识别 (Local Error Detection)
识别示范中潜在的操作失误或偏差段落。
模型在训练中学习到潜在错误的价值下降。
Open Questions Unanswered questions from this research
- 1 如何在更复杂、多模态环境中扩展ReTVL的偏好学习机制，确保在多源信息融合时仍能有效捕获局部错误。
- 2 自动标注重试事件的技术方案，降低人工标注成本，提高大规模数据的可用性。
- 3 模型在长时序、多任务场景中的泛化能力，特别是在连续多任务切换或多机器人协作中的表现。
- 4 如何结合在线学习机制，使模型能在实际操作中持续更新和适应新的错误类型。
- 5 对不同类型的错误（如探索性错误、策略性错误）是否都能有效捕获，模型的局限性和改进空间。
Applications
Immediate Applications
工业机器人装配优化
利用ReTVL提升机器人在复杂装配任务中的错误识别与修正能力，减少人工干预，提升生产效率。
家庭服务机器人
增强机器人在家庭环境中的自主修正能力，应对家具摆放、物品抓取等多样化操作中的错误，提升用户体验。
医疗机器人辅助手术
通过学习手术中的微小操作错误与修正行为，提高机器人手术的安全性与精确性，降低风险。
Long-term Vision
自主学习与适应系统
结合ReTVL实现机器人在未知环境中的自主学习能力，逐步减少对人类示范的依赖，迈向真正的自主适应。
多机器人协作与修正
在多机器人系统中，利用误差与修正的价值信息，实现协作中的错误检测与动态调整，推动智能制造和服务的自动化升级。
Abstract
Human demonstrations for robot imitation learning often contain mistakes and corrective behaviors, such as imprecise grasps, object misalignment, unstable contact, and repeated attempts. While these segments are commonly treated as noisy or suboptimal data, they provide valuable evidence about when execution deviates from a desirable path and how task feasibility can be restored. However, existing reward and value models often rely on monotonic progress assumptions, which capture coarse task advancement but may overlook local execution errors and corrective behaviors in imperfect demonstrations. In this work, we propose ReTVL (ReTry-Supervised Value Learning), a framework for learning mistake-sensitive value functions from mixed-quality robot demonstrations by leveraging retry events as sparse supervision. ReTVL captures the local degradation-and-recovery structure around mistakes by combining global progress calibration with local pairwise preference learning induced by sparsely annotated retry keypoints. The learned value model is then used to reweight demonstration chunks for downstream behavior cloning, reducing the influence of harmful execution errors while preserving useful corrective behaviors. Experiments on real-robot manipulation tasks show that ReTVL produces more fine-grained value estimates than progress-based baselines and improves imitation learning from imperfect demonstrations.
References (20)
SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation
Qianzhong Chen, Justin Yu, Mac Schwager et al.
Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons
Anthony Liang, Yigit Korkmaz, Jiahui Zhang et al.
Compliant Residual DAgger: Improving Real-World Contact-Rich Manipulation with Human Corrections
Xiaomeng Xu, Yifan Hou, Zeyi Liu et al.
AgiBot World Colosseo: A Large-Scale Manipulation Platform for Scalable and Intelligent Embodied Systems
AgiBot-World-Contributors, Qingwen Bu, Jisong Cai et al.
Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations
Daniel S. Brown, Wonjoon Goo, S. Niekum
GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulation
Yunfei Li, Xiao Ma, Jiafeng Xu et al.
Imitation Learning from Imperfect Demonstration
Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao et al.
DART: Noise Injection for Robust Imitation Learning
Michael Laskey, Jonathan Lee, Roy Fox et al.
Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
Xue Bin Peng, Aviral Kumar, Grace Zhang et al.
Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos
Annie S. Chen, Suraj Nair, Chelsea Finn
Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics
Minttu Alakuijala, Reginald McLean, Isaac Woungang et al.
DemoDICE: Offline Imitation Learning with Supplementary Imperfect Demonstrations
Geon-hyeong Kim, Seokin Seo, Jongmin Lee et al.
Maximum Entropy Inverse Reinforcement Learning
Brian D. Ziebart, Andrew L. Maas, J. Bagnell et al.
HG-DAgger: Interactive Imitation Learning with Human Experts
Michael Kelly, Chelsea Sidrane, K. Driggs-Campbell et al.
Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality
Songyuan Zhang, Zhangjie Cao, Dorsa Sadigh et al.
ARM: Advantage Reward Modeling for Long-Horizon Manipulation
Yiming Mao, Zi-You Yu, Weixin Mao et al.
Algorithms for Inverse Reinforcement Learning
Andrew Y. Ng, Stuart J. Russell
Active Preference-Based Learning of Reward Functions
Dorsa Sadigh, A. Dragan, S. Sastry et al.
Apprenticeship learning via inverse reinforcement learning
P. Abbeel, A. Ng
LIV: Language-Image Representations and Rewards for Robotic Control
Y. Ma, William Liang, Vaidehi Som et al.