Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning
PEEU framework enables small multimodal models to achieve 30.6% success in web navigation by autonomous exploration and hindsight experience, outperforming larger models.
Key Findings
Methodology
The PEEU framework consists of two main stages: autonomous environment exploration and hindsight experience utilization. During exploration, the model autonomously sets goals based on initial webpage content, constructs exploration trees through goal-driven interactions, and collects trajectories. In the utilization stage, the model extracts atomic experiences by comparing pre- and post-action states using GPT-4, then employs inverse alignment to synthesize high-level, strictly constrained task descriptions. These experiences are used to train policy models via SFT and GRPO algorithms, emphasizing high-level task construction and experience reuse. The approach addresses the limitations of traditional coarse or atomic task training by focusing on high-level experience synthesis, significantly improving cross-website generalization.
Key Results
- On seven unseen real-world websites, the 7B model trained with PEEU achieved a success rate of 30.6%, surpassing baseline models such as the original training (7.8%) and larger models like Qwen2.5-VL-32B (22.7%).
- Across different model sizes (3B and 7B) and data scales (0.1k and 2k trajectories), PEEU consistently outperformed baselines, validating the importance of high-level task training and experience-based data synthesis.
- The TDHAF analysis framework revealed that high-level task training enhances out-of-domain multi-level generalization, especially in complex, multi-step tasks, demonstrating the effectiveness of high-level experience synthesis for robust planning.
Significance
This work advances the field of web automation by demonstrating that small models can achieve competitive performance through strategic experience synthesis and high-level task training. It addresses core challenges in task decomposition, environment generalization, and data alignment, which have hindered the deployment of lightweight autonomous agents in real-world scenarios. The approach offers a scalable solution for cross-website task planning, reducing reliance on large models and extensive labeled data. Its implications extend to broader autonomous systems, including intelligent assistants and robotic agents, by providing a framework for scalable, efficient, and generalizable task learning in complex environments.
Technical Contribution
The main technical contributions include the integration of autonomous goal-driven exploration with hindsight experience alignment, forming a novel training pipeline that emphasizes high-level task construction. The introduction of the TDHAF framework enables systematic analysis of hierarchical generalization capabilities, providing insights into how different task granularities affect out-of-domain transfer. The combination of GPT-4-based goal setting, trajectory extraction, and experience fusion with SFT and GRPO training algorithms results in a robust, scalable approach that significantly outperforms existing methods in real-world web navigation benchmarks. This work bridges the gap between exploration-based learning and supervised fine-tuning, opening new avenues for efficient, generalizable autonomous agents.
Novelty
This research is pioneering in systematically combining autonomous environment exploration, hindsight experience alignment, and hierarchical task analysis within a unified framework for web navigation. Unlike prior works that focus on atomic operations or coarse task training, this approach emphasizes high-level task synthesis and experience reuse, enabling models to generalize better across unseen websites. The introduction of TDHAF provides a novel analytical tool to understand hierarchical generalization, which is rarely addressed in existing literature. The method's ability to outperform larger models with fewer data and smaller parameters marks a significant step forward in scalable autonomous agent design.
Limitations
- The reliance on large pre-trained models like GPT-4 for goal setting and experience summarization incurs high computational costs, limiting scalability in resource-constrained environments.
- Exploration tree construction may struggle in highly complex or rapidly changing web environments, affecting the robustness of experience collection.
- The inverse alignment process depends on environment information quality; incomplete or noisy data can introduce biases, impacting training effectiveness.
- Current framework primarily targets static web environments; extending to dynamic, real-time scenarios remains challenging and requires further research.
Future Work
Future research will focus on reducing computational overhead by developing lightweight goal-setting modules, possibly through reinforcement learning or unsupervised methods. Enhancing environment modeling to handle dynamic web content and multi-modal inputs will be prioritized, aiming for more robust exploration in real-time scenarios. Additionally, integrating multi-agent systems and multi-task learning could extend the framework's applicability to complex automation tasks across diverse domains. Exploring transfer learning techniques to adapt high-level experience synthesis to other environments, such as robotic manipulation or industrial automation, also presents promising directions.
AI Executive Summary
Web automation has become a critical component of intelligent systems, enabling machines to perform complex tasks across diverse online platforms. Traditional approaches relied heavily on rule-based systems or extensive labeled datasets, which limited scalability and adaptability. Recent advances in multimodal large language models (MLLMs) have opened new possibilities, but their effectiveness in task planning and generalization remains constrained by weak planning capabilities and limited cross-site transfer.
This paper introduces the PEEU (Planning Experience Exploration and Utilization) framework, a novel approach designed to empower small-scale multimodal models with robust web navigation skills. Inspired by human learning, PEEU emphasizes autonomous exploration and experience-based learning. The framework involves two key stages: first, the model autonomously explores unfamiliar web environments by setting goals and constructing exploration trees through goal-driven interactions; second, it extracts atomic experiences from these trajectories using GPT-4, then employs inverse alignment to synthesize high-level, strictly constrained training data.
The core innovation lies in combining autonomous environment interaction with hindsight experience alignment, enabling the model to generate high-quality, aligned training samples that significantly improve its ability to generalize across unseen websites. This approach addresses the common problem of trajectory-goal mismatch and environment noise, which have hampered traditional coarse or atomic task training methods. By focusing on high-level task construction, the model learns more abstract, transferable skills, leading to superior out-of-domain performance.
Extensive experiments on seven real-world web navigation benchmarks demonstrate the effectiveness of PEEU. The 7B model trained with this method achieves a success rate of 30.6%, outperforming larger models and baseline approaches. The framework's robustness is further validated through the TDHAF analysis, which shows that high-level task training enhances multi-level generalization, especially in out-of-domain scenarios. These results highlight the importance of high-level experience synthesis and hierarchical task decomposition in building scalable, generalizable autonomous agents.
Overall, this work marks a significant step toward practical, efficient web automation. It offers a scalable solution that reduces reliance on large models and extensive data, paving the way for more adaptable and intelligent online assistants. Future directions include optimizing computational efficiency, extending to dynamic environments, and integrating multi-agent collaboration, aiming to realize fully autonomous, versatile web agents capable of complex multi-task operations across diverse online ecosystems.
Deep Analysis
Background
网页自动化作为人工智能应用的一个重要方向，经历了从基于规则的脚本和模板方法，到利用深度学习模型实现自主操作的演变。早期方法依赖于人工编码的规则，缺乏灵活性和泛化能力。随着深度学习的发展，特别是多模态大模型（如GPT系列、CLIP等）在理解和生成多模态内容方面取得突破，网页任务自动化逐渐走向智能化。代表性工作包括Wang等（2024a）提出的多模态网页导航系统，以及Li等（2025d）提出的任务分解策略。这些方法在特定任务上取得一定成功，但在跨网站泛化、多任务协作和复杂环境适应方面仍存在瓶颈。尤其是在实际应用中，网页内容不断变化，环境信息不完整，模型难以保持稳定高效的操作能力。近年来，研究者开始结合强化学习、模仿学习和迁移学习，试图提升模型在多样化网页环境中的适应性和泛化能力，但仍未解决根本性的问题。
Core Problem
当前网页自动化模型普遍面临规划能力不足、泛化能力有限的核心问题。具体表现为：模型在处理复杂、多层次、多任务场景时，缺乏有效的任务分解和经验利用机制，导致在新网站或变化环境中表现不佳。传统方法依赖大量标注数据或预定义规则，难以应对环境的动态变化和内容的不断更新。此外，粗粒度任务训练容易出现轨迹与目标不匹配的问题，影响模型的学习效率和效果。如何设计一种既能自主探索环境，又能高效利用过去经验，提升模型在不同网页环境中的迁移能力，成为亟待解决的关键难题。这不仅关系到网页自动化的实用性，也影响到多模态学习和自主智能体的未来发展。
Innovation
本研究的核心创新在于提出PEEU（Planning Experience Exploration and Utilization）框架，结合自主环境探索、逆向轨迹对齐和高层任务构建，系统性地解决了传统方法中的局限。具体创新点包括：
- �� 自主目标设定：模型在未见网页中自主设定目标，构建探索树，增强环境交互能力，减少对人工规则的依赖。
- �� 后见经验利用：通过比较轨迹前后状态，提取原子经验（如点击、滚动、输入等操作），并逆向对齐，生成高层次、严格匹配的任务描述。
- �� 任务分解分析：引入TDHAF，系统分析不同任务层级（低、中、高）上的泛化能力，为多层次任务规划提供理论基础。
- �� 经验融合与训练：结合SFT和GRPO算法，将高层任务和轨迹数据融合，训练出具有强泛化能力的策略模型。
- �� 实验验证：在七个真实网页导航任务中，模型表现优异，验证了高层任务和逆向经验的重要性，突破了以往仅依赖原子操作的局限。
Methodology
- �� 探索阶段：模型通过输入网页URL，利用GPT-4自动设定目标，生成任务列表D = {d1, d2, ..., dn}，每个任务代表待探索的目标。• 轨迹采集：模型在环境中执行动作，构建探索树R = (V, E)，V为网页状态集合，E为动作边，采集轨迹τ = {(s0, a0), ..., (sm, am)}，其中s0为起始状态（首页）。• 经验提取：利用GPT-4比较轨迹前后状态，提取原子经验ϵt = M(st, at, st+1)，形成轨迹经验μ = (ϵ1, ..., ϵT)。• 逆向对齐：将经验μ融合，利用映射Φ生成高层任务描述˜d，确保轨迹严格匹配目标。• 训练阶段：采用SFT和GRPO算法，利用高层任务和轨迹数据训练策略模型π：S × H × ˜D → A。• 评估：在七个真实网页任务中，采用轨迹成功率作为性能指标，验证模型的跨网站泛化能力。• 实验设置：保持轨迹数量一致，使用GPT-4进行目标设定和总结，确保公平性。模型在不同规模（3B、7B）和数据量（0.1k、2k）下进行测试，验证方法的普适性。
Experiments
实验在多个真实网页场景中进行，包括购物、搜索、地图导航、学术资源等类别。训练数据来自0.1k和2k轨迹，测试在七个未见网站上进行。基线模型包括Atomic-Prompt、Trajectory-Prompt、Coarse和Atomic方法，采用相同的训练参数确保公平。评价指标为轨迹成功率（Step Success Rate）。训练采用SFT和GRPO，参数调优确保模型在有限数据下的学习效果。通过不同模型规模和数据规模的对比，验证PEEU在跨网站泛化中的优势。结合TDHAF分析不同任务层级的泛化能力，验证高层任务训练的有效性。实验结果显示，采用PEEU的模型在7B规模下成功率达到30.6%，明显优于传统方法和大模型，验证了高层任务和逆向经验的必要性。
Results
- �� 在七个未见网页任务中，7B模型采用PEEU训练后成功率达30.6%，远超未利用经验的模型（7.8%）和更大模型Qwen2.5-VL-32B（22.7%）。
- �� 在不同模型规模（3B、7B）和数据规模（0.1k、2k）下，PEEU表现出稳定的优越性，验证了高层任务训练的普适性。
- �� TDHAF分析显示，高层任务训练增强模型在OOD场景中的多层次泛化能力，特别是在复杂、多步骤任务中表现优越。
- �� 逆向对齐机制有效缓解轨迹与目标错配问题，提升模型的任务理解和执行能力。
Abstract
Multimodal web agents can assist humans in operating repetitive GUI tasks, where effective task planning is essential for decomposing complex tasks into executable actions. While small open source MLLMs are cost efficient and privacy preserving compared with commercial large models, they suffer from weak planning and limited cross website generalization. To address these limitations, we introduce the planning experience exploration and utilization (PEEU) method, which autonomously explores environments to discover experiences and utilizes hindsight experience to synthesize strictly aligned, high level training data. To quantitatively analyze the generalization behaviors driving this performance, we propose the task decomposition hierarchical analysis framework (TDHAF) to systematically study compositional generalization across three task granularities: low, middle and high levels. Our analysis reveals that mastering low level atomic skills does not guarantee high level planning competence, while high level task training yields stronger OOD generalization. Experiments on real world benchmarks demonstrate PEEU's superior effectiveness: our 7B model achieves 30.6% accuracy, outperforming the much larger Qwen2.5-VL-32B model. These demonstrate constructing hindsight high level tasks and leveraging experiences is crucial for OOD planning abilities of small MLLMs.
References (20)
Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents
Yu Gu, Boyuan Zheng, Boyu Gou et al.
Large Language Models for Planning: A Comprehensive and Systematic Survey
Pengfei Cao, Tianyi Men, Wen Liu et al.
GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration
Yue Fan, Handong Zhao, Ruiyi Zhang et al.
Small Language Models are the Future of Agentic AI
Peter Belcák, Greg Heinrich, Shizhe Diao et al.
A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models
Liang-bo Ning, Ziran Liang, Zhuohang Jiang et al.
Building Self-Evolving Agents via Experience-Driven Lifelong Learning: A Framework and Benchmark
Yuxuan Cai, Yipeng Hao, Jie Zhou et al.
GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents
Yuqi Zhou, Sunhao Dai, Shuai Wang et al.
RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment
Zhuoran Jin, Hongbang Yuan, Tianyi Men et al.
Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model
Tianle Li, Jihai Zhang, Yongming Rao et al.
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
Yaowei Zheng, Richong Zhang, Junhao Zhang et al.
Memory in the Age of AI Agents
Yuyang Hu, Shichun Liu, Yanwei Yue et al.
Scaling Web Agent Training through Automatic Data Generation and Fine-grained Evaluation
Lajanugen Logeswaran, Jaekyeom Kim, Sungryull Sohn et al.
Unlocking the Future: Exploring Look-Ahead Planning Mechanistic Interpretability in Large Language Models
Tianyi Men, Pengfei Cao, Zhuoran Jin et al.
A Troublemaker with Contagious Jailbreak Makes Chaos in Honest Towns
Tianyi Men, Pengfei Cao, Zhuoran Jin et al.
GPT-4V(ision) is a Generalist Web Agent, if Grounded
Boyuan Zheng, Boyu Gou, Jihyung Kil et al.
GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents
Run Luo, Lu Wang, Wanwei He et al.
Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models
Yunxin Li, Zhenyu Liu, Zitao Li et al.
Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents
Tianyi Men, Zhuoran Jin, Pengfei Cao et al.
Welcome to the Era of Experience
David Silver, Richard Sutton
ReAct: Synergizing Reasoning and Acting in Language Models
Shunyu Yao, Jeffrey Zhao, Dian Yu et al.