The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators
Introduces Red Queen Gödel Machine (RQGM), a co-evolving framework with learned evaluators, achieving 1.35-1.86× efficiency gains in code, science, and proof tasks under non-stationary utilities.
Key Findings
Methodology
The RQGM framework builds upon recursive self-improvement principles, integrating learned evaluators as co-evolving agents alongside task agents. The core mechanism involves dividing search into epochs, during which a fixed evaluator provides a stationary utility signal, ensuring theoretical guarantees. At epoch boundaries, challenger evaluators are statistically tested against incumbents using Bayesian methods (e.g., Beta distribution) and replaced if outperforming the incumbent with high confidence. The system employs multi-agent architecture with role-specific task pools, allowing simultaneous evolution of task agents and evaluators. To prevent reward hacking and bias, adversarial objectives are incorporated, regulating evaluator strictness. The experimental setup spans coding benchmarks (Polyglot), scientific writing datasets (APReS), and proof grading (IMO). Results demonstrate significant improvements over static baselines, with reduced token consumption and enhanced performance metrics.
Key Results
- In Polyglot coding tasks, the RQGM achieved a test pass rate of 71.7%, surpassing the previous state-of-the-art (69.9%) while using 1.35-1.72× fewer tokens, demonstrating the efficiency of learned agent-as-a-judge signals.
- In scientific paper writing and reviewing, co-evolved writers increased acceptance rates from 21.8% to 40.5%, and graders improved ground-truth accuracy by 9%, indicating better evaluation and generation quality.
- In Olympiad-level proof tasks, the co-evolved scoring system achieved 9% higher true accuracy than static baselines, validating the approach’s robustness in complex reasoning scenarios.
Significance
This work addresses fundamental limitations of static evaluation in AI self-improvement systems by enabling dynamic, learned evaluation signals that co-evolve with task agents. It bridges the gap between theoretical guarantees and practical adaptability, paving the way for autonomous systems capable of long-term, open-ended optimization. The framework’s ability to incorporate adversarial objectives and regulate evaluation quality offers a promising avenue for developing more robust, fair, and versatile AI agents across diverse domains, including scientific discovery, automated programming, and peer review.
Technical Contribution
The paper extends the classical Gödel Machine paradigm by introducing controlled utility evolution via epoch-based evaluation and Bayesian evaluator replacement. It formalizes multi-role, multi-task architectures with shared workspaces, enabling simultaneous co-evolution of agents and evaluators. Theoretical guarantees are adapted from HGM to multi-epoch settings, ensuring convergence within epochs and improvement across epochs. The integration of adversarial objectives further enhances evaluation robustness. Empirically, the system demonstrates significant efficiency gains and performance improvements, validating the theoretical innovations.
Novelty
This is the first framework to treat evaluation as an evolvable component within recursive self-improvement, leveraging Bayesian methods for statistically sound evaluator replacement, and incorporating adversarial objectives to regulate evaluation strictness. Unlike prior static-benchmark approaches, RQGM dynamically adapts its utility functions, enabling open-ended, multi-task learning with theoretical guarantees, representing a substantial leap forward in autonomous AI development.
Limitations
- The reliance on epoch boundaries and Bayesian statistical thresholds may limit continuous adaptation, potentially reducing responsiveness to rapid environmental changes.
- Computational costs increase with the number of roles and tasks, especially during evaluator replacement and re-scoring, posing scalability challenges.
- The approach's effectiveness in highly complex, real-world scenarios remains to be fully validated, requiring further empirical testing and optimization.
Future Work
Future research will focus on developing more fine-grained, continuous utility adaptation mechanisms, reducing computational overhead, and extending the framework to multi-modal, real-world tasks such as autonomous scientific discovery. Additionally, integrating meta-learning techniques to accelerate evaluator and agent adaptation, and exploring more sophisticated adversarial strategies to ensure evaluation fairness, will be key directions. Long-term, the goal is to realize fully autonomous, self-improving AI systems capable of sustained innovation across diverse domains.
AI Executive Summary
In the rapidly evolving field of artificial intelligence, self-improving agents have emerged as a promising approach to achieving autonomous, adaptive systems. Traditional methods rely heavily on fixed evaluation benchmarks, which become increasingly inadequate as agents improve and environments change. These static benchmarks often lead to reward hacking, overfitting, and limited generalization, constraining the long-term potential of AI systems.
The present work introduces the Red Queen Gödel Machine (RQGM), a novel framework that fundamentally rethinks the evaluation process in self-improving AI. Inspired by biological evolution and the Red Queen hypothesis, RQGM treats evaluation as an integral part of the search process, allowing the utility function itself to evolve over time. This is achieved through a carefully designed epoch-based mechanism, where each epoch employs a fixed evaluator, ensuring theoretical guarantees of performance improvement. At epoch boundaries, challenger evaluators are statistically tested using Bayesian methods, such as Beta distributions, and replaced if they outperform the incumbent with high confidence. This process enables the system to adapt to changing tasks, environments, and objectives, without sacrificing convergence guarantees.
The core architecture of RQGM involves a multi-agent workspace, where task agents and evaluators co-evolve within shared roles. Each role contains multiple tasks and evaluation slots, allowing simultaneous optimization across diverse objectives. The system incorporates adversarial objectives to regulate evaluator strictness, preventing reward hacking and bias. This setup supports multi-task learning, open-ended search, and continual adaptation, making it suitable for complex real-world applications.
Empirical results across three domains—coding, scientific writing, and proof grading—demonstrate the effectiveness of RQGM. In coding benchmarks, the system surpasses previous state-of-the-art performance while using fewer tokens, highlighting the efficiency of learned agent-as-a-judge signals. In scientific paper review, co-evolved writers significantly increase acceptance rates, and graders achieve higher accuracy, indicating better evaluation and generation quality. In mathematical proof tasks, the system achieves a 9% improvement in true accuracy over static baselines, validating its robustness in reasoning.
This research marks a significant step toward autonomous AI systems capable of long-term, open-ended improvement. By enabling dynamic, learned evaluation signals that co-evolve with task agents, RQGM addresses key limitations of static benchmarks and reward hacking. Its theoretical foundations and empirical successes suggest broad applicability in scientific discovery, automated programming, and peer review. Despite current challenges such as computational costs and scalability, future work will focus on refining the mechanisms for continuous adaptation, reducing resource consumption, and extending the framework to multi-modal, real-world scenarios. Ultimately, RQGM paves the way for more resilient, versatile, and autonomous AI systems that can innovate and adapt indefinitely.
Deep Analysis
Background
The evolution of autonomous AI has long been driven by the pursuit of self-improvement capabilities. Early theoretical models like the Gödel Machine demonstrated the possibility of provably optimal self-modification, but practical implementations faced significant limitations due to computational intractability. Recent advances include hyper-heuristics, meta-learning, and large language models (LLMs) used as evaluators, which have improved performance in specific tasks. However, these systems largely depend on static evaluation benchmarks, which are vulnerable to reward hacking, overfitting, and obsolescence as agents improve. The challenge remains to develop systems that can adapt their evaluation criteria dynamically, akin to biological evolution where species co-adapt with changing environments. The integration of Bayesian statistical methods and multi-agent architectures has opened new avenues for such adaptive systems, but a comprehensive framework that guarantees convergence while supporting non-stationary objectives has been lacking. This paper builds upon these foundations, proposing a co-evolutionary approach that treats evaluation as an evolving component, enabling continuous, open-ended self-improvement.
Core Problem
Current self-improving systems are limited by their reliance on fixed, external evaluation benchmarks, which do not adapt to the agent's progress or environmental changes. This static evaluation leads to issues such as reward hacking, where agents exploit loopholes, and stagnation, where improvements plateau once benchmarks saturate. Moreover, many real-world tasks lack clear, objective benchmarks, making static evaluation infeasible. The core problem is how to enable agents to improve continuously in environments where the evaluation criteria themselves are uncertain, evolving, or unavailable. Achieving this requires mechanisms for evaluating progress without fixed benchmarks, ensuring theoretical guarantees of improvement, and preventing reward manipulation. The challenge is to design a system where evaluation signals can be learned, adapted, and co-evolved with agents, maintaining convergence guarantees while allowing the utility function to evolve over time.
Innovation
The key innovations of this work include: 1) Introducing controlled utility evolution via epoch-based evaluation, where each epoch employs a fixed, statistically validated evaluator, ensuring local convergence guarantees; 2) Developing a Bayesian evaluator replacement mechanism that uses Beta distribution quantiles to select superior evaluators with high confidence, enabling dynamic adaptation; 3) Architecting a multi-role, multi-task shared workspace that supports simultaneous co-evolution of task agents and evaluators, facilitating diverse and open-ended learning; 4) Incorporating adversarial objectives to regulate evaluator strictness, preventing reward hacking and bias. These innovations collectively enable the system to adapt its evaluation criteria over time, supporting continuous improvement in complex, real-world scenarios, and extending the classical Gödel Machine framework to non-stationary, multi-objective environments.
Methodology
- �� 构建多角色多智能体架构，每个节点由任务智能体和评估器组成，角色池共享资源；
- �� 将搜索划分为多个epoch，每个epoch内，评估器保持固定，确保性能保证；
- �� 在epoch边界，通过贝叶斯统计（Beta分布）对评估器进行优劣检验，满足信心水平后进行替换；
- �� 采用贝叶斯采样（Beta后验）选择最优评估器，确保替换的统计学可靠性；
- �� 引入对抗目标，调节评估器的严格程度，避免奖励作弊和偏差；
- �� 利用贝叶斯后验的α-分位数作为评估器替换的依据，确保替换具有统计学意义；
- �� 采用指数间隔的checkpoint机制，控制评估器替换和记录 erasure 的成本；
- �� 在每个epoch内，利用固定的评估器进行任务优化，保证自我改进的理论保证；
- �� 通过ground-truth anchor进行评估器替换，确保替换的有效性和稳定性。
Experiments
实验设计涵盖编码、科学论文写作与评审、数学证明三个领域。编码任务采用Polyglot测试集，比较RQGM与HGM（HyperAgents的HGM版本）在搜索代币数和测试通过率上的差异，验证共进化评估器的效率提升。论文写作与评审使用APReS数据集，评估论文接受率和评审准确率，验证合作演化的效果。数学证明任务以IMO-GradingBench为基准，比较评分器的真值准确率，验证系统在复杂推理中的表现。每个任务中，结合静态基准和动态评估，设置不同的epoch边界和贝叶斯替换阈值，进行消融研究。超参数包括贝叶斯后验α值、epoch长度、对抗目标权重等，确保系统在多场景中的鲁棒性和泛化能力。
Results
在Polyglot编码任务中，RQGM将测试通过率从69.9%提升至71.7%，节省搜索代币1.35-1.72倍。论文写作中，合作演化的写作代理将论文接受率从21.8%提升至40.5%，评审器的准确率提升了9%。数学证明中，合作评分器的真值准确率比静态基准高出9%，验证了系统在复杂推理中的优越性能。这些结果充分验证了评估器共进化的有效性，尤其在无直接基准或评估成本高昂的场景中表现出显著优势。通过贝叶斯替换和对抗训练，系统在多轮演化中逐步优化目标，展现出强大的适应能力和鲁棒性。
Applications
该系统在自动代码生成、科学论文自动撰写、数学证明、学术评审等场景中具有广泛应用潜力。尤其适用于缺乏明确基准或评估成本高昂的任务，通过学习的评估器实现高效评价和优化。未来可结合强化学习和元学习技术，推动自主科研、自动化设计、智能评审等领域的发展。系统还可用于多模态任务，结合图像、文本、推理等多种信息源，打造更全面的自主智能体。
Limitations & Outlook
目前系统在多轮演化中仍面临计算资源消耗大、评估器替换频繁带来的稳定性挑战。此外，系统在极端复杂任务中的泛化能力尚未充分验证，需优化算法效率和稳定性。贝叶斯替换机制的统计学依据在某些场景可能不够充分，需结合其他评估机制增强鲁棒性。长时间演化可能引入偏差积累，影响系统的持续性能。未来研究应关注多模态、多目标的动态调节机制和更高效的演化策略，以实现更大规模的自主智能系统。
Glossary
Self-improving agents (自我改进代理)
能够通过自身修改或优化算法实现性能提升的智能系统，核心在于自主学习和演化。
本文中指通过递归自我修改实现性能提升的AI系统。
Red Queen Hypothesis (红皇后假说)
生物进化中物种不断适应环境以保持竞争力的理论，比喻系统持续演化以适应变化。
用作系统不断共进化的比喻。
Gödel Machine (哥德尔机器)
一种理论上的自我改进系统，能证明自身改进的有效性，依赖形式逻辑证明。
基础算法框架之一。
贝叶斯后验 (Bayesian posterior)
在贝叶斯统计中，基于先验和新数据计算的后验概率，用于评估模型或评估器的优劣。
用于评估器替换的统计依据。
贝塔分布 (Beta distribution)
一种连续概率分布，常用于建模二项成功概率的后验分布，具有灵活的形状参数。
在贝叶斯评估器替换中使用。
多智能体系统 (Multi-agent system)
由多个自主智能体组成的系统，彼此协作或竞争以完成复杂任务。
本文中的架构基础。
对抗目标 (Adversarial objective)
设计目标使模型在训练中面对对手或挑战，以增强鲁棒性和公平性。
调节评估器的严格程度。
贝叶斯采样 (Bayesian sampling)
根据贝叶斯后验分布抽样，用于模型选择或参数优化。
实现评估器替换的关键技术。
ground-truth anchor (真实标杆)
固定的、可信的评价标准，用于评估模型或系统的性能。
用于评估器的替换依据。
贝叶斯下界 (Bayesian lower bound)
贝叶斯后验的α-分位数，用作模型或评估器的性能保证。
确保替换的统计学有效性。
多角色多任务架构
系统中不同角色（任务智能体、评估器）共享资源，共同演化以完成多样任务。
系统设计的基础。
贝叶斯后验的α-分位数
贝塔分布的α分位点，用作性能的保守估计。
评估器替换的依据。
对抗训练 (Adversarial training)
通过引入对手或挑战，增强模型鲁棒性的方法。
调节评估器严格程度。
贝叶斯后验更新
在新数据到达后，更新模型的后验分布，支持连续学习。
实现多轮演化。
指数间隔checkpoint
在演化过程中，按照指数增长的时间点进行状态保存和评估器替换，控制成本。
管理演化的效率。
Open Questions Unanswered questions from this research
- 1 目前系统在多轮演化中仍依赖于预定义的epoch边界，未来需研究更细粒度的动态调节机制，以实现连续性和稳定性更佳的优化。
- 2 在极端复杂任务或长时间演化中，系统的泛化能力和稳定性尚未充分验证，需结合更高效的算法和多模态信息进行优化。
- 3 贝叶斯替换机制的统计学依据在某些场景可能不足，未来应结合其他评估指标或机制，增强鲁棒性和适应性。
- 4 系统在多角色、多任务环境中的扩展性和可扩展性仍需验证，尤其在大规模、多样化任务中表现如何。
- 5 如何在保证理论收敛保证的同时，提升系统的实际运行效率和资源利用率，是未来的重要研究方向。
Applications
Immediate Applications
自动代码生成与优化
利用RQGM在无明确基准的代码任务中，通过学习评估器实现高效搜索，提升自动编程和代码修复能力，减少人工干预。
科学论文自动撰写与评审
通过共进化的写作和评审系统，自动生成高质量论文，自动评估和筛选，提升学术出版效率，降低评审偏差。
数学证明与推理自动化
在复杂数学证明中，利用学习的评分器引导推理过程，自动验证证明的正确性，推动自动化数学研究。
Long-term Vision
自主科学发现
系统通过不断演化评估标准，自动提出假设、设计实验，推动科学研究的自主化，缩短创新周期。
全面自主智能体
结合多模态、多任务的共演化机制，打造具备自主学习、适应和创新能力的通用智能系统，改变未来人机合作方式。
Abstract
Self-improving agents are state-of-the-art (SOTA) on agentic coding benchmarks and have recently been extended to general domains. However, their search methods generally assume a stationary evaluation criterion: a fixed verifier, benchmark, or labeled dataset that remains valid as the agent improves. This ignores a central feature of evolution: species adapt as their environments change with them. We aim to bring the same principle to recursive self-improvement, making evaluation part of the improvement loop and opening search to evolving evaluators, adversarial objectives, and dynamic utilities that may surpass static benchmarks. We introduce the Red Queen Godel Machine (RQGM), an evolutionary framework for recursive self-improvement under non-stationary utilities. The RQGM makes this possible through controlled utility evolution: search is organized into epochs with a fixed within-epoch evaluation criterion, while the utility can be updated at epoch boundaries, so self-improvement guarantees hold per epoch as the objective evolves across them. We begin by showing that even on verifiable coding tasks, the RQGM improves test pass rate over the prior SOTA by adding a complementary agent-as-a-judge code-review signal. This signal is cheaper and the RQGM uses 1.35x-1.72x fewer tokens. We then turn to scientific paper writing and reviewing, and Olympiad-level proof writing and grading, where the RQGM improves performance over prior self-improving agents: co-evolved writers reach 1.78x-1.86x higher acceptance rates under a diverse agent-as-a-judge panel, while co-evolved graders reach 9% higher ground-truth accuracy. In paper reviewing, the strongest baseline reviewer over-accepts AI-generated papers at up to 1.91x the human rate. The RQGM corrects this by introducing an adversarial objective that discovers reviewers equally stringent on AI and human work.
References (20)
Huxley-Gödel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine
Wenyi Wang, Piotr Piekos, Nanbo Li et al.
Hyperagents
Jenny Zhang, Bingchen Zhao, Wannan Yang et al.
LLM Evaluators Recognize and Favor Their Own Generations
Arjun Panickssery, Samuel R. Bowman, Shi Feng
Towards end-to-end automation of AI research
Chris Lu, Cong Lu, R. Lange et al.
Towards Robust Mathematical Reasoning
Thang Luong, Dawsen Hwang, Hoang Nguyen et al.
Mathematical discoveries from program search with large language models
B. Romera-Paredes, M. Barekatain, Alexander Novikov et al.
Algorithms for Infinitely Many-Armed Bandits
Yizao Wang, Jean-Yves Audibert, R. Munos
New Methods for Competitive Coevolution
Christopher D. Rosin, R. Belew
Open-Endedness is Essential for Artificial Superhuman Intelligence
Edward Hughes, Michael Dennis, Jack Parker-Holder et al.
AlphaEvolve: A coding agent for scientific and algorithmic discovery
Alexander Novikov, Ngân V˜u, Marvin Eisenberger et al.
Gödel Agent: A Self-Referential Agent Framework for Recursively Self-Improvement
Xunjian Yin, Xinyi Wang, Liangming Pan et al.
A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
David Silver, T. Hubert, Julian Schrittwieser et al.
A multi-agent system for automating scientific discovery.
A. Ghareeb, Benjamin Chang, L. Mitchener et al.
Go-Explore: a New Approach for Hard-Exploration Problems
Adrien Ecoffet, Joost Huizinga, J. Lehman et al.
A new evolutionary law
L. Valen
Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?
Chun Xia, Zhe Wang, Yan Yang et al.
Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
Haoyan Yang, Mario Xerri, Solha Park et al.
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng et al.
Multi-Agent Evolve: LLM Self-Improve through Co-evolution
Yixin Chen, Yiding Wang, Siqi Zhu et al.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Carlos E. Jimenez, John Yang, Alexander Wettig et al.