COvolve: Adversarial Co-Evolution of Large-Language-Model-Generated Policies and Environments via Two-Player Zero-Sum Game
Summary: arXiv:2603.28386v1 Announce Type: new
Abstract: A central challenge in building continually improving agents is that training environments are typically static or manually constructed. This restricts continual learning and generalization beyond the training distribution. We address this with COvolve, a co-evolutionary framework that leverages large language models (LLMs) to generate both environments and agent policies, expressed as executable Python code. We model the interaction between environment and policy designers as a two-player zero-sum game, ensuring adversarial co-evolution in which environments expose policy weaknesses and policies adapt in response. This process induces an automated curriculum in which environments and policies co-evolve toward increasing complexity. To guarantee robustness and prevent forgetting as the curriculum progresses, we compute the mixed-strategy Nash equilibrium (MSNE) of the zero-sum game, thereby yielding a meta-policy. This MSNE meta-policy ensures that the agent does not forget to solve previously seen environments while learning to solve previously unseen ones. Experiments in urban driving, symbolic maze-solving, and geometric navigation showcase that COvolve produces progressively more complex environments. Our results demonstrate the potential of LLM-driven co-evolution to achieve open-ended learning without predefined task distributions or manual intervention.
Introduction to COvolve
The COvolve framework presents a significant advancement in the realm of artificial intelligence, particularly in the development of continually improving agents. Traditional training paradigms often face limitations due to static environments, which can hinder the agent’s ability to adapt and generalize beyond its initial training. COvolve addresses this challenge by utilizing the capabilities of large language models to dynamically generate both the environments and the policies that the agents will utilize.
Methodology
COvolve operates on a two-player zero-sum game model, where the environment designer and the policy designer engage in an adversarial relationship. This interaction helps in the following ways:
- Environment Generation: LLMs create diverse and challenging environments that expose the weaknesses of the policies.
- Policy Adaptation: In response, the policies evolve to better handle the challenges posed by these environments.
- Automated Curriculum: The co-evolution process leads to an automated learning curriculum, gradually increasing in complexity.
Ensuring Robustness
To maintain robustness and prevent the phenomenon of “forgetting” as agents encounter new challenges, COvolve computes the mixed-strategy Nash equilibrium (MSNE) of the zero-sum game. This equilibrium serves as a meta-policy, ensuring that agents retain the ability to solve previously encountered environments while also adapting to new ones.
Experimental Validation
Extensive experiments conducted in various domains, including urban driving, symbolic maze-solving, and geometric navigation, validate the effectiveness of COvolve. The results indicate that the environments generated through this framework not only become progressively more complex but also foster agents that are capable of continuous learning and adaptation.
Conclusion
The COvolve framework signifies a promising direction for future research in AI, emphasizing the potential of LLMs in creating adaptive learning environments. This approach holds the key to achieving open-ended learning without the need for fixed task distributions or manual intervention, paving the way for more intelligent and flexible AI systems.