Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?
Summary: arXiv:2604.10547v1 Announce Type: new
Introduction
The recent introduction of Agent^2 RL-Bench represents a significant advancement in the evaluation of agentic reinforcement learning (RL) post-training. This benchmark aims to determine whether large language model (LLM) agents can autonomously design, implement, and execute complete RL pipelines that enhance foundation models. The importance of such capabilities cannot be overstated, as RL post-training is increasingly seen as a crucial driver of model alignment and specialization in the rapidly evolving field of artificial intelligence.
Challenges with Existing Benchmarks
Current benchmarks largely remain static, primarily focusing on supervised fine-tuning. While these methods can yield impressive results, they fail to assess the dynamic and interactive nature of RL engineering. Agent^2 RL-Bench addresses these challenges head-on by introducing six tasks across three levels of complexity:
- Static rule-based training
- Closed-loop online RL
- Trajectory collection
Each of these levels adds structural requirements that prior levels do not impose, creating a more comprehensive testing environment for agentic RL post-training capabilities.
Features of Agent^2 RL-Bench
The benchmark is designed with several key features that enhance its utility for researchers:
- Isolated workspaces for testing
- A grading API for performance evaluation
- Runtime instrumentation that records every submission and code revision
- Automated post-hoc analysis that generates structured run reports
These features enable the first automated diagnostic of agent-driven post-training behavior, providing insights into the effectiveness and efficiency of various RL strategies.
Findings from the Benchmark
Initial findings from testing multiple agent stacks across five agent systems and six driver LLMs reveal intriguing results. For instance, on the ALFWorld task, an RL-only agent demonstrated an impressive improvement from a performance score of 5.97 to 93.28 through supervised fine-tuning (SFT) warm-up and generalized reinforcement policy optimization (GRPO) with online rollouts. However, results varied significantly across different tasks. For example, the DeepSearchQA task only saw marginal improvement, with an increase of just 2.75 within evaluation noise.
Impact of Driver Choice
Another notable observation is the substantial impact that driver choice has on interactive tasks. Within the same scaffold, switching drivers changed the interactive improvement from nearly zero to an impressive +78 percentage points, emphasizing the importance of selecting the right driver for optimal performance.
Conclusion
More broadly, Agent^2 RL-Bench reveals that supervised pipelines currently overshadow agent-driven post-training under fixed budgets. While online RL may ultimately serve as the best route for improvement in specific scenarios, such as ALFWorld, it is clear that further exploration and refinement of agentic RL post-training techniques are necessary.
For those interested in diving deeper, the code for Agent^2 RL-Bench is available at GitHub.