EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training
Summary: arXiv:2604.19485v1 Announce Type: cross
Abstract
Reinforcement learning (RL) for LLM post-training faces a fundamental design choice: whether to use a learned critic as a baseline for policy optimization. Classical theory favors critic-based methods such as Proximal Policy Optimization (PPO) for variance reduction, yet critic-free alternatives like Generalized REINFORCE with Policy Optimization (GRPO) have gained widespread adoption due to their simplicity and competitive performance. This article explores the implications of using a learned critic in sparse-reward settings.
Introduction
In the evolving landscape of reinforcement learning, particularly in the context of large language model (LLM) post-training, the choice between employing a learned critic or opting for a critic-free approach has become crucial. The traditional stance supports critic-based methods for their ability to reduce variance in policy optimization. However, this article presents a compelling case for the potential drawbacks of using learned critics, particularly in scenarios with sparse rewards.
Key Insights
Our research reveals that in environments with sparse rewards, a learned critic can inadvertently introduce estimation noise that surpasses the signal it captures. This phenomenon results in an increase in advantage variance rather than the anticipated reduction. To address this, we approach baseline selection through the lens of Kalman filtering, unifying PPO and GRPO as two extremes of Kalman gain.
Explained Variance (EV)
We introduce the concept of explained variance (EV), which can be computed from a single training batch. Our findings indicate that positive EV signifies that the critic effectively reduces variance, while a zero or negative EV indicates an inflation of variance. This insight is pivotal for determining the appropriateness of utilizing a learned critic in various training scenarios.
Explained Variance Policy Optimization (EVPO)
Building on the insights regarding EV, we propose Explained Variance Policy Optimization (EVPO). This innovative approach dynamically monitors batch-level EV at each training step, enabling the model to switch adaptively between critic-based and batch-mean advantage estimation. This methodology ensures that variance does not exceed the lower variance option at each training iteration.
Experimental Results
Our empirical evaluations across four distinct tasks—ranging from classical control and agentic interaction to mathematical reasoning—demonstrate that EVPO consistently outperforms both PPO and GRPO. Notably, this holds true irrespective of which fixed baseline exhibits superior performance on a given task.
Conclusion
In conclusion, our analysis confirms that the adaptive gating mechanism in EVPO effectively tracks the maturation of the critic throughout the training process. Furthermore, the theoretically derived zero threshold for EV proves to be empirically optimal, establishing a new benchmark in the landscape of reinforcement learning techniques for LLM post-training.
Future Work
Future research will focus on expanding the scope of EVPO beyond the tested tasks and exploring its applicability in more complex environments, as well as investigating the underlying mechanisms that govern the relationship between critic maturation and performance optimization.