From $P(y|x)$ to $P(y)$: Investigating Reinforcement Learning in Pre-train Space
Summary: arXiv:2604.14142v1 Announce Type: cross
Abstract: While reinforcement learning with verifiable rewards (RLVR) significantly enhances LLM reasoning by optimizing the conditional distribution P(y|x), its potential is fundamentally bounded by the base model’s existing output distribution. Optimizing the marginal distribution P(y) in the Pre-train Space addresses this bottleneck by encoding reasoning ability and preserving broad exploration capacity. Yet, conventional pre-training relies on static corpora for passive learning, leading to a distribution shift that hinders targeted reasoning enhancement. In this paper, we introduce PreRL (Pre-train Space RL), which applies reward-driven online updates directly to P(y).
We theoretically and empirically validate the strong gradient alignment between log P(y) and log P(y|x), establishing PreRL as a viable surrogate for standard RL. Furthermore, we uncover a critical mechanism: Negative Sample Reinforcement (NSR) within PreRL serves as an exceptionally effective driver for reasoning. NSR-PreRL rapidly prunes incorrect reasoning spaces while stimulating endogenous reflective behaviors, increasing transition and reflection thoughts by 14.89x and 6.54x, respectively.
Key Innovations and Findings
Leveraging these insights, we propose Dual Space RL (DSRL), a Policy Reincarnation strategy that initializes models with NSR-PreRL to expand the reasoning horizon before transitioning to standard RL for fine-grained optimization. Extensive experiments demonstrate that DSRL consistently outperforms strong baselines, proving that pre-train space pruning effectively steers the policy toward a refined correct reasoning subspace.
Understanding the Methodology
- Reinforcement Learning with Verifiable Rewards (RLVR): This method enhances reasoning capabilities in large language models (LLMs) by optimizing conditional distributions.
- PreRL Framework: Introduces reward-driven online updates to the marginal distribution P(y), aimed at overcoming limitations of static pre-training.
- Negative Sample Reinforcement (NSR): A mechanism within PreRL that accelerates the pruning of errors in reasoning, allowing models to focus on correct paths.
- Dual Space RL (DSRL): A novel approach that utilizes NSR-PreRL for initial reasoning capacity expansion before transitioning to standard RL for refinement.
Implications for Future Research
The findings from this paper open new avenues for improving LLMs through enhanced reasoning capabilities. By shifting focus from traditional learning methods to dynamic, reward-driven approaches, researchers can better address existing limitations in model performance. The integration of mechanisms like NSR not only aids in refining reasoning but also introduces a framework for developing more robust AI systems capable of complex decision-making.
As AI continues to evolve, the insights gained from this research will be crucial in paving the way for future advancements in reinforcement learning, potentially reshaping how models learn and reason in complex environments.