Intentional Updates for Streaming Reinforcement Learning
Summary: arXiv:2604.19033v1 Announce Type: cross
Abstract
In gradient-based learning, a step size chosen in parameter units does not produce a predictable per-step change in function output. This often leads to instability in the streaming setting (i.e., batch size=1), where stochasticity is not averaged out and update magnitudes can momentarily become arbitrarily big or small. Instead, we propose intentional updates: first specify the intended outcome of an update and then solve for the step size that approximately achieves it.
Introduction
This strategy has precedent in online supervised linear regression via Normalized Least Mean Squares algorithm, which selects a step size to yield a specified change in the function output proportional to the current error. We extend this principle to streaming deep reinforcement learning by defining appropriate intended outcomes:
- Intentional TD: Aims for a fixed fractional reduction of the TD error.
- Intentional Policy Gradient: Aims for a bounded per-step change in the policy, limiting local KL divergence.
Methodology
Our proposed methods utilize practical algorithms that combine eligibility traces and diagonal scaling. By focusing on defined outcomes, these algorithms aim to stabilize the learning process in environments where traditional approaches may falter.
Results
Empirical results indicate that these methods yield state-of-the-art streaming performance. In numerous scenarios, the performance of intentional updates frequently matches or even surpasses that of batch and replay-buffer approaches, demonstrating their effectiveness in practical applications.
Conclusion
Intentional updates represent a significant advancement in the field of streaming reinforcement learning. By addressing the inherent instability caused by stochastic updates, this approach not only enhances performance but also provides a clearer framework for future research. The combination of intended outcomes with established reinforcement learning techniques opens new avenues for developing more robust learning systems.
Future Work
Future research will focus on refining these algorithms further and exploring their applicability across a broader range of environments and tasks. Additionally, we aim to investigate the integration of intentional updates with other reinforcement learning paradigms to enhance their performance and stability.