Saliency-Guided Representation with Consistency Policy Learning for Visual Unsupervised Reinforcement Learning
Summary: arXiv:2604.05931v1 Announce Type: cross
Abstract: Zero-shot unsupervised reinforcement learning (URL) offers a promising direction for building generalist agents capable of generalizing to unseen tasks without additional supervision. Among existing approaches, successor representations (SR) have emerged as a prominent paradigm due to their effectiveness in structured, low-dimensional settings. However, SR methods struggle to scale to high-dimensional visual environments.
Through empirical analysis, we identify two key limitations of SR in visual URL:
- SR objectives often lead to suboptimal representations that attend to dynamics-irrelevant regions, resulting in inaccurate successor measures and degraded task generalization.
- These flawed representations hinder SR policies from modeling multi-modal skill-conditioned action distributions and ensuring skill controllability.
To address these limitations, we propose Saliency-Guided Representation with Consistency Policy Learning (SRCP), a novel framework that improves zero-shot generalization of SR methods in visual URL. The SRCP framework decouples representation learning from successor training by introducing a saliency-guided dynamics task to capture dynamics-relevant representations, thereby improving successor measure and task generalization.
Moreover, it integrates a fast-sampling consistency policy with URL-specific classifier-free guidance and tailored training objectives to enhance skill-conditioned policy modeling and controllability.
Extensive experiments on 16 tasks across 4 datasets from the ExORL benchmark demonstrate that SRCP achieves state-of-the-art zero-shot generalization in visual URL and is compatible with various SR methods. The results indicate that SRCP not only addresses the limitations of traditional SR methods but also opens new avenues for research in the field of visual unsupervised reinforcement learning.
Key Takeaways
- SRCP framework enhances zero-shot generalization in visual reinforcement learning tasks.
- Decoupling representation learning and successor training allows for better dynamics-relevant representation capture.
- Integration of a fast-sampling consistency policy improves the controllability of skill-conditioned actions.
- Experimental results indicate compatibility with existing SR methods while achieving superior performance.