Extreme Value Monte Carlo Tree Search for Classical Planning
Summary: arXiv:2405.18248v3 Announce Type: replace
Abstract: Despite being successful in board games and reinforcement learning (RL), Monte Carlo Tree Search (MCTS) combined with Multi Armed Bandits (MABs) has seen limited success in domain-independent classical planning until recently. Previous work (Wissow and Asai 2024) showed that UCB1, designed for bounded rewards, does not perform well as applied to cost-to-go estimates in classical planning, which are unbounded in ℝ, and showed improved performance using a Gaussian reward MAB instead. This paper further sharpens our understanding of ideal bandits for planning tasks.
Introduction
The integration of Monte Carlo Tree Search (MCTS) with Multi Armed Bandits (MABs) has revolutionized approaches in various fields, particularly board games and reinforcement learning. However, its application to domain-independent classical planning has been limited, raising questions about the efficacy of existing methodologies.
Challenges in Current Approaches
Recent research indicates two significant issues in the current application of MABs to classical planning:
- Under-specification of Gaussian MABs: Gaussian MABs are noted to under-specify the support of cost-to-go estimates, which range from $(-\infty,\infty)$. This broad support can lead to inefficiencies in planning tasks.
- Lack of Theoretical Justification: The Full Bellman backup method, as proposed by Schulte and Keller in 2014, lacks a solid theoretical foundation, raising concerns about its reliability in practical applications.
Proposed Solutions
To address these challenges, the authors of the paper employ Peaks-Over-Threshold Extreme Value Theory, offering a dual resolution to both issues. This theoretical framework allows for a more refined estimation of cost-to-go values while also providing a robust basis for the bandit algorithm.
Introduction of UCB1-Uniform
The paper introduces a novel bandit algorithm, termed UCB1-Uniform. This approach not only enhances the performance of classical planning tasks but also stands on a solid theoretical footing:
- Regret Bound: The authors formally prove a regret bound for UCB1-Uniform, establishing its effectiveness in minimizing the potential loss over time.
- Empirical Demonstration: The performance of UCB1-Uniform is empirically demonstrated through various classical planning scenarios, showcasing significant improvements over previous methods.
Conclusion
This research marks a significant step forward in the application of MCTS and MABs in classical planning. By refining the theoretical underpinnings and introducing UCB1-Uniform, the authors pave the way for more efficient and effective planning algorithms in the future.
As the field of artificial intelligence continues to evolve, findings such as these contribute to a deeper understanding and more robust methodologies, ultimately fostering advancements in both theory and application.