Hierarchical Active Inference using Successor Representations
Active inference, a neurally-inspired model for inferring actions based on the free energy principle (FEP), has been proposed as a unifying framework for understanding perception, action, and learning in the brain. This innovative approach has previously been utilized to model ecologically important tasks such as navigation and planning. However, the challenge of scaling active inference to solve complex large-scale problems in real-world environments remains a significant hurdle for researchers.
Drawing inspiration from the existence of multi-scale hierarchical representations within the human brain, a new model for planning actions based on hierarchical active inference has been introduced. This approach integrates a hierarchical model of the environment with successor representations, enabling more efficient planning processes.
Key Contributions of the Study
- Learning Higher-Level Abstract States: The research demonstrates how lower-level successor representations can be leveraged to learn higher-level abstract states. This hierarchical learning can enhance the model’s understanding of complex environments.
- Bootstrapping Higher-Level Actions: The findings reveal how planning based on active inference at the lower level can be utilized to bootstrap and learn higher-level abstract actions. This allows for more sophisticated decision-making processes.
- Facilitating Efficient Planning: The study illustrates how the learned higher-level abstract states and actions can significantly facilitate efficient planning. This efficiency is crucial for tackling more complex tasks.
Experimental Validation
The research team presents results showcasing the performance of their hierarchical active inference approach on a variety of planning and reinforcement learning (RL) problems. These problems include:
- Four Rooms Task: A classic problem in the reinforcement learning domain, where agents must navigate through a series of interconnected rooms.
- Key-Based Navigation Task: This task requires agents to navigate towards goals while interacting with objects in the environment.
- Partially Observable Planning Problem: A scenario where agents must make decisions with incomplete information about the environment.
- Mountain Car Problem: A well-known benchmark in the RL community where the agent must drive a car up a hill, requiring strategic planning to succeed.
- PointMaze: A family of navigation tasks with continuous state and action spaces that further test the capabilities of the model.
Conclusion
The results from this study represent, to the best of our knowledge, the first application of learned hierarchical state and action abstractions to active inference in FEP-based theories of brain function. This pioneering work opens new avenues for enhancing planning and decision-making in complex environments, bridging the gap between theoretical neuroscience and practical applications in artificial intelligence.