Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets
Summary: arXiv:2405.17573v3 Announce Type: replace-cross
Abstract
This study delves into the mechanics of Leaky ResNets, which serve as a bridge between traditional ResNets and Fully-Connected neural networks, influenced by an ‘effective depth’ hyper-parameter, denoted as $\tilde{L}$. As we approach the infinite depth limit, we explore the concept of ‘representation geodesics’ $A_{p}$—these are continuous trajectories in representation space, akin to NeuralODEs, that transition from the input state at $p=0$ to the output state at $p=1$. The objective of these paths is to minimize the parameter norm of the network.
Key Findings
Our research introduces a Lagrangian and Hamiltonian framework that elucidates the dynamics of feature learning in Leaky ResNets. The framework emphasizes the significance of two pivotal terms:
- Kinetic Energy: This term promotes smaller derivatives of the layers, denoted as $\partial_{p}A_{p}$, thereby encouraging smoother transitions in the representation space.
- Potential Energy: This aspect favors low-dimensional representations, quantified by what we term the ‘Cost of Identity.’
The interplay between these two forces provides an insightful perspective on how feature learning occurs within ResNets. Specifically, we investigate the emergence of a bottleneck structure that has been documented in prior research. For larger values of $\tilde{L}$, we observe that the potential energy term prevails, resulting in a marked separation of timescales in the learning process.
Learning Dynamics
In this regime, the representation undergoes a rapid transition from high-dimensional inputs to a low-dimensional space, characterized by slower movements within the confines of this low-dimensional representation, before ultimately transitioning back to potentially high-dimensional outputs. This dynamic showcases the adaptive nature of the learning process, where the model can effectively manage complexity and dimensionality through the strategic balancing of kinetic and potential energies.
Training Methodology
Drawing inspiration from the identified bottleneck phenomenon, we propose a novel training approach that incorporates an adaptive layer step-size. This adjustment allows the model to effectively adapt to the observed separation of timescales, enhancing both training efficiency and representation learning. By fine-tuning the step sizes, the network can better navigate the complex landscape of feature representations, leading to improved performance across various tasks.
Conclusion
Our findings contribute to a deeper understanding of the mechanics underlying feature learning in Leaky ResNets. The introduction of Hamiltonian dynamics to model these processes not only enhances our theoretical insights but also opens avenues for practical advancements in neural network training strategies. As we continue to explore the complexities of neural architectures, this work emphasizes the importance of understanding the underlying principles that govern their behavior and performance.