Curvature-Aware Optimization for High-Accuracy Physics-Informed Neural Networks
Summary: arXiv:2604.05230v1 Announce Type: cross
Abstract
Efficient and robust optimization is essential for neural networks, enabling scientific machine learning models to converge rapidly to very high accuracy — faithfully capturing complex physical behavior governed by differential equations. In this work, we present advanced optimization strategies to accelerate the convergence of physics-informed neural networks (PINNs) for challenging partial differential equations (PDEs) and ordinary differential equations (ODEs).
Introduction
The integration of physics-informed neural networks (PINNs) into scientific computing has emerged as a pivotal advancement in capturing the intricate behaviors of physical systems. However, the optimization of these networks poses a significant challenge, particularly when dealing with complex differential equations. The need for efficient optimization techniques is critical, as it directly influences the accuracy and reliability of the models being developed.
Advanced Optimization Strategies
In our research, we explore a variety of advanced optimization techniques to enhance the training process of PINNs. The following methods are discussed:
- Natural Gradient (NG) Optimizer: This method leverages the geometry of the parameter space, allowing for more effective updates that consider the curvature of the loss surface.
- Self-Scaling BFGS: This quasi-Newton method adapts the scaling of the variables dynamically, improving convergence rates in high-dimensional problems.
- Broyden Optimizers: These optimizers are designed to efficiently approximate the inverse Hessian and provide robust convergence properties.
Application to Differential Equations
We apply these optimization techniques to several benchmark problems, including:
- The Helmholtz equation, which describes wave propagation.
- Stokes flow, relevant in fluid dynamics.
- The inviscid Burgers equation, a fundamental model in gas dynamics.
- Euler equations for high-speed flows, essential in aerodynamics.
- Stiff ODEs arising in pharmacokinetics and pharmacodynamics.
New PINN-Based Methods
Beyond the optimization strategies, we also introduce innovative PINN-based methods aimed at solving the inviscid Burgers and Euler equations. These methods are rigorously tested against high-order numerical methods, ensuring a comprehensive evaluation of performance and accuracy.
Challenges and Solutions
One of the significant challenges addressed in our work is the scaling of quasi-Newton optimizers for batched training. By developing new strategies, we enable efficient and scalable solutions for large data-driven problems, facilitating broader applications in scientific computing.
Conclusion
Our findings highlight the importance of curvature-aware optimization techniques in enhancing the performance of physics-informed neural networks. The proposed methods not only accelerate convergence but also maintain high accuracy, paving the way for future advancements in scientific machine learning.