On Divergence Measures for Training GFlowNets
Summary: arXiv:2410.09355v2 Announce Type: cross
Abstract
Generative Flow Networks (GFlowNets) are amortized inference models designed to sample from unnormalized distributions over composable objects, with applications in generative modeling for tasks in fields such as causal discovery, NLP, and drug discovery. Traditionally, the training procedure for GFlowNets seeks to minimize the expected log-squared difference between a proposal (forward policy) and a target (backward policy) distribution, which enforces certain flow-matching conditions.
While this training procedure is closely related to variational inference (VI), directly attempting standard Kullback-Leibler (KL) divergence minimization can lead to proven biased and potentially high-variance estimators. Therefore, we first review four divergence measures, namely, Renyi-$\alpha$’s, Tsallis-$\alpha$’s, reverse and forward KL’s, and design statistically efficient estimators for their stochastic gradients in the context of training GFlowNets.
Then, we verify that properly minimizing these divergences yields a provably correct and empirically effective training scheme, often leading to significantly faster convergence than previously proposed optimization. To achieve this, we design control variates based on the REINFORCE leave-one-out and score-matching estimators to reduce the variance of the learning objectives’ gradients. Our work contributes by narrowing the gap between GFlowNets training and generalized variational approximations, paving the way for algorithmic ideas informed by the divergence minimization viewpoint.
Key Points
- Generative Flow Networks (GFlowNets): Innovative models for sampling from complex distributions.
- Training Procedure: Focuses on minimizing the expected log-squared difference for better flow matching.
- Divergence Measures: Analysis of Renyi-$\alpha$’s, Tsallis-$\alpha$’s, and KL divergences to improve training.
- Variance Reduction: Implementation of control variates to enhance the efficiency of learning objectives.
- Empirical Results: Demonstration of faster convergence and improved training outcomes.
Conclusion
The study of divergence measures for training GFlowNets presents significant advancements in generative modeling. By addressing the limitations of traditional KL divergence minimization, this approach introduces a more robust framework for sampling from unnormalized distributions. The insights gained from the empirical results suggest that these new training methods can not only enhance the speed of convergence but also the overall effectiveness of GFlowNets in various applications.
As the field of generative modeling continues to evolve, the contributions of this research pave the way for future innovations, encouraging further exploration into divergence minimization techniques and their potential applications across diverse domains.