Heavy-Tailed Class-Conditional Priors for Long-Tailed Generative Modeling
Summary: arXiv:2509.02154v2 Announce Type: replace-cross
Abstract: Variational Autoencoders (VAEs) with global priors trained under an imbalanced empirical class distribution can lead to underrepresentation of tail classes in the latent space. While $t^3$VAE improves robustness via heavy-tailed Student’s $t$-distribution priors, its single global prior still allocates mass proportionally to class frequency. We address this latent geometric bias by introducing C-$t^3$VAE, which assigns a per-class Student’s $t$ joint prior over latent and output variables. This design promotes uniform prior mass across class-conditioned components.
Introduction
The landscape of generative modeling has evolved significantly with the introduction of Variational Autoencoders (VAEs). However, the challenge remains when dealing with imbalanced datasets, particularly in long-tailed distributions. Traditional VAEs often fail to adequately represent tail classes, resulting in a loss of information and reduced generation quality.
Problem Statement
Despite advancements in VAE methodologies, such as the adoption of heavy-tailed distributions, the reliance on a single global prior continues to perpetuate biases. This bias manifests in the latent space, where underrepresented classes struggle to gain adequate representation. Consequently, solutions are needed to mitigate these issues and enhance generative capabilities across all classes.
Proposed Solution: C-$t^3$VAE
In response to the limitations observed in existing models, we propose the C-$t^3$VAE framework. Key features of this model include:
- Utilization of a per-class Student’s $t$ joint prior, enhancing representation across all classes.
- Promotion of uniform prior mass distribution, reducing latent geometric bias.
- Derivation of a closed-form objective grounded in the $\gamma$-power divergence, optimizing performance in imbalanced scenarios.
- Implementation of an equal-weight latent mixture to balance class generation.
Experimental Evaluation
To evaluate the effectiveness of the C-$t^3$VAE, we conducted experiments on several benchmark datasets, including SVHN-LT, CIFAR100-LT, and CelebA. Our findings reveal:
- C-$t^3$VAE consistently achieves lower Fréchet Inception Distance (FID) scores compared to $t^3$VAE and Gaussian-based VAE baselines, particularly in settings with severe class imbalance.
- In per-class F1 evaluations, C-$t^3$VAE outperforms the conditional Gaussian VAE, especially in highly imbalanced contexts.
Threshold for Model Competitiveness
Our analysis also identifies a critical threshold for imbalance denoted as $\rho < 5$, where Gaussian-based models maintain competitive performance. However, beyond this threshold ($\rho \geq 5$), the C-$t^3$VAE demonstrates significantly improved class-balanced generation and mode coverage, addressing the shortcomings of previous approaches effectively.
Conclusion
In conclusion, the introduction of C-$t^3$VAE represents a significant advancement in long-tailed generative modeling. By addressing latent geometric biases and ensuring equitable representation across class-conditioned components, our model sets a new standard for performance in imbalanced datasets. Future research may further explore the implications of heavy-tailed priors in other generative frameworks, potentially broadening the applicability of these insights.