Bi-Level Optimization for Single Domain Generalization
Summary: arXiv:2604.06349v1 Announce Type: cross
Abstract: Generalizing from a single labeled source domain to unseen target domains, without access to any target data during training, remains a fundamental challenge in robust machine learning. We address this underexplored setting, known as Single Domain Generalization (SDG), by proposing BiSDG, a bi-level optimization framework that explicitly decouples task learning from domain modeling.
BiSDG simulates distribution shifts through surrogate domains constructed via label-preserving transformations of the source data. To capture domain-specific context, we propose a domain prompt encoder that generates lightweight modulation signals to produce augmenting features via feature-wise linear modulation. The learning process is formulated as a bi-level optimization problem: the inner objective optimizes task performance under fixed prompts, while the outer objective maximizes generalization across the surrogate domains by updating the domain prompt encoder.
We further develop a practical gradient approximation scheme that enables efficient bi-level training without second-order derivatives. Extensive experiments on various SDG benchmarks demonstrate that BiSDG consistently outperforms prior methods, setting new state-of-the-art performance in the SDG setting.
Key Features of BiSDG
- Decoupled Learning: BiSDG separates task learning from domain modeling, which enhances the adaptability of the model across different domains.
- Surrogate Domains: The framework utilizes label-preserving transformations of the source data to create surrogate domains, allowing for the simulation of distribution shifts.
- Domain Prompt Encoder: This novel component generates modulation signals that provide domain-specific features, improving the model’s performance.
- Bi-Level Optimization: The inner and outer objectives work together to optimize task performance and generalization, respectively, providing a robust training mechanism.
- Efficiency: A practical gradient approximation scheme is introduced, making the training process efficient without the need for second-order derivatives.
Impact on Machine Learning
The introduction of BiSDG marks a significant advancement in the realm of single domain generalization. By effectively addressing the challenge of generalizing from a single source domain, BiSDG sets a new benchmark for future research in this area. The techniques employed not only enhance performance but also pave the way for more robust machine learning models capable of adapting to unseen target domains.
Conclusion
In conclusion, the BiSDG framework represents a forward leap in the capabilities of single domain generalization methodologies. With its innovative approach to bi-level optimization and domain modeling, it holds promise for a wide range of applications in machine learning, ultimately contributing to the development of more resilient and adaptable AI systems.