Does Unification Come at a Cost? Uni-SafeBench: A Safety Benchmark for Unified Multimodal Large Models
Summary: arXiv:2604.00547v1 Announce Type: new
Introduction
Unified Multimodal Large Models (UMLMs) have emerged as a transformative force in the artificial intelligence landscape, allowing for the integration of understanding and generation capabilities within a single model architecture. These models leverage the deep fusion of multimodal features to enhance performance across diverse tasks. However, the unification of these capabilities is not without its drawbacks, particularly concerning safety. As UMLMs become increasingly prevalent, it is crucial to evaluate the potential safety risks associated with their use.
The Challenge of Safety in UMLMs
Current safety benchmarks primarily target isolated tasks that either focus on understanding or generation. This approach overlooks the unique safety challenges posed by the holistic nature of UMLMs. As these models are deployed in real-world scenarios, the ability to handle multiple tasks simultaneously raises concerns about their overall safety performance. In response to these challenges, researchers have developed a new benchmark, Uni-SafeBench, aimed at systematically assessing the safety of UMLMs.
Introducing Uni-SafeBench
Uni-SafeBench presents a comprehensive safety benchmark designed to evaluate UMLMs through a taxonomy of six major safety categories across seven different task types. This framework allows for a more nuanced understanding of the potential risks associated with these models. The benchmark encompasses a range of tasks, including:
- Text understanding
- Text generation
- Image understanding
- Image generation
- Audio processing
- Video analysis
- Multimodal interaction
The Role of Uni-Judger
To ensure rigorous evaluation, the researchers introduced Uni-Judger, a framework designed to decouple contextual safety from intrinsic safety. This distinction is vital as it allows for a more accurate assessment of how UMLMs perform in real-world conditions. The findings from comprehensive evaluations across Uni-SafeBench indicate a troubling trend: while the unification of capabilities enhances performance, it significantly degrades the inherent safety of the underlying large language models (LLMs).
Key Findings
One of the most concerning revelations from the research is that open-source UMLMs demonstrate considerably lower safety performance compared to multimodal large models that are specialized for either generation or understanding tasks. This discrepancy highlights the risks associated with relying on unified models for diverse applications, and it calls for a reevaluation of how these models are developed and deployed.
Conclusion
As the field of artificial intelligence evolves, the introduction of Uni-SafeBench serves as a critical step toward fostering safer development practices in the realm of Unified Multimodal Large Models. By open-sourcing all resources related to this benchmark, the researchers aim to systematically expose the safety risks inherent in UMLMs and promote the pursuit of safer artificial general intelligence (AGI) solutions.