Mosaic: Multimodal Jailbreak against Closed-Source VLMs via Multi-View Ensemble Optimization
Summary: arXiv:2604.09253v1 Announce Type: cross
Abstract
Vision-Language Models (VLMs) are powerful but remain vulnerable to multimodal jailbreak attacks. Existing attacks mainly rely on either explicit visual prompt attacks or gradient-based adversarial optimization. While the former is easier to detect, the latter produces subtle perturbations that are less perceptible, but is usually optimized and evaluated under homogeneous open-source surrogate-target settings, leaving its effectiveness on commercial closed-source VLMs under heterogeneous settings unclear.
To examine this issue, we study different surrogate-target settings and observe a consistent gap between homogeneous and heterogeneous settings, a phenomenon we term surrogate dependency. Motivated by this finding, we propose Mosaic, a Multi-view ensemble optimization framework for multimodal jailbreak against closed-source VLMs, which alleviates surrogate dependency under heterogeneous surrogate-target settings by reducing over-reliance on any single surrogate model and visual view.
Core Components of Mosaic
Mosaic incorporates three core components:
- Text-Side Transformation Module: This module perturbs refusal-sensitive lexical patterns, enhancing the model’s ability to bypass restrictions imposed by VLMs.
- Multi-View Image Optimization Module: This component updates perturbations under diverse cropped views to avoid overfitting to a single visual perspective, thereby improving the robustness of the jailbreak.
- Surrogate Ensemble Guidance Module: This module aggregates optimization signals from multiple surrogate VLMs, effectively reducing surrogate-specific bias and increasing the overall effectiveness of the attack.
Experimental Results
Extensive experiments on safety benchmarks demonstrate that Mosaic achieves state-of-the-art Attack Success Rate and Average Toxicity against commercial closed-source VLMs. By leveraging multi-view ensemble optimization, Mosaic not only addresses the challenges posed by surrogate dependency but also enhances the attack’s effectiveness across various settings.
The implications of these findings are significant for the development of more robust and secure VLMs. As the landscape of AI continues to evolve, understanding and mitigating vulnerabilities in these models is crucial for ensuring their safe deployment in real-world applications.
Conclusion
Mosaic represents a significant advancement in the field of multimodal jailbreak attacks. By combining diverse optimization strategies and reducing reliance on single surrogate models, it paves the way for potential improvements in both the security of VLMs and the methodologies used to evaluate their vulnerabilities. Ongoing research in this area will be vital in shaping the future of AI safety and reliability.