CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval
Summary: arXiv:2604.15663v1 Announce Type: cross
Abstract
Code search, framed as information retrieval (IR), underpins modern software engineering and increasingly powers retrieval-augmented generation (RAG), improving code discovery, reuse, and the reliability of LLM-based coding. Yet existing code IR models remain largely text-centric and often overlook the visual and structural aspects inherent in programming artifacts such as web interfaces, data visualizations, SVGs, schematic diagrams, and UML.
Introduction to CodeMMR
To bridge this gap, we introduce MMCoIR, the first comprehensive benchmark for evaluating multimodal code IR across five visual domains, eight programming languages, and eleven libraries. This benchmark highlights the challenges of the task through extensive evaluation. Following this, we propose CodeMMR, a unified retrieval model that jointly embeds natural language, code, and images into a shared semantic space through instruction-based multimodal alignment.
Key Features of CodeMMR
CodeMMR exhibits several notable features:
- Joint embedding of natural language, code, and images.
- Instruction-based multimodal alignment for enhanced retrieval accuracy.
- Strong generalization across various modalities and programming languages.
Performance Evaluation
In our extensive evaluations, CodeMMR outperformed competitive baselines, such as UniIR, GME, and VLM2Vec, by an average of 10 points on nDCG@10. This improvement underscores the model’s effectiveness in retrieving relevant code snippets and associated visual resources, significantly benefiting software developers and programmers.
Integration with RAG
Moreover, integrating CodeMMR into RAG enhances code generation fidelity and visual grounding on unseen code generation tasks. This advancement underscores the potential of multimodal retrieval as a core enabler for next-generation intelligent programming systems, which can intuitively understand and generate code based on a blend of textual and visual information.
Conclusion
The introduction of CodeMMR marks a significant leap in the realm of code retrieval and generation. By addressing the limitations of traditional text-centric models, it opens new avenues for enhancing the software development process. As the demand for more intelligent and context-aware coding tools continues to rise, CodeMMR stands out as a pivotal innovation that leverages the power of multimodal information.
Datasets Availability
For researchers and developers interested in exploring or building upon this work, datasets are available at HuggingFace, facilitating further advancements in multimodal code retrieval and generation.