Distilled Large Language Model-Driven Dynamic Sparse Expert Activation Mechanism
Summary: arXiv:2603.26735v1 Announce Type: cross
Abstract
High inter-class similarity, extreme scale variation, and limited computational budgets hinder reliable visual recognition across diverse real-world data. Existing vision-centric and cross-modal approaches often rely on rigid fusion mechanisms and heavy annotation pipelines, leading to sub-optimal generalization. We propose the Distilled Large Language Model (LLM)-Driven Sparse Mixture-of-Experts (DS-MoE) framework, which integrates text-guided dynamic routing and lightweight multi-scale comprehension.
Introduction
The demand for reliable visual recognition has surged in various industries, driven by the need to process and analyze complex visual data. However, traditional approaches face significant challenges, including:
- High inter-class similarity, making it difficult to distinguish between similar objects.
- Extreme scale variation, where visual objects can appear in vastly different sizes and resolutions.
- Limited computational budgets, which restrict the feasibility of deploying complex models in real-time applications.
Challenges in Existing Approaches
Current vision-centric and cross-modal techniques often depend on rigid fusion mechanisms, which can lead to inefficiencies. Heavy annotation pipelines are also a common bottleneck, resulting in:
- Sub-optimal generalization across diverse datasets.
- Increased time and resource expenditures during the training and deployment phases.
The DS-MoE Framework
The DS-MoE framework introduces an innovative solution that enhances visual recognition capabilities. Key features of the framework include:
- Text-Guided Dynamic Routing: This mechanism aligns textual semantics with defect-specific visual patterns, enabling a more nuanced understanding of the data.
- Sparse Mixture-of-Experts Architecture: Task-relevant experts are activated based on semantic relevance, effectively addressing inter-class ambiguity.
- Lightweight MobileSAM Encoder: This component allows for real-time inference while maintaining multi-scale defect details, making the framework suitable for practical applications.
Performance Evaluation
Extensive experiments conducted on various datasets, including PCB, aluminum foil, and mold defects, demonstrate the efficacy of the DS-MoE framework. The results indicate that:
- The DS-MoE framework outperforms existing pure vision models significantly.
- It surpasses popular models such as YOLOv8 and YOLOX with gains of +13.9, +1.4, and +2.0 percentage points in mean Average Precision (mAP) at 0.5:0.95 across BBMP, aluminum, and PCB datasets, respectively.
- Moreover, the framework shows improvements in both precision and recall metrics, underscoring its robustness.
Conclusion
The DS-MoE framework marks a significant advancement in visual recognition technology, addressing critical challenges faced by existing models. By leveraging language models for dynamic routing and employing a sparse architecture, the framework sets a new benchmark for performance in real-world applications.