Neural Network Pruning via QUBO Optimization
Summary: arXiv:2604.05856v1 Announce Type: cross
Abstract: Neural network pruning can be formulated as a combinatorial optimization problem, yet most existing approaches rely on greedy heuristics that ignore complex interactions between filters. Formal optimization methods such as Quadratic Unconstrained Binary Optimization (QUBO) provide a principled alternative but have so far underperformed due to oversimplified objective formulations based on metrics like the L1-norm.
In recent research, a novel approach has been proposed to enhance the effectiveness of neural network pruning through a unified Hybrid QUBO framework. This innovative framework aims to bridge the gap between heuristic importance estimation and global combinatorial optimization. Below are key aspects of this research:
- Integration of Sensitivity Metrics: The Hybrid QUBO framework incorporates gradient-aware sensitivity metrics into its formulation. Specifically, it utilizes first-order Taylor expansion and second-order Fisher information to create a more refined linear term.
- Data-Driven Activation Similarity: The quadratic term of the QUBO objective is enhanced by using data-driven activation similarity. This integration allows the model to effectively capture both individual filter relevance and the functional redundancy between filters.
- Dynamic Capacity-Driven Search: To ensure target sparsity, the framework introduces a dynamic capacity-driven search mechanism. This feature strictly enforces the desired sparsity level without distorting the optimization landscape, thus maintaining the integrity of the model.
- Two-Stage Pipeline: The proposed methodology includes a two-stage pipeline, which features a Tensor-Train (TT) Refinement stage. This stage acts as a gradient-free optimizer that fine-tunes the solution derived from the QUBO, directly aligning it with the true evaluation metric.
Experiments conducted on the SIDD image denoising dataset have shown that the Hybrid QUBO framework significantly outperforms traditional methods. In particular, it exceeds the effectiveness of greedy Taylor pruning and standard L1-based QUBO approaches. Furthermore, the TT Refinement stage provides consistent improvements, especially at larger combinatorial scales.
This research highlights the immense potential of hybrid combinatorial formulations in achieving robust, scalable, and interpretable neural network compression. The findings emphasize that by leveraging advanced optimization techniques, neural network pruning can be refined to enhance performance, reduce complexity, and improve overall efficiency in practical applications.
In conclusion, the proposed Hybrid QUBO framework represents a significant advancement in the field of neural network pruning, offering a sophisticated solution that harmonizes heuristic strategies with formal optimization techniques. This innovative approach not only addresses the shortcomings of previous methodologies but also sets a new standard for future research in neural network optimization.