Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA
Summary: arXiv:2604.12526v1 Announce Type: cross
Abstract: Continual machine unlearning aims to remove the influence of data that should no longer be retained, while preserving the usefulness of the model on everything else. This setting becomes especially difficult when deletion requests arrive sequentially, because the model must repeatedly adapt without erasing previously retained knowledge. Low-Rank Adaptation (LoRA) offers an efficient way to implement such updates, but naively combining many sequential LoRA modules leads to parameter collision, causing strong interference between tasks.
In response to this challenge, we propose a static alternative based on Singular Value Decomposition (SVD)-guided orthogonal subspace projection. Our method constrains each new LoRA update during training so that it lies in the orthogonal complement of the subspaces used by earlier unlearning tasks. This preserves task isolation without requiring dynamic routing at deployment.
Introduction
As artificial intelligence systems evolve, the capability for continual learning and unlearning becomes essential. When data deletion requests arise, it is crucial for models to adapt without losing previously acquired knowledge. The traditional approach to integrating new updates into existing models poses risks of interference and degradation in performance.
Methodology
Our proposed method leverages SVD to guide the orthogonal projection of new LoRA updates. This involves:
- Identifying the existing subspaces associated with prior tasks.
- Ensuring that new updates are constrained to operate within the orthogonal complement of these subspaces, thereby preventing overlap and interference.
- Maintaining model performance while effectively executing unlearning tasks.
Experimentation
We conducted extensive experiments using two benchmark datasets: CIFAR-100 with a ResNet-20 architecture and the MNIST dataset. The experiments were designed to evaluate the model’s performance over a series of thirty sequential unlearning tasks.
Results
The results reveal significant findings:
- State-of-the-art static fusion techniques led to a reduction in retained accuracy from 60.39% to 12.70% after successive tasks.
- In contrast, our proposed constrained optimization method maintained a baseline performance of approximately 58.1%, showcasing its effectiveness in retaining model accuracy while achieving strong unlearning efficacy.
Conclusion
In conclusion, the use of SVD-based orthogonal subspace projection presents a promising solution to the challenges of continual machine unlearning. By ensuring task isolation through orthogonal constraints, our method offers a robust framework for updating machine learning models in response to sequential data deletion requests.
Further research and experimentation will be needed to explore the full potential of this approach across various architectures and datasets, paving the way for more resilient AI systems capable of adapting to evolving data privacy requirements.