SPARE: Self-distillation for PARameter-Efficient Removal
Summary: arXiv:2602.07058v2
Announce Type: replace-cross
Abstract
Machine Unlearning aims to remove the influence of specific data or concepts from trained models while preserving overall performance—a capability increasingly required by data protection regulations and responsible AI practices. Despite recent progress, unlearning in text-to-image diffusion models remains challenging due to high computational costs and the difficulty of balancing effective forgetting with retention of unrelated concepts.
Introducing SPARE
We introduce Self-distillation for PARameter Efficient Removal (SPARE), a two-stage unlearning method for image generation that combines parameter localization with self-distillation. This innovative approach addresses the growing need for efficient unlearning mechanisms while ensuring the integrity of non-targeted concepts within models.
How SPARE Works
SPARE operates through the following two key stages:
- Parameter Localization: SPARE first identifies the parameters that are most responsible for generating the unwanted concepts. This is achieved using gradient-based saliency techniques. The updates to these parameters are then constrained through sparse low-rank adapters, ensuring that only lightweight, localized modifications are made.
- Self-Distillation Objective: In the second stage, SPARE utilizes a self-distillation objective that effectively overwrites the unwanted concept with a user-defined surrogate concept, while maintaining the model’s behavior concerning other unrelated concepts.
Innovative Timestep Sampling
In addition to the two-stage process, SPARE introduces a timestep sampling scheme specifically designed for diffusion models. This scheme targets only the crucial timesteps associated with a given concept, leading to more efficient unlearning processes. By focusing on essential timesteps, SPARE minimizes unnecessary computational overhead, enhancing performance.
Performance and Results
SPARE has demonstrated its efficacy by surpassing the current state-of-the-art on the UnlearnCanvas benchmark. Through extensive ablation studies conducted on various datasets, we have shown that SPARE provides fine-grained control over the forgetting-retention trade-off. These results are particularly promising as they indicate that SPARE achieves strong concept erasure while maintaining high retainability across diverse domains.
Conclusion
In conclusion, SPARE offers a robust solution for selective unlearning in diffusion-based image generation models. By effectively balancing the need to forget unwanted concepts while retaining the integrity of other data, SPARE addresses a significant challenge in the field of responsible AI. As data protection regulations continue to evolve, methods like SPARE will play a crucial role in ensuring compliance and ethical AI practices.