Towards Scalable Lifelong Knowledge Editing with Selective Knowledge Suppression
Summary: arXiv:2604.19089v1 Announce Type: new
Abstract
Large language models (LLMs) require frequent knowledge updates to reflect changing facts and mitigate hallucinations. To meet this demand, lifelong knowledge editing has emerged as a continual approach to modify specific pieces of knowledge without retraining the entire model. Existing parameter editing methods struggle with stability during sequential edits due to catastrophic forgetting. While retrieval-based approaches are proposed to alleviate this issue, their applicability remains limited across various datasets because of high training costs.
To address these limitations and enhance scalability in lifelong settings, we propose LightEdit. Our framework first selects relevant knowledge from retrieved information to modify the query effectively. It then incorporates a decoding strategy to suppress the model’s original knowledge probabilities, thereby enabling efficient edits based on the selected information.
Key Features of LightEdit
- Selective Knowledge Modification: LightEdit focuses on selecting relevant knowledge from retrieved information, allowing for precise updates without overhauling the entire model.
- Decoding Strategy: The innovative decoding mechanism suppresses original knowledge probabilities, ensuring that edits reflect the newly integrated information accurately.
- Cost-Effective Scalability: By minimizing training costs, LightEdit can be easily adapted to various datasets, promoting broader applicability in real-world scenarios.
Performance Evaluation
Extensive experiments conducted on multiple benchmarks, including ZSRE, Counterfact, and RIPE, demonstrate that LightEdit significantly outperforms existing lifelong knowledge editing methods. The results indicate that the framework not only maintains stability during sequential edits but also achieves effective knowledge updates with reduced resource consumption.
Conclusion
LightEdit represents a significant advancement in the field of lifelong knowledge editing for large language models. By implementing selective knowledge suppression, it addresses the challenges of catastrophic forgetting and high training costs, paving the way for more agile and efficient knowledge management in AI systems. As the demand for up-to-date information grows, solutions like LightEdit will be crucial for the ongoing development and deployment of LLMs in diverse applications.