Revisiting Catastrophic Forgetting in Continual Knowledge Graph Embedding
Knowledge Graph Embeddings (KGEs) are pivotal in enabling a variety of downstream tasks that leverage the intricate structure of Knowledge Graphs (KGs). As these graphs evolve, introducing new entities and facts, there arises a need for methods that can adaptively update embeddings over time. This has given rise to the concept of Continual Knowledge Graph Embedding (CKGE), which aims to address the challenges posed by the dynamic nature of KGs. However, a significant issue within CKGE is catastrophic forgetting, where a model’s performance on previously learned tasks diminishes as new information is incorporated.
Current approaches to CKGE primarily tackle catastrophic forgetting by limiting the extent to which existing embeddings can change when new entities are added. While this strategy appears effective at first glance, recent findings suggest that this perspective is too narrow and overlooks critical dynamics at play. Specifically, when new entities are introduced into a knowledge graph, their embeddings can inadvertently interfere with those of previously learned entities. This interference can lead to the model erroneously predicting new entities instead of returning correct answers for previously learned ones. We term this phenomenon “entity interference.”
Entity interference has not been adequately addressed in existing CKGE evaluation protocols, leading to potentially misleading assessments of catastrophic forgetting. Consequently, many CKGE methods may appear to perform better than they actually do, as the impact of entity interference is not factored into performance evaluations. To elucidate this issue, we propose a revised CKGE evaluation protocol that explicitly accounts for entity interference, providing a more accurate representation of a model’s performance and its susceptibility to catastrophic forgetting.
Key Findings and Implications
Our experiments across multiple benchmarks reveal the significant impact of entity interference on performance assessments. Ignoring this effect can lead to an overestimation of CKGE methods’ efficacy by as much as 25%, particularly in scenarios characterized by considerable entity growth. This finding underscores the importance of developing more nuanced evaluation metrics that reflect the true performance of CKGE methods in the face of new information.
- Understanding Catastrophic Forgetting: Catastrophic forgetting remains a critical challenge in evolving machine learning models, particularly in the context of knowledge graphs where new entities are frequently added.
- Entity Interference: The phenomenon of entity interference highlights the complexities of embedding updates and the necessity of considering interactions between embeddings.
- Revised Evaluation Protocol: The introduction of a corrected evaluation protocol offers a more rigorous framework for assessing CKGE methods, ensuring that performance metrics accurately reflect the effects of entity interference.
Moreover, our analysis delves into how various CKGE methods and KGE models respond to different sources of forgetting, leading to the introduction of a tailored catastrophic forgetting metric specifically designed for CKGE contexts. This metric seeks to provide a clearer understanding of the performance degradation associated with catastrophic forgetting and entity interference.
In conclusion, as the field of knowledge graph embedding continues to advance, it is crucial to revisit and refine evaluation methodologies. By addressing the overlooked issue of entity interference and introducing more comprehensive assessment protocols, we can better understand CKGE performance, ultimately leading to more robust and reliable applications of knowledge graphs in real-world scenarios.