Leveraging LLM-GNN Integration for Open-World Question Answering over Knowledge Graphs
Summary: arXiv:2604.13979v1 Announce Type: cross
Abstract: Open-world Question Answering (OW-QA) over knowledge graphs (KGs) aims to answer questions over incomplete or evolving KGs. Traditional KGQA assumes a closed world where answers must exist in the KG, limiting real-world applicability. In contrast, open-world QA requires inferring missing knowledge based on graph structure and context. Large language models (LLMs) excel at language understanding but lack structured reasoning. Graph neural networks (GNNs) model graph topology but struggle with semantic interpretation. Existing systems integrate LLMs with GNNs or graph retrievers. Some support open-world QA but rely on structural embeddings without semantic grounding. Most assume observed paths or complete graphs, making them unreliable under missing links or multi-hop reasoning.
Introducing GLOW: A Hybrid System for Open-World KGQA
In response to the limitations of traditional systems, we present GLOW, a hybrid system that combines a pre-trained GNN and an LLM for open-world KGQA. The architecture of GLOW allows for enhanced reasoning capabilities that facilitate the extraction of meaningful insights from incomplete knowledge graphs.
How GLOW Works
The GLOW system operates through a two-step process:
- Graph Neural Network (GNN) Prediction: The GNN component predicts the top-k candidate answers by analyzing the graph structure. This step leverages the interconnected nature of knowledge graphs to identify potential answers based on available data.
- Structured Prompting for LLM: The predicted candidates, along with relevant KG facts, are serialized into a structured prompt (e.g., triples and candidates) that guides the LLM’s reasoning process. This structured approach allows the LLM to engage in joint reasoning over both symbolic and semantic signals.
Advantages of GLOW
GLOW offers several advantages over previous systems:
- Joint Reasoning: By combining the strengths of both LLMs and GNNs, GLOW enables more robust reasoning capabilities that do not solely rely on retrieval or fine-tuning methods.
- Generalization Capability: To evaluate the generalization of the system, we introduce GLOW-BENCH, a 1,000-question benchmark over incomplete KGs across diverse domains, allowing for comprehensive performance assessment.
- Improved Performance: GLOW has demonstrated superior performance compared to existing LLM-GNN systems on standard benchmarks, achieving an outstanding 53.3% performance rate and an average improvement of 38% on GLOW-BENCH.
Conclusion
GLOW represents a significant advancement in the field of open-world question answering over knowledge graphs. With its innovative integration of LLMs and GNNs, GLOW not only enhances the ability to answer questions based on incomplete data but also sets a new standard for evaluating such systems. The code and data for GLOW are available on GitHub, facilitating further research and development in this promising area.