Frugal Knowledge Graph Construction with Local LLMs: A Zero-Shot Pipeline, Self-Consistency and Wisdom of Artificial Crowds
In a groundbreaking study published on arXiv (arXiv:2604.11104v1), researchers have unveiled a novel approach to knowledge graph construction that leverages local large language models (LLMs) in a zero-shot pipeline. This innovative method is executed entirely on consumer-grade hardware, making it accessible and practical for a wider audience.
Key Findings
The paper outlines an empirical study that integrates a multi-model zero-shot pipeline for constructing and exploiting knowledge graphs. Here are some of the key findings of the research:
- The study showcases a reproducible evaluation framework that incorporates two external benchmarks: DocRED and HotpotQA, along with synthetic data styled after WebQuestionsSP and the RAGAS evaluation framework.
- On a dataset of 500 document-level relations, the system achieved a promising F1 score of 0.70 ± 0.041 in zero-shot scenarios, in contrast to a score of 0.80 achieved by the supervised DREEAM model.
- For text-to-query tasks, the model demonstrated an accuracy rate of 0.80 ± 0.06 over 200 samples, illustrating its effectiveness in understanding and responding to user queries.
- In the realm of multi-hop reasoning, the model achieved an Exact Match (EM) score of 0.46 ± 0.04 on 500 HotpotQA questions, with a RAGAS faithfulness score of 0.96 ± 0.04 on a subset of 50 samples.
Diversity Mechanisms and Self-Consistency
The researchers further explored the diversity mechanisms that are essential for tackling challenging multi-hop reasoning tasks. Their findings revealed that:
- Among 181 questions deemed unsolvable at zero temperature, implementing self-consistency (k=5, T=0.7) yielded an improvement of up to 23% EM using a single Mixture-of-Experts (MoE) model.
- In contrast, a cross-model oracle approach, utilizing three different architectures across five samples, achieved a significantly higher EM of 46.4%.
- The study also uncovered an intriguing agreement paradox. It suggested that high consensus among model samples could indicate collective hallucination instead of a reliable answer, reflecting concepts discussed in previous research by Moussaïd et al. regarding the wisdom of crowds.
- When extending to the full pipeline of 500 questions, self-consistency (k=3) raised the EM from 0.46 to 0.48 ± 0.04, demonstrating the impact of this approach on overall performance.
Efficiency and Environmental Impact
Further analysis showed that a confidence-routing cascade mechanism (Phi-4 → GPT-OSS, k=5) achieved an EM of 0.55 ± 0.04, marking the highest result obtained in the study, with 45.4% of the questions being rerouted. Notably, the entire system operates within approximately five hours on a single RTX 3090 GPU, consuming no training resources and resulting in a carbon footprint estimated at just 0.09 kg CO2 equivalent.
Conclusion
This research represents a significant leap forward in knowledge graph construction methodologies, highlighting the potential of local LLMs in delivering substantial results without the need for extensive training or costly infrastructure. The findings pave the way for future investigations into the scalability and efficacy of AI-driven knowledge management systems.