Towards Effective In-context Cross-domain Knowledge Transfer via Domain-invariant-neurons-based Retrieval
Summary: arXiv:2604.05383v1 Announce Type: new
Abstract
Large language models (LLMs) have made notable progress in logical reasoning, yet still fall short of human-level performance. Current boosting strategies rely on expert-crafted in-domain demonstrations, limiting their applicability in expertise-scarce domains, such as specialized mathematical reasoning, formal logic, or legal analysis.
In this work, we demonstrate the feasibility of leveraging cross-domain demonstrating examples to boost the LLMs’ reasoning performance. Despite substantial domain differences, many reusable implicit logical structures are shared across domains. In order to effectively retrieve cross-domain examples for unseen domains under investigation, we further propose an effective retrieval method, called domain-invariant neurons-based retrieval (DIN-Retrieval).
Introduction
As advancements in artificial intelligence continue to evolve, large language models (LLMs) are increasingly recognized for their capabilities in various reasoning tasks. However, a significant gap remains between AI reasoning abilities and human-level performance, particularly in specialized domains. The reliance on in-domain demonstrations often hampers the flexibility of LLMs in tackling complex problems encountered in areas such as mathematics, logic, and law.
Overview of DIN-Retrieval
The proposed DIN-Retrieval method aims to bridge this gap by employing a novel retrieval approach to enhance the reasoning capabilities of LLMs across diverse domains. The methodology consists of two primary stages:
- Universal Representation Summary: DIN-Retrieval first generates a hidden representation that is applicable across different domains, highlighting the common logical structures shared among them.
- Inference and Retrieval: During the inference stage, the DIN vector is utilized to identify and retrieve structurally compatible cross-domain demonstrations, thereby facilitating in-context learning for the LLMs.
Experimental Results
The effectiveness of the DIN-Retrieval method has been demonstrated through a series of experiments focused on transferring mathematical and logical reasoning capabilities. The results indicate a significant performance improvement:
- Our method achieved an average improvement of 1.8 over existing state-of-the-art techniques.
- The experiments were conducted across multiple settings to ensure robustness and applicability of the findings.
Conclusion
In conclusion, the DIN-Retrieval approach offers a promising avenue for enhancing the reasoning performance of LLMs in expertise-scarce domains by leveraging cross-domain knowledge. As the field of AI continues to grow, strategies like DIN-Retrieval will be vital in bridging the performance gap between AI systems and human reasoning capabilities. For those interested in exploring this methodology further, the implementation is available at GitHub Repository.