CoDA: Towards Effective Cross-domain Knowledge Transfer via CoT-guided Domain Adaptation
Summary: arXiv:2604.19488v1 Announce Type: new
Abstract: Large language models (LLMs) have achieved substantial advances in logical reasoning, yet they continue to lag behind human-level performance. In-context learning provides a viable solution that boosts the model’s performance via prompting its input with expert-curated, in-domain exemplars. However, in many real-world, expertise-scarce domains, such as low-resource scientific disciplines, emerging biomedical subfields, or niche legal jurisdictions, such high-quality in-domain demonstrations are inherently limited or entirely unavailable, thereby constraining the general applicability of these approaches.
To mitigate this limitation, recent efforts have explored the retrieval of cross-domain samples as surrogate in-context demonstrations. Nevertheless, the resulting gains remain modest. This is largely attributable to the pronounced domain shift between source and target distributions, which impedes the model’s ability to effectively identify and exploit underlying shared structures or latent reasoning patterns. Consequently, when relying solely on raw textual prompting, LLMs struggle to abstract and transfer such cross-domain knowledge in a robust and systematic manner.
Introduction to CoDA
To address these issues, we propose CoDA, which employs a lightweight adapter to directly intervene in the intermediate hidden states. By combining feature-based distillation of CoT-enriched reference representations with Maximum Mean Discrepancy (MMD) for kernelized distribution matching, our method aligns the latent reasoning representation of the source and target domains.
Key Features of CoDA
- Lightweight Adapter: CoDA utilizes a streamlined adapter that facilitates intervention without significantly increasing computational overhead.
- Feature-based Distillation: This technique allows the model to incorporate enriched reference representations, improving its understanding of cross-domain knowledge.
- Maximum Mean Discrepancy (MMD): MMD is employed for efficient kernelized distribution matching, aiding in the alignment of different domain representations.
Experimental Validation
Extensive experimental results on multiple logical reasoning tasks across various model families validate the efficacy of CoDA. Our findings indicate that CoDA significantly outperforms the previous state-of-the-art baselines by a large margin.
Conclusion
In conclusion, CoDA represents a significant advancement in the field of domain adaptation for large language models. By addressing the inherent challenges of cross-domain knowledge transfer, CoDA enhances the models’ ability to perform logical reasoning tasks more effectively. This work opens up new avenues for research and application in low-resource fields, where access to high-quality in-domain data is often limited.
The implications of this research extend beyond academia, potentially impacting various industries that rely on effective knowledge transfer and reasoning capabilities. As LLMs continue to evolve, approaches like CoDA will be crucial in bridging the gap between human-like reasoning and machine learning performance.