CoE: Collaborative Entropy for Uncertainty Quantification in Agentic Multi-LLM Systems
Summary: arXiv:2603.28360v1 Announce Type: new
Abstract
Uncertainty estimation in multi-LLM systems remains largely single-model-centric: existing methods quantify uncertainty within each model but do not adequately capture semantic disagreement across models. To address this gap, we propose Collaborative Entropy (CoE), a unified information-theoretic metric for semantic uncertainty in multi-LLM collaboration. CoE is defined on a shared semantic cluster space and combines two components: intra-model semantic entropy and inter-model divergence to the ensemble mean.
CoE is not a weighted ensemble predictor; it is a system-level uncertainty measure that characterizes collaborative confidence and disagreement. We analyze several core properties of CoE, including:
- Non-negativity
- Zero-value certainty under perfect semantic consensus
- The behavior of CoE when individual models collapse to delta distributions
These results clarify when reducing per-model uncertainty is sufficient and when residual inter-model disagreement remains. We also present a simple CoE-guided, training-free post-hoc coordination heuristic as a practical application of the metric.
Experimental Results
Experiments conducted on TriviaQA and SQuAD with LLaMA-3.1-8B-Instruct, Qwen-2.5-7B-Instruct, and Mistral-7B-Instruct show that CoE provides stronger uncertainty estimation than standard entropy- and divergence-based baselines. The advantages of CoE become more pronounced as additional heterogeneous models are introduced, illustrating its effectiveness in multi-LLM environments.
Conclusion
Overall, CoE offers a useful uncertainty-aware perspective on multi-LLM collaboration. By effectively quantifying both intra- and inter-model uncertainties, CoE not only enhances understanding of model behavior but also offers practical tools for improving collaborative AI systems.