RPRA: Predicting an LLM-Judge for Efficient but Performant Inference
Summary: arXiv:2604.12634v1 Announce Type: new
Abstract
Large language models (LLMs) face a fundamental trade-off between computational efficiency (e.g., number of parameters) and output quality, especially when deployed on computationally limited devices such as phones or laptops. One way to address this challenge is by following the example of humans and having models ask for help when they believe they are incapable of solving a problem on their own. By allowing smaller models to respond to queries when they believe they can provide good responses, and deferring to larger models when they do not, we can overcome this trade-off.
Introduction
In this paper, we investigate the viability of Predict-Answer/Act (PA) and Reason-Predict-Reason-Answer/Act (RPRA) paradigms where models predict—prior to responding—how an LLM judge would score their output. This approach aims to enhance the performance of smaller models while maintaining computational efficiency.
Methodology
We evaluate three approaches to predict LLM judges:
- Zero-shot prediction: Smaller models make predictions without prior examples.
- Prediction using an in-context report card: Models use feedback provided in a structured format to improve their predictions.
- Supervised fine-tuning: Smaller models are trained on labeled data to refine their prediction abilities.
Results
Our results demonstrate that larger models, particularly those focused on reasoning, perform well when predicting generic LLM judges in a zero-shot context. Smaller models, however, show significant improvements when subjected to fine-tuning or provided with an in-context report card. Specifically, the mean improvements in prediction accuracy across various datasets are noteworthy:
- Report cards achieved a mean improvement of up to 55%.
- Fine-tuning led to an average improvement of 52%.
Discussion
These findings suggest that models can learn to predict their own performance limitations effectively. By integrating the RPRA framework, AI systems can become more efficient and self-aware, ultimately enhancing their decision-making capabilities. This development paves the way for more robust AI applications that can function efficiently on a range of devices without compromising the quality of their outputs.
Conclusion
As the demand for efficient AI solutions continues to grow, the RPRA approach holds promise in bridging the gap between computational efficiency and output quality. Our research highlights the potential of smaller models to improve their prediction capabilities, making them valuable assets in real-world applications where resource limitations are a concern.