Select Smarter, Not More: Prompt-Aware Evaluation Scheduling with Submodular Guarantees
Summary: arXiv:2604.11328v1 Announce Type: new
Abstract
Automatic prompt optimization (APO) hinges on the quality of its evaluation signal, yet scoring every prompt candidate on the full training set is prohibitively expensive. Existing methods either fix a single evaluation subset before optimization begins (principled but prompt-agnostic) or adapt it heuristically during optimization (flexible but unstable and lacking formal guarantees). We observe that APO naturally maps to an online adaptive testing problem: prompts are examinees, training examples are test items, and the scheduler should select items that best discriminate among the strongest candidates.
Introduction
This insight motivates Prompt-Aware Online Evaluation Scheduling (POES), which integrates an Item Response Theory (IRT)-based discrimination utility, a facility-location coverage term, and switching-cost-aware warm-start swaps into a unified objective that is provably monotone submodular. This yields a (1-1/e) greedy guarantee for cold starts and bounded drift for warm-start updates.
Methodology
POES employs an adaptive controller that modulates the exploration-exploitation balance based on optimization progress. This allows the evaluation process to be more dynamic and responsive to the needs of the ongoing optimization.
Results
Across 36 tasks spanning three benchmark families, POES achieves the highest overall average accuracy, demonstrating a 6.2 percent improvement over the best baseline with negligible token overhead (approximately 4 percent) at the same evaluation budget. Moreover, principled selection at k = 20 examples matches or exceeds the performance of naive evaluation at k = 30-50, reducing token consumption by 35-60 percent. This highlights a crucial finding: selecting smarter is more effective than selecting more.
Conclusion
Our results demonstrate that evaluation scheduling is a first-class component of Automatic Prompt Optimization, rather than merely an implementation detail. By leveraging the principles of submodularity and adaptive evaluation, POES provides a robust framework for improving the efficiency and effectiveness of prompt optimization processes.
Key Takeaways
- POES introduces a new paradigm for evaluation scheduling in Automatic Prompt Optimization.
- The methodology integrates several advanced techniques to create a unified objective.
- Empirical results show significant improvements in accuracy while reducing token consumption.
- The study illustrates the importance of principled evaluation selection over merely increasing the number of evaluations.