Let’s Have a Conversation: Designing and Evaluating LLM Agents for Interactive Optimization
Summary: arXiv:2604.02666v1 Announce Type: new
Abstract
Optimization is as much about modeling the right problem as solving it. Identifying the right objectives, constraints, and trade-offs demands extensive interaction between researchers and stakeholders. Large language models (LLMs) can empower decision-makers with optimization capabilities through interactive optimization agents that can propose, interpret, and refine solutions. However, it is fundamentally harder to evaluate a conversation-based interaction than traditional one-shot approaches. This paper proposes a scalable and replicable methodology for evaluating optimization agents through conversations.
Key Insights
- Role-playing Stakeholders: We build LLM-powered decision agents that role-play diverse stakeholders, each governed by an internal utility function but communicating like a real decision-maker.
- Extensive Data Generation: We generate thousands of conversations in a school scheduling case study to provide a robust dataset for evaluation.
- One-shot Evaluation Limitations: Results show that one-shot evaluation is severely limiting; the same optimization agent converges to much higher-quality solutions through conversations.
- Tailored Optimization Agents: This methodology demonstrates that tailored optimization agents, endowed with domain-specific prompts and structured tools, can lead to significant improvements in solution quality in fewer interactions.
- Comparison with General-purpose Chatbots: Tailored agents outperform general-purpose chatbots, showcasing the advantages of specialized design in optimization tasks.
Evaluation Methodology
The proposed methodology emphasizes the importance of conversational interactions in evaluating optimization agents. Unlike traditional one-shot evaluations, conversations allow for a more nuanced understanding of the decision-making process. By simulating dialogues among stakeholders, the agents can iteratively refine their proposals based on feedback, leading to more effective solutions.
Case Study: School Scheduling
In our school scheduling case study, we tested the capabilities of LLM-powered agents across various scenarios. The generated conversations provided insights into how different stakeholder perspectives influence the optimization process. The results indicated that through interaction, agents could discover innovative solutions that would not have emerged through static evaluations.
Implications for Practice
The findings of this research hold significant implications for the field of optimization technology. By integrating operations research expertise into the design of optimization agents, we can enhance their ability to facilitate interactive deployments. This not only improves solution quality but also expands the reach of optimization technologies in practice, making them more accessible to decision-makers across different sectors.
Conclusion
This paper highlights the transformative potential of conversational AI in the realm of optimization. By demonstrating the effectiveness of LLM-powered agents in generating high-quality solutions through interactive optimization, we pave the way for future research and development in this exciting intersection of artificial intelligence and operational efficiency.