SkillMOO: Multi-Objective Optimization of Agent Skills for Software Engineering
Summary: arXiv:2604.09297v1 Announce Type: cross
Abstract
Agent skills provide modular, task-specific guidance for LLM-based coding agents, but manually tuning skill bundles to balance success rate, cost, and runtime is expensive and fragile. We present SkillMOO, a multi-objective optimization framework that automatically evolves skill bundles using LLM-proposed edits and NSGA-II survivor selection: a solver agent evaluates candidate skill bundles on coding tasks and an optimizer agent proposes bundle edits based on failure analysis. On three SkillsBench software engineering tasks, SkillMOO improves pass rate by up to 131% while reducing cost up to 32% relative to the best baseline per task at low optimization overhead. Pattern analysis reveals pruning and substitution as primary drivers of improvement, suggesting effective bundles favor minimal, focused content over accumulated instructions.
Introduction
In the realm of software engineering, the evolution of coding agents powered by large language models (LLMs) has transformed how developers approach programming tasks. However, the optimization of agent skills remains a significant challenge. Traditional methods of tuning skill bundles are often labor-intensive, requiring extensive manual intervention and expertise.
The SkillMOO Framework
SkillMOO addresses these challenges by introducing a sophisticated multi-objective optimization framework. The key components of SkillMOO include:
- LLM-Proposed Edits: Leveraging the capabilities of LLMs, SkillMOO generates potential edits for skill bundles based on their performance in coding tasks.
- NSGA-II Survivor Selection: This genetic algorithm-based method helps in selecting the best candidate bundles, ensuring a diverse set of solutions that balance multiple objectives.
- Solver and Optimizer Agents: The framework employs a dedicated solver agent to evaluate the effectiveness of various skill bundles while the optimizer agent analyzes failures to suggest improvements.
Performance Improvements
SkillMOO’s application on three SkillsBench software engineering tasks has demonstrated remarkable outcomes:
- Pass rate improvement of up to 131%, signifying enhanced effectiveness in task completion.
- Cost reduction of up to 32%, optimizing resource allocation and time management.
- Low optimization overhead, ensuring that the tuning process is efficient and manageable.
Analysis of Improvements
Further analysis reveals that the key drivers behind the performance enhancements are:
- Pruning: The process of removing unnecessary or redundant instructions, leading to more streamlined and effective skill bundles.
- Substitution: Replacing less effective instructions with more relevant and impactful ones, thereby enhancing the overall quality of the skill sets.
This pattern indicates that successful bundles tend to favor minimalistic and focused content over a compilation of extensive instructions, promoting efficiency and clarity in agent skills.
Conclusion
SkillMOO represents a significant advancement in the optimization of agent skills for software engineering. By automating the evolution of skill bundles, it mitigates the challenges associated with manual tuning, offering a robust solution that enhances both performance and cost-effectiveness. As the landscape of software development continues to evolve, frameworks like SkillMOO will play a crucial role in shaping the future of coding agents.