Graph of Skills: Dependency-Aware Structural Retrieval for Massive Agent Skills
Summary: arXiv:2604.05333v1 Announce Type: new
Abstract: Skill usage has become a core component of modern agent systems and can substantially improve agents’ ability to complete complex tasks. In real-world settings, where agents must monitor and interact with numerous personal applications, web browsers, and other environment interfaces, skill libraries can scale to thousands of reusable skills. Scaling to larger skill sets introduces two key challenges. First, loading the full skill set saturates the context window, driving up token costs, hallucination, and latency.
In this paper, we present Graph of Skills (GoS), an inference-time structural retrieval layer for large skill libraries. GoS constructs an executable skill graph offline from skill packages, then at inference time retrieves a bounded, dependency-aware skill bundle through hybrid semantic-lexical seeding, reverse-weighted Personalized PageRank, and context-budgeted hydration. On SkillsBench and ALFWorld, GoS improves average reward by 43.6% over the vanilla full skill-loading baseline while reducing input tokens by 37.8%, and generalizes across three model families: Claude Sonnet, GPT-5.2 Codex, and MiniMax. Additional ablation studies across skill libraries ranging from 200 to 2,000 skills further demonstrate that GoS consistently outperforms both vanilla skills loading and simple vector retrieval in balancing reward, token efficiency, and runtime.
Key Features of Graph of Skills
- Executable Skill Graph Construction: GoS builds an executable skill graph offline, optimizing retrieval during inference.
- Dependency-Aware Skill Bundles: The method retrieves a limited set of skills that account for dependencies, reducing unnecessary token usage.
- Hybrid Retrieval Techniques: Combines semantic and lexical approaches to enhance skill selection.
- Performance Improvements: Achieves a 43.6% increase in average reward while lowering input tokens by 37.8% compared to traditional methods.
- Broad Generalization: The approach shows effectiveness across different model families, including Claude Sonnet, GPT-5.2 Codex, and MiniMax.
Implications for Future Research
The development of the Graph of Skills marks a significant advancement in the field of artificial intelligence and agent systems. By addressing the challenges associated with large skill libraries, GoS opens new avenues for research and application in various domains, including:
- Enhanced user interaction through more efficient and context-aware systems.
- Potential applications in personal assistants, gaming, robotic control, and more.
- Further exploration of hybrid retrieval methods in optimizing AI performance.
As the landscape of AI continues to evolve, the Graph of Skills could serve as a foundational framework for developing more sophisticated and capable agent systems, ultimately leading to improved user experiences and more effective task completion.