Towards Optimal Agentic Architectures for Offensive Security Tasks
Summary: arXiv:2604.18718v1 Announce Type: cross
Abstract: Agentic security systems increasingly audit live targets with tool-using LLMs, but prior systems fix a single coordination topology, leaving unclear when additional agents help and when they only add cost. We treat topology choice as an empirical systems question.
In a world where cybersecurity threats are ever-evolving, it is crucial to develop advanced security systems capable of effectively auditing live targets. Recent research has focused on the implementation of agentic security systems that utilize tool-using Large Language Models (LLMs) to enhance security measures. However, one critical limitation of existing systems is their reliance on a single coordination topology. This raises questions regarding the effectiveness of deploying additional agents—do they contribute positively, or do they simply increase costs without delivering tangible benefits?
This study aims to address these questions by treating the choice of topology as an empirical systems issue. To facilitate this investigation, we introduce a controlled benchmark comprising 20 interactive targets, categorized into two groups: 10 web/API targets and 10 binary targets. Each target exposes a single endpoint-reachable ground-truth vulnerability, allowing us to evaluate the performance of different architectures in both whitebox and blackbox modes.
The core study encompasses a total of 600 runs that span five architecture families, three model families, and both access modes. In addition, a separate pilot study consisting of 60 long-context runs is reported in the appendix, providing further insights into the findings.
Key Findings
- Detection Rates: The core benchmark results reveal that detection-any reaches 58.0%, while validated detection reaches 49.8%. The MAS-Indep architecture achieves the highest validated detection rate at 64.2%.
- Efficiency: The SAS architecture stands out as the most efficient baseline, operating at a cost of $0.058 per validated finding.
- Comparison of Modes: Whitebox evaluations significantly outperform blackbox assessments, with validated detection rates of 67.0% versus 32.7%, respectively.
- Target Types: The study also indicates that web targets outperform binary targets, achieving validated detection rates of 74.3% compared to 25.3%.
To further understand the implications of these findings, we employed bootstrap confidence intervals and paired target-level deltas. The analysis highlights that the primary factors influencing performance are observability and domain characteristics. Interestingly, some leading whitebox topologies remain statistically close in terms of performance, suggesting that there is still room for optimization.
Cost-Quality Frontier
The main takeaway from this research is the identification of a non-monotonic cost-quality frontier. While broader coordination among agents can enhance coverage, it does not necessarily dominate the landscape once factors like latency, token cost, and exploit-validation difficulty are considered. This finding emphasizes the importance of a balanced approach in designing agentic architectures for offensive security tasks.
In conclusion, this study provides significant insights into the performance dynamics of agentic security systems. As the cybersecurity landscape continues to evolve, understanding the interplay between coordination topologies, detection rates, and associated costs will be vital in crafting effective security solutions.