TraceSafe: A Systematic Assessment of LLM Guardrails on Multi-Step Tool-Calling Trajectories
Summary: arXiv:2604.07223v1 Announce Type: cross
As large language models (LLMs) evolve from static chatbots into autonomous agents, the primary vulnerability surface shifts from final outputs to intermediate execution traces. While safety guardrails are well-benchmarked for natural language responses, their efficacy remains largely unexplored within multi-step tool-use trajectories.
To address this gap, we introduce TraceSafe-Bench, the first comprehensive benchmark specifically designed to assess mid-trajectory safety. This benchmark encompasses 12 risk categories, ranging from security threats (e.g., prompt injection, privacy leaks) to operational failures (e.g., hallucinations, interface inconsistencies), featuring over 1,000 unique execution instances.
Key Findings
Our evaluation of 13 LLM-as-a-guard models and 7 specialized guardrails yields three critical findings:
- Structural Bottleneck: Guardrail efficacy is driven more by structural data competence (e.g., JSON parsing) than semantic safety alignment. Performance correlates strongly with structured-to-text benchmarks (ρ=0.79) but shows near-zero correlation with standard jailbreak robustness.
- Architecture over Scale: Model architecture influences risk detection performance more significantly than model size. General-purpose LLMs consistently outperform specialized safety guardrails in trajectory analysis.
- Temporal Stability: Accuracy remains resilient across extended trajectories. Increased execution steps allow models to pivot from static tool definitions to dynamic execution behaviors, actually improving risk detection performance in later stages.
Conclusion
Our findings suggest that securing agentic workflows requires jointly optimizing for structural reasoning and safety alignment to effectively mitigate mid-trajectory risks. With the introduction of TraceSafe-Bench, researchers and developers can better understand the vulnerabilities associated with multi-step tool-using trajectories and enhance the safety features of LLMs as they continue to develop into more autonomous systems.
This comprehensive assessment not only highlights the importance of mid-trajectory safety but also sets the foundation for future research aimed at improving the robustness and reliability of autonomous agents operating in complex environments.