ProbGuard: Probabilistic Runtime Monitoring for LLM Agent Safety
Summary: arXiv:2508.00500v3 Announce Type: replace
Abstract: Large Language Model (LLM) agents increasingly operate across domains such as robotics, virtual assistants, and web automation. However, their stochastic decision-making introduces safety risks that are difficult to anticipate during execution. Existing runtime monitoring frameworks, such as AgentSpec, primarily rely on reactive safety rules that detect violations only when unsafe behavior is imminent or has already occurred, limiting their ability to handle long-horizon dependencies.
We present ProbGuard, a proactive runtime monitoring framework for LLM agents that anticipates safety violations through probabilistic risk prediction. ProbGuard abstracts agent executions into symbolic states and learns a Discrete-Time Markov Chain (DTMC) from execution traces to model behavioral dynamics. At runtime, the monitor estimates the probability that future executions will reach unsafe states and triggers interventions when this risk exceeds a user-defined threshold.
To improve robustness, ProbGuard incorporates semantic validity constraints in the abstraction and provides PAC-style guarantees on the learned model under standard assumptions. We evaluate ProbGuard in two safety-critical domains: autonomous driving and embodied household agents. Across evaluated scenarios, ProbGuard consistently predicts traffic law violations and collisions in advance, with warnings up to 38.66 seconds ahead of occurrence. In embodied agent tasks, ProbGuard reduces unsafe behavior by up to 65.37% while preserving up to 80.4% task completion.
Key Features of ProbGuard
- Proactive Monitoring: Unlike existing frameworks that react to unsafe behavior, ProbGuard anticipates potential violations before they occur.
- Probabilistic Risk Prediction: By modeling agent behavior through a Discrete-Time Markov Chain, ProbGuard can estimate the likelihood of reaching unsafe states.
- Intervention Mechanism: The system triggers interventions when the estimated risk exceeds a user-defined threshold, enhancing overall safety.
- Semantic Validity Constraints: Incorporating these constraints improves the robustness of the monitoring framework.
- PAC-Style Guarantees: ProbGuard provides probabilistic guarantees on the learned model under standard assumptions, ensuring reliable performance.
Evaluation Results
ProbGuard has been rigorously tested in two critical domains:
- Autonomous Driving: The framework effectively predicts violations of traffic laws and potential collisions, providing warnings up to 38.66 seconds in advance.
- Embodied Household Agents: In this context, ProbGuard reduced instances of unsafe behavior by up to 65.37% while maintaining a high task completion rate of up to 80.4%.
Conclusion
ProbGuard represents a significant advancement in the field of runtime monitoring for LLM agents. By focusing on proactive safety measures and probabilistic modeling, it addresses the limitations of existing frameworks. The open-source implementation, designed to integrate with the LangChain agent framework, offers a minimal runtime overhead, making it a practical solution for enhancing the safety of LLM applications across various domains.