Signals: Trajectory Sampling and Triage for Agentic Interactions
Summary: arXiv:2604.00356v1 Announce Type: new
Abstract: Agentic applications based on large language models increasingly rely on multi-step interaction loops involving planning, action execution, and environment feedback. While such systems are now deployed at scale, improving them post-deployment remains challenging. Agent trajectories are voluminous and non-deterministic, and reviewing each one, whether through human review or auxiliary LLMs, is slow and cost-prohibitive.
We propose a lightweight, signal-based framework for triaging agentic interaction trajectories. Our approach computes cheap, broadly applicable signals from live interactions and attaches them as structured attributes for trajectory triage, identifying interactions likely to be informative without affecting online agent behavior.
We organize signals into a coarse-grained taxonomy spanning:
- Interaction: misalignment, stagnation, disengagement, satisfaction
- Execution: failure, loop
- Environment: exhaustion
This taxonomy is designed for computation without model calls. In a controlled annotation study on τ-bench, a widely used benchmark for tool-augmented agent evaluation, we demonstrate that signal-based sampling achieves an 82% informativeness rate compared to 74% for heuristic filtering and 54% for random sampling. This results in a 1.52x efficiency gain per informative trajectory.
The advantage of our proposed method is robust across reward strata and task domains. This confirms that signals provide genuine per-trajectory informativeness gains rather than merely oversampling obvious failures. The results indicate that lightweight signals can serve as practical sampling infrastructure for agentic systems and suggest a viable path toward preference data construction and post-deployment optimization.
Conclusion
As the landscape of agentic applications continues to evolve, the need for effective post-deployment evaluation and optimization strategies becomes increasingly urgent. The proposed signal-based framework not only simplifies the process of trajectory triage but also enhances the efficiency of identifying informative interactions. By leveraging structured attributes and a taxonomy of signals, developers and researchers can focus their resources on the most relevant data, ultimately leading to improved agent performance and user satisfaction.
This innovative approach has the potential to transform how agentic systems are monitored and refined in real-world settings, paving the way for more intelligent and responsive applications across various domains.