The Art of Building Verifiers for Computer Use Agents
In the rapidly evolving field of artificial intelligence, verifying the success of computer use agent (CUA) trajectories presents a critical challenge. Without reliable verification mechanisms, both evaluation and training signals become unreliable, leading to potential setbacks in AI advancements. A recent paper titled “The Art of Building Verifiers for Computer Use Agents,” available on arXiv, introduces a novel verification system known as the Universal Verifier, which aims to address these challenges.
The Universal Verifier is built upon four key principles that significantly enhance the reliability of CUA trajectory evaluations:
- Constructing Rubrics with Meaningful, Non-overlapping Criteria: This principle is designed to reduce noise in evaluations. By ensuring that the criteria do not overlap, the verification process becomes more precise and trustworthy.
- Separating Process and Outcome Rewards: This strategy allows for the capture of complementary signals, particularly in scenarios where an agent may follow the correct steps but encounters obstacles, or conversely, succeeds through unexpected means.
- Distinguishing Between Controllable and Uncontrollable Failures: The cascading-error-free strategy employed allows for a more nuanced understanding of failures, enabling finer-grained evaluations that can inform subsequent training processes.
- Divide-and-Conquer Context Management Scheme: This approach ensures that all screenshots within a trajectory are attended to, thereby improving reliability, especially for tasks with longer horizons.
Validation of the Universal Verifier’s efficacy was conducted using CUAVerifierBench, a newly developed set of CUA trajectories equipped with both process and outcome human labels. Remarkably, the Universal Verifier demonstrated a high level of agreement with human evaluations, matching human consensus rates.
One of the most significant findings was the notable reduction in false positive rates, achieving near-zero rates when compared to existing baselines such as WebVoyager, which had rates exceeding 45%, and WebJudge, with rates above 22%. These improvements are attributed to the cumulative effects of the design principles outlined above.
In a compelling comparison, the research also revealed that an auto-research agent could achieve approximately 70% of expert quality in just 5% of the time required by human experts. However, this auto-research agent fell short in discovering all necessary strategies to replicate the Universal Verifier’s success.
To promote further research and development in this area, the authors have open-sourced the Universal Verifier system along with CUAVerifierBench. Interested parties can access these resources at https://github.com/microsoft/fara.
The advancements presented in this paper herald a significant step forward in the field of AI verification, offering a robust framework that enhances the reliability of computer use agents and their trajectories.