Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding
Summary: arXiv:2603.26211v1 Announce Type: cross
Abstract: Autoregressive (AR) vision-language models (VLMs) have long dominated multimodal understanding, reasoning, and graphical user interface (GUI) grounding. Recently, discrete diffusion vision-language models (DVLMs) have shown strong performance in multimodal reasoning, offering bidirectional attention, parallel token generation, and iterative refinement. However, their potential for GUI grounding remains unexplored. In this work, we evaluate whether discrete DVLMs can serve as a viable alternative to AR models for GUI grounding.
Introduction
The exploration of vision-language models has significantly advanced the field of artificial intelligence, particularly in the realms of understanding and reasoning across multiple modalities. Traditionally, autoregressive models have led this domain, yet recent developments in discrete diffusion models present new opportunities for improvement, particularly in the area of GUI grounding.
Methodology
In our research, we adapt LLaDA-V specifically for tasks involving single-turn action and bounding-box prediction. This adaptation redefines GUI grounding as a task of text generation from multimodal inputs. To enhance the accuracy of bounding-box geometry representation, we introduce a novel hybrid masking schedule that integrates both linear and deterministic masking techniques.
Results
Our findings indicate a significant improvement in grounding accuracy, achieving up to a 6.1 point increase in Step Success Rate (SSR) when compared to the traditional linear masking approach used in the GUI-adapted LLaDA-V. Evaluations conducted across four diverse datasets, which included web, desktop, and mobile interfaces, consistently demonstrated that the adapted diffusion model with hybrid masking outperforms its linear-masked counterpart.
Comparative Analysis
Despite being trained with limited pretraining data, our diffusion model performs competitively against autoregressive counterparts. Systematic ablation studies reveal that increasing the number of diffusion steps, generation length, and block length correlates with higher accuracy. However, it is noteworthy that accuracy tends to plateau after a certain threshold of diffusion steps, indicating a balance must be struck between accuracy and latency.
Training Data and Latency Reduction
Expanding the training dataset to incorporate a broader range of GUI domains has been shown to effectively reduce latency—by approximately 1.3 seconds—while simultaneously enhancing grounding accuracy by an average of 20 points across various benchmarks.
Conclusion
The results of our study underscore the potential of discrete DVLMs as a robust modeling framework for GUI grounding. This research marks a significant step toward the development of diffusion-based GUI agents, paving the way for future advancements in multimodal understanding and interaction.
Future Work
- Further exploration of hybrid masking techniques.
- In-depth analysis of latency versus accuracy trade-offs.
- Expansion of training datasets to cover more diverse application domains.
- Investigation into the scalability of the proposed models.