Mobile GUI Agent Privacy Personalization with Trajectory Induced Preference Optimization
Summary: arXiv:2604.11259v1 Announce Type: new
Abstract: Mobile GUI agents powered by Multimodal Large Language Models (MLLMs) can execute complex tasks on mobile devices. Despite this progress, most existing systems still optimize task success or efficiency, neglecting users’ privacy personalization. In this paper, we study the often-overlooked problem of agent personalization.
As mobile technology continues to evolve, the introduction of Mobile GUI agents has transformed how users interact with their devices. However, a critical aspect of this advancement is the need for privacy personalization, which has not been adequately addressed in existing systems. This article delves into the findings of a new study that proposes a novel approach to optimize user preferences concerning privacy.
Understanding Personalization in Mobile GUI Agents
Personalization is essential for enhancing user experience, especially for mobile users who have varying preferences regarding privacy and utility. The study highlights that user preferences can lead to significant differences in execution trajectories. For instance:
- Privacy-first Users: These users tend to prioritize their privacy by opting for protective actions such as refusing permissions, logging out of applications, and minimizing their exposure to data collection.
- Utility-first Users: In contrast, these users prioritize efficiency and are more likely to grant permissions and engage with services that enhance convenience, often at the expense of their privacy.
These differing preferences result in variable-length execution trajectories, which pose challenges for conventional preference optimization methods. The study reveals that standard optimization techniques may become unstable and less informative when faced with such structural heterogeneity.
Introducing Trajectory Induced Preference Optimization (TIPO)
To tackle the challenges associated with personalization in mobile GUI agents, the researchers introduce a new method called Trajectory Induced Preference Optimization (TIPO). This innovative approach consists of two key components:
- Preference-Intensity Weighting: This feature emphasizes crucial privacy-related actions within the execution trajectory, ensuring that users’ privacy preferences are prioritized during task execution.
- Padding Gating: This mechanism helps to suppress alignment noise, which can distort the optimization process and prevent the accurate representation of user preferences.
The effectiveness of TIPO was evaluated using the Privacy Preference Dataset, and the results are promising. The study reported significant improvements in persona alignment and distinction while also maintaining robust task executability. The key performance metrics achieved were:
- Success Rate (SR): 65.60%
- Compliance Rate: 46.22%
- Privacy Distinction (PD): 66.67%
These results demonstrate that TIPO outperforms existing optimization methods across a variety of GUI tasks, paving the way for enhanced personalization in mobile GUI agents.
Conclusion
The work presented in this study represents a significant step forward in addressing the critical issue of privacy personalization in mobile GUI agents. By implementing TIPO, developers can create more user-centric applications that respect and prioritize user privacy without sacrificing task efficiency. The code and dataset will be publicly available at https://github.com/Zhixin-L/TIPO.