A Comparative Study of Demonstration Selection for Practical Large Language Models-based Next POI Prediction
Summary: arXiv:2604.06207v1 Announce Type: cross
Abstract
This paper investigates demonstration selection strategies for predicting a user’s next point-of-interest (POI) using large language models (LLMs), aiming to accurately forecast a user’s subsequent location based on historical check-in data. While in-context learning (ICL) with LLMs has recently gained attention as a promising alternative to traditional supervised approaches, the effectiveness of ICL significantly depends on the selected demonstration.
Although previous studies have examined methods such as random selection, embedding-based selection, and task-specific selection, there remains a lack of comprehensive comparative analysis among these strategies. To bridge this gap and clarify the best practices for real-world applications, we comprehensively evaluate existing demonstration selection methods alongside simpler heuristic approaches such as geographical proximity, temporal ordering, and sequential patterns.
Key Findings
Extensive experiments conducted on three real-world datasets indicate that these heuristic methods consistently outperform more complex and computationally demanding embedding-based methods, both in terms of computational cost and prediction accuracy. Notably, in certain scenarios, LLMs using demonstrations selected by these simpler heuristic methods even outperform existing fine-tuned models, without requiring further training.
Demonstration Selection Methods Evaluated
- Random Selection: Selecting demonstrations randomly from the dataset.
- Embedding-based Selection: Utilizing embeddings to select the most relevant demonstrations.
- Task-specific Selection: Choosing demonstrations based on the specific task requirements.
- Geographical Proximity: Selecting demonstrations based on the physical proximity of the user’s previous locations.
- Temporal Ordering: Considering the time sequence of user check-ins to select relevant demonstrations.
- Sequential Patterns: Utilizing historical patterns in user behavior for demonstration selection.
Implications for Future Research
The findings of this study emphasize the importance of demonstration selection strategies in leveraging the capabilities of LLMs for predicting user behavior. As the demand for accurate and efficient POI prediction systems continues to rise, understanding which methods yield the best results can significantly impact the design and implementation of future applications in various domains.
Source Code
Our source code is available at: GitHub Repository.