Automatic Combination of Sample Selection Strategies for Few-Shot Learning
Summary: arXiv:2402.03038v2 Announce Type: replace-cross
Abstract
In few-shot learning, the selection of samples has a significant impact on the performance of the model. While effective sample selection strategies are well-established in supervised settings, research on large language models largely overlooks them, favouring strategies specifically tailored to individual in-context learning settings. In this paper, we propose a new method for Automatic Combination of SamplE Selection Strategies (ACSESS) to leverage the strengths and complementarity of various well-established selection objectives.
Introduction
Few-shot learning has emerged as a critical area of research in machine learning, particularly in situations where labeled data is scarce. The effectiveness of few-shot learning models heavily relies on the careful selection of training samples. Traditional sample selection methods have shown promise in supervised environments, but their application in few-shot learning contexts, especially with large language models, remains underexplored.
Methodology
We introduce the ACSESS method, which aims to automatically combine multiple sample selection strategies. By investigating 23 different sample selection strategies, we assess their impact on five in-context learning models and three few-shot learning approaches, specifically meta-learning and few-shot fine-tuning. The experiments are conducted across six text datasets and eight image datasets.
Experimental Results
The results of our experiments reveal several key findings:
- The ACSESS method consistently outperforms all individual selection strategies.
- In several cases, the performance of ACSESS is on par with or exceeds that of in-context learning specific baselines.
- Sample selection proves to be particularly effective on smaller datasets, yielding significant improvements when only a few shots are used.
- As the number of shots increases, the advantages of sample selection diminish, suggesting an optimal range for its effectiveness.
Conclusion
The findings from our research underscore the importance of effective sample selection in few-shot learning contexts. The ACSESS method offers a novel approach to integrating various sample selection strategies, thereby enhancing model performance. This research not only fills a gap in the current literature on few-shot learning but also paves the way for further exploration into automated methods for sample selection in machine learning.
Future Work
Future research could explore the extension of the ACSESS method to other domains and datasets, as well as investigate the potential of incorporating additional selection strategies. Furthermore, understanding the underlying reasons for the diminishing returns of sample selection as the number of shots increases remains an intriguing avenue for future studies.