Sentiment Analysis for Software Engineering: How Far Can Zero-Shot Learning (ZSL) Go?
Summary: arXiv:2604.13826v1 Announce Type: cross
Abstract: Sentiment analysis in software engineering focuses on understanding emotions expressed in software artifacts. Previous research highlighted the limitations of applying general off-the-shelf sentiment analysis tools within the software engineering domain and indicated the need for specialized tools tailored to various software engineering contexts. The development of such tools heavily relies on supervised machine learning techniques that necessitate annotated datasets. Acquiring such datasets is a substantial challenge, as it requires domain-specific expertise and significant effort.
Objective
This study explores the potential of Zero-Shot Learning (ZSL) to address the scarcity of annotated datasets in sentiment analysis within software engineering.
Method
We conducted an empirical experiment to evaluate the performance of various ZSL techniques, including:
- Embedding-based techniques
- NLI-based techniques
- TARS-based techniques
- Generative-based techniques
We assessed the performance of these techniques under different label setups to examine the impact of label configurations. Additionally, we compared the results of the ZSL techniques with state-of-the-art fine-tuned transformer-based models. Finally, we performed an error analysis to identify the primary causes of misclassifications.
Results
Our findings demonstrate that ZSL techniques, particularly those combining expert-curated labels with embedding-based or generative-based models, can achieve macro-F1 scores comparable to fine-tuned transformer-based models. The error analysis revealed that:
- Subjectivity in annotation
- Polar facts
are the main contributors to ZSL misclassifications.
Conclusion
This study demonstrates the potential of ZSL for sentiment analysis in software engineering. ZSL can provide a solution to the challenge of annotated dataset scarcity by reducing reliance on annotated datasets.