Classifying Problem and Solution Framing in Congressional Social Media
Summary: arXiv:2604.03247v1 Announce Type: cross
Abstract: Policy setting in the USA according to the “Garbage Can” model differentiates between “problem” and “solution” focused processes. In this paper, we study a large dataset of US Senator postings on Twitter (1.68m tweets in total). Our objective is to develop an automated method to label Senatorial posts as either in the problem or solution streams.
Two academic policy experts labeled a subset of 3967 tweets as either problem, solution, or other (anything not problem or solution). We split off a subset of 500 tweets into a test set, with the remaining 3467 used for training. During development, this training set was further split by 60/20/20 proportions for fitting, validation, and development test sets.
Research Methodology
We investigated supervised learning methods for building problem/solution classifiers directly on the training set. The evaluation of model performance was conducted in terms of F1 score on the validation set. This approach allowed us to rapidly iterate through various models and hyperparameters.
- Dataset Size: The total dataset comprised 1.68 million tweets from US Senators.
- Labeling Process: A subset of 3967 tweets was manually labeled by policy experts.
- Training and Testing: The data was divided into training (3467 tweets) and testing (500 tweets) subsets.
- Model Evaluation: We focused on achieving a high F1 score to ensure the accuracy of our classifications.
Model Development
Our model development involved using a BERTweet Base model, which is specifically designed for processing tweets. We adopted a cross-validation methodology, allowing us to assess the model’s performance across different subsets of the data. This iterative process was essential for refining our model and enhancing its predictive capability.
Through this rigorous evaluation, we achieved an average weighted F1 score of above 0.8 across the three categories: problem, solution, and other. This high level of accuracy indicates that our automated classification method is effective in distinguishing between problem-oriented and solution-oriented tweets.
Implications of the Research
The findings from this research have important implications for understanding how social media is utilized in the policy-making process. By classifying tweets into problem and solution streams, we can gain insights into the priorities and focus areas of US Senators as they engage with their constituents and the broader public.
This automated classification approach not only aids in analyzing the content of political discourse on social media but also contributes to the development of more effective communication strategies for policymakers. As social media continues to play a pivotal role in shaping public opinion and informing policy discussions, our research underscores the importance of leveraging technology to enhance understanding and engagement in the political arena.