SUMMIR: A Hallucination-Aware Framework for Ranking Sports Insights from LLMs
In the rapidly evolving landscape of online sports journalism, the ability to extract meaningful pre-game and post-game insights from articles has become increasingly essential. This necessity not only enhances user engagement but also aids in the comprehension of sports narratives. A recent paper, titled “SUMMIR: A Hallucination-Aware Framework for Ranking Sports Insights from LLMs”, addresses this pressing challenge by proposing an innovative solution for automatic insight extraction from sports articles.
The authors of the paper have curated an extensive dataset comprising 7,900 news articles that cover 800 matches across four major sports: Cricket, Soccer, Basketball, and Baseball. This dataset serves as the foundation for developing a robust framework that leverages advanced large language models (LLMs) to generate comprehensive insights.
Methodology
To ensure the contextual relevance of the extracted insights, the researchers employed a two-step validation pipeline that utilizes both open-source and proprietary LLMs. The methodology can be summarized in the following steps:
- Data Curation: The dataset was meticulously compiled to include a variety of articles that span multiple sports disciplines.
- Model Selection: The team utilized multiple state-of-the-art LLMs, including GPT-4o, Qwen2.5-72B-Instruct, Llama-3.3-70B-Instruct, and Mixtral-8x7B-Instruct-v0.1, to generate insights.
- Factual Accuracy Assessment: The accuracy of the generated outputs was rigorously evaluated using a FactScore-based methodology.
- Hallucination Detection: The SummaC (Summary Consistency) framework with GPT-4o was employed to detect any hallucinations in the generated content.
Introducing SUMMIR
The culmination of this research is the introduction of SUMMIR (Sentence Unified Multimetric Model for Importance Ranking), a novel architecture designed to rank insights based on user-specific interests. This innovative model not only provides high-quality, relevant insights but also highlights significant differences in factual consistency and interestingness across the various LLMs utilized in the study.
The findings of this research indicate that SUMMIR effectively generates reliable and engaging insights from sports news content, paving the way for enhanced user experiences in sports journalism. The framework promises to contribute significantly to the field of automated journalism by ensuring that the insights provided are both accurate and tailored to the interests of users.
Further Research and Availability
For those interested in exploring the technical details of this framework, the source code is available on GitHub at the following link:
SUMMIR GitHub Repository.
This work not only addresses the current gaps in automated insight generation but also sets the stage for future advancements in the realm of sports journalism.