Semantic Interaction for Narrative Map Sensemaking: An Insight-based Evaluation
Summary: arXiv:2603.29651v1 Announce Type: cross
Abstract: Semantic interaction (SI) enables analysts to incorporate their cognitive processes into AI models through direct manipulation of visualizations. While SI frameworks for narrative extraction have been proposed, empirical evaluations of their effectiveness remain limited.
This paper presents a user study that evaluates SI for narrative map sensemaking, involving 33 participants under three conditions: a timeline baseline, a basic narrative map, and an interactive narrative map with SI capabilities. The results show that the map-based prototypes yielded more insights than the timeline baseline, with the SI-enabled condition reaching statistical significance and the basic map condition trending in the same direction.
Key Findings
The study identified several significant findings regarding the effectiveness of semantic interaction in narrative map sensemaking:
- The SI-enabled condition exhibited the highest mean performance among the three tested scenarios.
- Differences between the basic narrative map and the timeline baseline were not statistically significant, although they indicated large effect sizes (d > 0.8), suggesting potential underpowering of the study.
- Qualitative analysis revealed two distinct approaches to semantic interaction: corrective and additive. These approaches allow analysts to impose quality judgments and organizational structures on the extracted narratives.
- Participants using SI achieved comparable exploration breadth with less parameter manipulation compared to those using other methods, indicating that SI provides an alternative pathway for model refinement.
Methodology
The user study consisted of 33 participants who were tasked with narrative map sensemaking using three different conditions:
- Timeline Baseline: A traditional timeline representation that serves as a standard for comparison.
- Basic Narrative Map: A simple map representation that lacks interactive capabilities.
- Interactive Narrative Map with SI: A dynamic map that allows users to engage with the data through semantic interaction, enhancing their analytical capabilities.
Conclusion
This research provides empirical evidence supporting the superiority of map-based representations over traditional timelines for narrative sensemaking. Additionally, it offers qualitative insights into how analysts utilize semantic interaction to refine narratives, highlighting the potential for further development in AI-assisted analytical tools.
Future work should focus on expanding the participant pool and conditions tested to further validate these findings and explore additional dimensions of semantic interaction in narrative extraction.