Interactive Program Synthesis for Modeling Collaborative Physical Activities from Narrated Demonstrations
Summary: arXiv:2509.24250v3 Announce Type: replace
Abstract: Teaching systems physical tasks has been a long-standing goal in Human-Computer Interaction (HCI). Historically, most research has concentrated on non-collaborative physical activities. However, collaborative tasks introduce an added layer of complexity, requiring systems to infer users’ assumptions about their teammates’ intents. This process is inherently ambiguous and dynamic, necessitating representations that are both interpretable and correctable. Such features enable users to inspect and refine system behavior effectively.
This article addresses the challenges associated with collaborative task learning by framing it as a program synthesis problem. Our innovative system represents behaviors as editable programs and employs narrated demonstrations—paired physical actions and natural language—as a unified modality for teaching, inspecting, and correcting system logic. Remarkably, this can be accomplished without requiring users to see or write any code. The same modality is utilized for the system to communicate its learning back to users, fostering a more intuitive interaction.
Study Overview
In a within-subjects study involving 20 participants, users were tasked with teaching multiplayer soccer tactics to our system. The results were promising, with significant insights into user interaction and system learning.
Key Findings
- 70% (14 out of 20) of participants successfully refined learned programs to align with their intent.
- 90% (18 out of 20) found it easy to correct the programs generated by the system.
The study surfaced unique challenges associated with representing learning as programs and the complexities of enabling users to teach collaborative physical activities. These challenges highlight the need for more robust frameworks to support user interaction in collaborative environments.
Challenges Identified
- Representation of Learning: Developing ways to effectively represent learned behaviors as editable programs proved to be a significant hurdle.
- User Instruction: Guiding users to teach collaborative physical activities in a manner that is both intuitive and effective posed additional complexities.
Mitigation Strategies
To address the challenges identified, we propose several strategies:
- Enhancing the interpretability of program representations to facilitate user understanding.
- Implementing interactive tutorials that guide users through the process of teaching and correcting the system.
- Creating visual feedback mechanisms that allow users to see the impact of their corrections in real-time.
Conclusion
The findings from our study reveal valuable insights into the complexities of collaborative task learning through program synthesis. By framing collaborative activities in this way, we can empower users to teach and refine system behaviors effectively. Future work will focus on refining these strategies and exploring broader applications of our approach in various collaborative settings.