FLEX: A Largescale Multimodal, Multiview Dataset for Learning Structured Representations for Fitness Action Quality Assessment
The recent publication on arXiv, titled “FLEX: A Largescale Multimodal, Multiview Dataset for Learning Structured Representations for Fitness Action Quality Assessment,” introduces a groundbreaking dataset aimed at transforming the realm of Action Quality Assessment (AQA) in fitness environments. This innovative dataset, designated as FLEX, addresses significant limitations in existing AQA datasets, which typically focus on single-view competitive sports and RGB video without incorporating essential multimodal signals and expert evaluations.
Understanding Action Quality Assessment (AQA)
Action Quality Assessment refers to the systematic quantification of how well an action is executed. This is particularly important in the context of gym weight training, where timely and accurate feedback is crucial for preventing injuries and optimizing performance. However, traditional datasets have failed to provide comprehensive data to facilitate effective AQA, necessitating the development of a more robust solution.
Introduction to FLEX
FLEX emerges as the first large-scale, multimodal, multiview dataset tailored for fitness AQA. It includes:
- Over 7,500 multiview recordings of 20 different weight-loaded exercises.
- Data collected from 38 subjects with varying skill levels.
- Synchronized RGB video, 3D pose estimations, surface electromyography (sEMG), and other physiological signals.
Expert Annotations and Fitness Knowledge Graph (FKG)
An integral aspect of FLEX is its expert annotations, which are organized into a comprehensive Fitness Knowledge Graph (FKG). This graph establishes connections between different actions, key steps, types of errors, and corresponding feedback. Such a structured approach not only facilitates a compositional scoring function for an interpretable quality assessment but also enhances the dataset’s utility for research and application in AI-powered fitness coaching.
Innovative Features and Applications
FLEX enables several advanced functionalities, including:
- Multimodal fusion of diverse data types for enriched analysis.
- Cross-modal prediction capabilities, exemplified by the novel Video→EMG task.
- Biomechanically oriented representation learning, promoting a deeper understanding of the mechanics involved in fitness actions.
Introducing FLEX-VideoQA
Building upon the FKG, the authors have also introduced FLEX-VideoQA, a structured question-answering benchmark. This benchmark is designed with hierarchical queries that facilitate cross-modal reasoning within vision-language models, further enhancing the scope of research possibilities and applications in the field.
Baseline Experiments and Future Directions
Initial baseline experiments conducted with FLEX demonstrate that incorporating multimodal inputs, multiview video, and detailed annotations considerably improves AQA performance. The dataset thus represents a significant step forward toward richer multimodal settings in AQA, laying a strong foundation for future developments in AI-assisted fitness assessment and coaching.
Accessing FLEX
Researchers and practitioners interested in utilizing the FLEX dataset can find both the dataset and accompanying code at https://github.com/HaoYin116/FLEX. For further insights and information, visit the project page at https://haoyin116.github.io/FLEX_Dataset.