Hierarchical and Multimodal Data for Daily Activity Understanding
Summary: arXiv:2504.17696v4 Announce Type: replace-cross
Abstract: Daily Activity Recordings for Artificial Intelligence (DARai, pronounced “Dahr-ree”) is a multimodal, hierarchically annotated dataset constructed to understand human activities in real-world settings. DARai consists of continuous scripted and unscripted recordings of 50 participants in 10 different environments, totaling over 200 hours of data from 20 sensors including multiple camera views, depth and radar sensors, wearable inertial measurement units (IMUs), electromyography (EMG), insole pressure sensors, biomonitor sensors, and gaze tracker.
To capture the complexity in human activities, DARai is annotated at three levels of hierarchy:
- High-level activities (L1): Independent tasks.
- Lower-level actions (L2): Patterns shared between activities.
- Fine-grained procedures (L3): Exact execution steps for actions.
The dataset annotations and recordings are designed to facilitate understanding, with 22.7% of L2 actions shared between L1 activities and 14.2% of L3 procedures shared between L2 actions. The overlap and unscripted nature of DARai allow for counterfactual activities within the dataset, enhancing the richness of the data.
Experiments utilizing various machine learning models demonstrate DARai’s significant value in addressing key challenges in human-centered applications. Specifically, we conduct:
- Unimodal sensor experiments for recognition and analysis of activities.
- Multimodal sensor fusion experiments for enhanced performance in recognition, temporal localization, and future action anticipation.
These experiments encompass all hierarchical annotation levels, providing a comprehensive view of the dataset’s capabilities. Additionally, to highlight the limitations of individual sensors, we also conduct domain-variant experiments enabled by DARai’s multi-sensor and counterfactual activity design setup.
The findings from these experiments underline the necessity of utilizing multimodal data for a more nuanced understanding of human activities. The insights gained can drive innovations in areas such as robotics, healthcare, and smart environments.
For researchers and practitioners interested in leveraging the DARai dataset, the code, documentation, and dataset are available at the dedicated DARai website: DARai Dataset.