Seeing the Imagined: A Latent Functional Alignment in Visual Imagery Decoding from fMRI Data
Summary: arXiv:2604.15374v1 Announce Type: cross
Abstract
Recent progress in visual brain decoding from functional Magnetic Resonance Imaging (fMRI) has been significantly driven by the availability of large-scale datasets such as the Natural Scenes Dataset (NSD) and the emergence of powerful diffusion-based generative models. However, existing decoding pipelines are primarily optimized for perceptual tasks, leaving their efficacy in the realm of mental imagery less understood. This article explores the adaptation of a state-of-the-art (SOTA) perception decoder, known as DynaDiff, to reconstruct imagined visual content from the Imagery-NSD benchmark.
Introduction
The intersection of neuroscience and artificial intelligence has paved the way for groundbreaking advancements in understanding how our brain processes and reconstructs visual information. As research continues to evolve, the decoding of visual imagery from fMRI data presents a unique challenge, particularly when differentiating between perception and imagination.
Methodology
In this study, we propose a novel approach termed “latent functional alignment,” which serves to map imagery-evoked neural activity into the conditioning space of a pretrained model while keeping other components fixed. This approach is particularly crucial in scenarios where the amount of matched imagery-perception supervision is limited. To address this, we introduce a retrieval-based augmentation strategy that selects semantically related perception trials from the NSD.
Results
Our experiments involved four subjects, and the results demonstrated that latent functional alignment consistently enhances high-level semantic reconstruction metrics. These improvements were observed in comparison to both the frozen pretrained baseline and a voxel-space ridge alignment baseline. Notably, our methodology enabled above-chance decoding from multiple cortical regions, indicating a promising avenue for future research.
Discussion
The findings of this study suggest that leveraging the semantic structure learned from perceptual tasks can stabilize and enhance visual imagery decoding, even under out-of-distribution conditions. This has significant implications for the broader field of cognitive neuroscience and artificial intelligence, particularly in applications such as brain-computer interfaces and the understanding of human cognition.
Conclusion
As we continue to unravel the complexities of our brain’s visual processing capabilities, the adaptation of perception decoders for mental imagery presents a fascinating frontier in neuroscience research. The latent functional alignment approach offers a robust framework for improving visual imagery decoding, holding promise for future advancements in both theoretical understanding and practical applications.
Future Work
- Further exploration of the latent functional alignment technique in varied contexts.
- Investigation into additional retrieval-based strategies for more robust decoding.
- Application of the findings to real-world scenarios, including brain-computer interfaces.
- Collaboration with cognitive scientists to deepen the understanding of mental imagery.