Towards neural foundation models for vision: Aligning EEG, MEG, and fMRI representations for decoding, encoding, and modality conversion.
Saved in:
| Title: | Towards neural foundation models for vision: Aligning EEG, MEG, and fMRI representations for decoding, encoding, and modality conversion. |
|---|---|
| Authors: | Ferrante, Matteo1 (AUTHOR) matteo.ferrante@uniroma2.it, Boccato, Tommaso1 (AUTHOR), Rashkov, Grigorii1 (AUTHOR), Toschi, Nicola1,2 (AUTHOR) |
| Source: | Information Fusion. Feb2026:Part B, Vol. 126, pN.PAG-N.PAG. 1p. |
| Subjects: | Electroencephalography, Functional magnetic resonance imaging, Magnetoencephalography, Artificial neural networks, Machine learning |
| Abstract: | This paper presents a novel approach towards creating a foundational model for aligning neural data and visual stimuli across cross-modal representations of brain activity by leveraging contrastive learning. We leverage electroencephalography (EEG), magnetoencephalography (MEG), and functional magnetic resonance imaging (fMRI) data. The framework is validated through three key experiments: decoding visual information from neural data, encoding images into neural representations, and converting between neural modalities. The results highlight the model's ability to accurately capture semantic information across different brain imaging techniques, underscoring its potential in decoding, encoding, and modality conversion tasks. Our results reveal that EEG, MEG, and fMRI signals – despite being collected from different subjects and datasets – can be aligned in a shared vision-grounded space that preserves semantic information. This finding suggests the existence of modality-invariant neural codes and offers a new framework for comparing and decoding brain activity across heterogeneous non-invasive recordings. • Unified Multimodal Representation : Aligns neural data with visual stimuli • Three Key Capabilities : Neural data decoding, encoding, and cross-modality conversion • High Decoding Accuracy : Achieves 93% CLIP 2-way accuracy with fMRI • Cross-Modality Consistency: Translates semantic content across EEG, MEG, and fMRI. • Towards a Neural Foundation Model : Advances a unified framework for neural data, [ABSTRACT FROM AUTHOR] |
| Copyright of Information Fusion is the property of Elsevier B.V. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
Be the first to leave a comment!
