E-mail senden E-Mail Adresse kopieren
2026-09-01

Collaborative Unpaired Multimodal Sensor Fusion for Image Classification

Zusammenfassung

Multimodal learning typically requires expensive paired data for training and assumes all modalities are available at inference. In many sensor-fusion settings, however, data are collected independently by different institutions, come from heterogeneous sensors, and are not aligned at the sample level. We introduce Unpaired Multimodal Learning (UML) as the problem of leveraging semantically related but unaligned data across modalities, without requiring explicit pairing or multimodal inference. This setting arises in satellite sensor fusion, where institutions collect data from diverse sensors (optical, multispectral, SAR), but paired acquisitions are rare, and data sharing is restricted. We propose a collaborative framework that combines modality-specific projections with a shared backbone, enabling cross-modal knowledge transfer without paired samples. A key element is post-hoc batch normalization calibration, which adapts the shared model to each modality. Our framework also extends naturally to federated training across institutions. Experiments on satellite sensor-fusion benchmarks and additional visual datasets show that the framework can improve unimodal baselines, with the clearest gains for weaker modalities and in low-data regimes. The effect is dataset-dependent, and comparisons with existing unpaired and domain-adaptation baselines indicate both the promise and limitations of this simple collaborative architecture.

Artikel

Veröffentlichungsdatum

2026-09-01

Letztes Änderungsdatum

2026-11-13