A Multi-Modal Deep Learning Framework for High-Resolution Alpine Land Use/Land Cover.
Saved in:
| Title: | A Multi-Modal Deep Learning Framework for High-Resolution Alpine Land Use/Land Cover. |
|---|---|
| Authors: | Dabove, Paolo1 (AUTHOR) paolo.dabove@polito.it, Rao, Deepak Sairam Madhusudhana2 (AUTHOR), Olivotto, Luca2,3 (AUTHOR), Pividori, Ludovico1,2 (AUTHOR), Filippa, Gianluca2,3 (AUTHOR), Morra di Cella, Umberto3 (AUTHOR) |
| Source: | Remote Sensing. Sep2026, Vol. 18 Issue 18, p3203. 37p. |
| Subjects: | Multispectral imaging, Machine learning, Mountain ecology, Digital elevation models |
| Abstract: | Highlights: What are the main findings? The proposed weakly supervised multimodal framework, trained using only 2517 sparsely distributed annotation polygons and without requiring exhaustive wall-to-wall pixel annotations, achieved a validation mIoU of 0.777 across fourteen alpine land-cover classes. The ablation experiments identified the multispectral stream as the principal contributor to classification performance in the investigated dataset. Removing this stream reduced the validation mIoU by more than 35 percentage points, whereas removing the normalised Digital Surface Model (nDSMs) reduced it by fewer than 5 percentage points. These results indicate that spectral information provided substantially greater discriminative value than topographic information under the adopted weak-supervision setting. What are the implications of the main finding? Learning from sparsely distributed annotation polygons substantially reduces the need for exhaustive pixel-wise labelling. This provides environmental agencies with a practical and scalable approach to producing high-resolution alpine land-cover maps, while retaining the need for expert involvement in defining the classification scheme, validating annotations, and reviewing the resulting maps. For weakly supervised alpine mapping, spectral breadth appears to matter more than backbone scale: the principal residual failure, Wetlands, is tentatively attributed to a moisture signal in the shortwave infrared that a six-band payload cannot acquire. Accurate Land Use and Land Cover (LULC) mapping in high-resolution alpine environments is challenging due to complex terrain, heterogeneous vegetation, seasonal snow and ice cover, and the limited spectral information provided by conventional aerial imagery. Although foundation models such as the Segment Anything Model (SAM) effectively capture structural features, their class-agnostic design, limits fine-grained semantic discrimination and typically requires large annotated datasets. This study proposes a multi-modal deep learning framework for alpine LULC mapping using sparse annotations, which would fall under the category of weakly supervised learning. The framework employs a dual-encoder architecture that integrates RGB imagery, six-band multispectral imagery, and custom adapters for spectral indices, and Digital Surface Models (DSMs). A SAM-based encoder extracts geometric and contextual features from RGB imagery, while a dedicated encoder learns complementary spectral representations from multispectral data. To address the boundary uncertainty introduced by sparse supervision, we propose post inference hybrid refinement strategy that combines a Canopy Height Model (CHM) derived from the DSM to improve tree crown delineation with edge-based refinement for low vegetation classes, such as shrubs, and mathematical methods to refine road and building edge delineation with DSMs. Experimental results across fourteen alpine classes highlight that the framework achieves a validation mIoU of 0.777, macro-averaged over the fourteen classes. For the present sensor choice and classification scheme, no comparable multi-modal baseline exists. Thus, results are reported in absolute terms. The present multi-modal data fusion sets the baseline for future scalable alpine LULC mapping applications. [ABSTRACT FROM AUTHOR] |
| Copyright of Remote Sensing is the property of MDPI and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | Highlights: What are the main findings? The proposed weakly supervised multimodal framework, trained using only 2517 sparsely distributed annotation polygons and without requiring exhaustive wall-to-wall pixel annotations, achieved a validation mIoU of 0.777 across fourteen alpine land-cover classes. The ablation experiments identified the multispectral stream as the principal contributor to classification performance in the investigated dataset. Removing this stream reduced the validation mIoU by more than 35 percentage points, whereas removing the normalised Digital Surface Model (nDSMs) reduced it by fewer than 5 percentage points. These results indicate that spectral information provided substantially greater discriminative value than topographic information under the adopted weak-supervision setting. What are the implications of the main finding? Learning from sparsely distributed annotation polygons substantially reduces the need for exhaustive pixel-wise labelling. This provides environmental agencies with a practical and scalable approach to producing high-resolution alpine land-cover maps, while retaining the need for expert involvement in defining the classification scheme, validating annotations, and reviewing the resulting maps. For weakly supervised alpine mapping, spectral breadth appears to matter more than backbone scale: the principal residual failure, Wetlands, is tentatively attributed to a moisture signal in the shortwave infrared that a six-band payload cannot acquire. Accurate Land Use and Land Cover (LULC) mapping in high-resolution alpine environments is challenging due to complex terrain, heterogeneous vegetation, seasonal snow and ice cover, and the limited spectral information provided by conventional aerial imagery. Although foundation models such as the Segment Anything Model (SAM) effectively capture structural features, their class-agnostic design, limits fine-grained semantic discrimination and typically requires large annotated datasets. This study proposes a multi-modal deep learning framework for alpine LULC mapping using sparse annotations, which would fall under the category of weakly supervised learning. The framework employs a dual-encoder architecture that integrates RGB imagery, six-band multispectral imagery, and custom adapters for spectral indices, and Digital Surface Models (DSMs). A SAM-based encoder extracts geometric and contextual features from RGB imagery, while a dedicated encoder learns complementary spectral representations from multispectral data. To address the boundary uncertainty introduced by sparse supervision, we propose post inference hybrid refinement strategy that combines a Canopy Height Model (CHM) derived from the DSM to improve tree crown delineation with edge-based refinement for low vegetation classes, such as shrubs, and mathematical methods to refine road and building edge delineation with DSMs. Experimental results across fourteen alpine classes highlight that the framework achieves a validation mIoU of 0.777, macro-averaged over the fourteen classes. For the present sensor choice and classification scheme, no comparable multi-modal baseline exists. Thus, results are reported in absolute terms. The present multi-modal data fusion sets the baseline for future scalable alpine LULC mapping applications. [ABSTRACT FROM AUTHOR] |
|---|---|
| ISSN: | 20724292 |
| DOI: | 10.3390/rs18183203 |
