Multimodal Prompt Learning for Spatial Reasoning in Remote Sensing Image Scene.
Saved in:
| Title: | Multimodal Prompt Learning for Spatial Reasoning in Remote Sensing Image Scene. |
|---|---|
| Authors: | Ren, Yan1 (AUTHOR), Qian, Haizhong1,2 (AUTHOR), Jiang, Bingchuan1 (AUTHOR) jbc021@163.com, Li, Tingting1,2 (AUTHOR), Wang, Xiao1 (AUTHOR), Sun, Long2 (AUTHOR), Yang, Li2 (AUTHOR) |
| Source: | Remote Sensing. Jun2026, Vol. 18 Issue 12, p1959. 28p. |
| Subjects: | Remote sensing, Prompt engineering, Machine learning |
| Abstract: | Highlights: What are the main findings? A Unified Visual-Semantic Triple Prompt Learning (UVSTPL) framework is proposed, which realizes effective fusion of visual and text modalities to enhance remote sensing scene spatial relationship reasoning. A novel Geo-RSSG dataset with detailed ground object annotations, precise spatial relationships, and rich attributes is constructed, and UVSTPL achieves state-of-the-art performance on this dataset. What are the implications of the main findings? The UVSTPL framework promotes the integration of vision-language multimodal learning and GeoAI, providing a new approach for intelligent interpretation of remote sensing data. The Geo-RSSG dataset bridges the gap in high-quality multimodal benchmarks for remote sensing reasoning tasks and provides a reliable foundation for future research in this field. A remote sensing scene graph (RSSG) enables machines to interpret interactions among ground objects in remote sensing images and supports semantic reasoning and description, thus making it a fundamental technique in the field. However, most existing scene reasoning approaches cannot fully utilize multimodal information, resulting in limited performance when inferring spatial relationships among ground objects. To this end, we propose a Unified Visual-Semantic Triple Prompt Learning (UVSTPL) framework, which integrates visual features with matched geospatial object labels, leverages a prompt learning module for multimodal feature extraction, and employs a refined UVTransE model to predict spatial relationships. The core principle of UVSTPL is to enhance semantic feature extraction and improve relationship prediction performance via the collaborative fusion of visual and linguistic modalities. To strengthen the model's ability to reason about the spatial relationships among ground objects in images, a novel Geo-RSSG dataset is constructed, which includes precise annotations of geographic entities, spatial relationships, and attributes. Extensive experiments demonstrate that the proposed UVSTPL method outperforms benchmark models on the spatial relationship prediction task. In comparison with the best baseline method, our approach improves prediction precision by 1.85%, mean precision by 8.49%, mean recall by 17.46%, and mean F1-score by 12.97%. This study offers valuable insights for advancing the understanding and cognitive capabilities of remote sensing scenes. [ABSTRACT FROM AUTHOR] |
| Copyright of Remote Sensing is the property of MDPI and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | Highlights: What are the main findings? A Unified Visual-Semantic Triple Prompt Learning (UVSTPL) framework is proposed, which realizes effective fusion of visual and text modalities to enhance remote sensing scene spatial relationship reasoning. A novel Geo-RSSG dataset with detailed ground object annotations, precise spatial relationships, and rich attributes is constructed, and UVSTPL achieves state-of-the-art performance on this dataset. What are the implications of the main findings? The UVSTPL framework promotes the integration of vision-language multimodal learning and GeoAI, providing a new approach for intelligent interpretation of remote sensing data. The Geo-RSSG dataset bridges the gap in high-quality multimodal benchmarks for remote sensing reasoning tasks and provides a reliable foundation for future research in this field. A remote sensing scene graph (RSSG) enables machines to interpret interactions among ground objects in remote sensing images and supports semantic reasoning and description, thus making it a fundamental technique in the field. However, most existing scene reasoning approaches cannot fully utilize multimodal information, resulting in limited performance when inferring spatial relationships among ground objects. To this end, we propose a Unified Visual-Semantic Triple Prompt Learning (UVSTPL) framework, which integrates visual features with matched geospatial object labels, leverages a prompt learning module for multimodal feature extraction, and employs a refined UVTransE model to predict spatial relationships. The core principle of UVSTPL is to enhance semantic feature extraction and improve relationship prediction performance via the collaborative fusion of visual and linguistic modalities. To strengthen the model's ability to reason about the spatial relationships among ground objects in images, a novel Geo-RSSG dataset is constructed, which includes precise annotations of geographic entities, spatial relationships, and attributes. Extensive experiments demonstrate that the proposed UVSTPL method outperforms benchmark models on the spatial relationship prediction task. In comparison with the best baseline method, our approach improves prediction precision by 1.85%, mean precision by 8.49%, mean recall by 17.46%, and mean F1-score by 12.97%. This study offers valuable insights for advancing the understanding and cognitive capabilities of remote sensing scenes. [ABSTRACT FROM AUTHOR] |
|---|---|
| ISSN: | 20724292 |
| DOI: | 10.3390/rs18121959 |
