Self-attention mechanism fusion method for bi-modal images

Junqing Li; Jiongyao Ye

doi:10.1117/12.2659376

22 November 2022 Self-attention mechanism fusion method for bi-modal images

Junqing Li, Jiongyao Ye

Proceedings Volume 12475, Second International Conference on Advanced Algorithms and Signal Image Processing (AASIP 2022); 124750E (2022) https://doi.org/10.1117/12.2659376
Event: Second International Conference on Advanced Algorithms and Signal Image Processing (AASIP 2022), 2022, Hulun Buir, China

Abstract

Multimodal information fusion can extract the important feature information of each modality and realize the information complementation between the modalities, which is a research direction of considerable interest at present. Extracting texture information from visible images and extracting typical features from infrared images are the main roles of dual-modal image fusion. To address the problems of information loss and low clarity of current deep learning-based fusion methods, this paper proposes a method to fuse bimodal images by embedding a network with attention mechanism. Due to the problems of low resolution and noise in infrared images, this method specifically uses encoder structures with different depths for infrared and visible images to extract shallow and deep features of each image, and then passes them through the attention-based fusion network to obtain the fused feature maps. A one-dimensional convolutional model based on a local cross-channel interaction mechanism without dimensionality reduction is used to construct the channel attention module in the fusion network, thus reducing the network complexity and improving the overall performance. The final fused image in this method relies on the decoder structure to generate. Experiments show that the method in this paper can fuse bimodal information better, and has some visual improvement under subjective evaluation compared with the comparison algorithm, and has 7.82%, 9.46%, 19.14% and 13.85% improvement in various objective evaluation indexes compared with DeepFuse, DenseFuse, FusionGAN and GANMcC respectively.

Citation Download Citation

Junqing Li and Jiongyao Ye "Self-attention mechanism fusion method for bi-modal images", Proc. SPIE 12475, Second International Conference on Advanced Algorithms and Signal Image Processing (AASIP 2022), 124750E (22 November 2022); https://doi.org/10.1117/12.2659376

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
9 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Image fusion

Infrared imaging

Visible radiation

Infrared radiation

Image processing

Computer programming

Feature extraction

Show All Keywords

Keywords/Phrases

Search In:

Publication Years