Image captioning using relevance attention and ITEM encoding

Hong Liang Zhang; Guang Ming Li

doi:10.1117/12.2624699

13 December 2021 Image captioning using relevance attention and ITEM encoding

Hong Liang Zhang, Guang Ming Li

Proceedings Volume 12087, International Conference on Electronic Information Engineering and Computer Technology (EIECT 2021); 120871Z (2021) https://doi.org/10.1117/12.2624699
Event: International Conference on Electronic Information Engineering and Computer Technology (EIECT 2021), 2021, Kunming, China

Abstract

The image caption task aims to generate a corresponding semantic description for an image. The current algorithm of generation captioning has these problems, that is insufficient information understanding in the process of generation image description and insufficient research of the relationship between image features. In order to solve these problems, we propose an image description method using the relevance attention mechanism and ITEM encoding. This model uses ITEM encoding to obtain text features containing contextual information and image information, so it can obtain richer semantic information. Then the relevant attention module is used to capture the relevant features which is a correlation between image features and text semantic information. Finally, the LSTM is used for decoding the relevant features to generate image caption. This paper conducts experiments to verify the effectiveness of the model on the MS COCO data set and Flickr 30k data set. The experimental results show that compared with the baseline model using visual attention, various evaluation indicators of our model have improved significantly.

Citation Download Citation

Hong Liang Zhang and Guang Ming Li "Image captioning using relevance attention and ITEM encoding", Proc. SPIE 12087, International Conference on Electronic Information Engineering and Computer Technology (EIECT 2021), 120871Z (13 December 2021); https://doi.org/10.1117/12.2624699

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
8 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Computer programming

Data modeling

Visualization

Information visualization

Image processing

Visual process modeling

Convolutional neural networks

Show All Keywords

Keywords/Phrases

Search In:

Publication Years