Multi-modal sentiment analysis based on expression and text

Wenjun Kao; Hai Yang; Chunling Liu; Jianguo Bai

doi:10.1117/12.3004687

9 October 2023 Multi-modal sentiment analysis based on expression and text

Wenjun Kao, Hai Yang, Chunling Liu, Jianguo Bai

Proceedings Volume 12791, Third International Conference on Advanced Algorithms and Neural Networks (AANN 2023); 127911J (2023) https://doi.org/10.1117/12.3004687
Event: Third International Conference on Advanced Algorithms and Neural Networks (AANN 2023), 2023, Qingdao, SD, China

Abstract

Sentiment analysis has always been a hot research topic in the field of natural language processing. With the development of artificial intelligence and big data technology, many data such as text, images, and videos that have personal emotional tendencies are accidentally generated. Initially, researchers mainly focused on studying text data, but over time, more and more people began to realize the limitations of single-mode analysis. By introducing multi-modal data such as images and videos, more dimensional information can be provided for sentiment analysis, thereby improving performance. Therefore, more and more researchers are exploring richer sources of emotional information to improve the effectiveness of sentiment analysis. This article proposes a multi-modal sentiment analysis method based on deep learning and designs an experimental model targeted towards video modal data. In terms of text, this article proposes a GloVe-based BiGRU word vector model to process data. For images, the method changes from using simple frame extraction to speaker anchoring, uses object detection technology to segment the region of interest for the subsequent model to extract image expression features. A pre-trained ResNet101 model is used to obtain vectors and generate a sequenced image matrix, which is then input to LSTM for processing. From the experimental results, it can be seen that this method performs better than existing methods in terms of accuracy and F1 scores.

(2023) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Wenjun Kao, Hai Yang, Chunling Liu, and Jianguo Bai "Multi-modal sentiment analysis based on expression and text", Proc. SPIE 12791, Third International Conference on Advanced Algorithms and Neural Networks (AANN 2023), 127911J (9 October 2023); https://doi.org/10.1117/12.3004687

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
5 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Data modeling

Image processing

Feature extraction

Image segmentation

Image fusion

Matrices

Analytical research

Show All Keywords

Keywords/Phrases

Search In:

Publication Years