A multimodal fusion human activity recognition method based on CNN and Gramian angular field

Qirui Wu; Ying Li; Hongyu Ouyang; Yundong Xuan

doi:10.1117/12.3004953

9 October 2023 A multimodal fusion human activity recognition method based on CNN and Gramian angular field

Qirui Wu, Ying Li, Hongyu Ouyang, Yundong Xuan

Proceedings Volume 12791, Third International Conference on Advanced Algorithms and Neural Networks (AANN 2023); 127911I (2023) https://doi.org/10.1117/12.3004953
Event: Third International Conference on Advanced Algorithms and Neural Networks (AANN 2023), 2023, Qingdao, SD, China

Abstract

In order to solve the problems of complex and time-consuming traditional manual feature engineering methods in the field of human activity recognition, incomplete grasp of local information by a single recurrent neural network (RNN) and lack of utilization of temporal information by a single convolutional neural network (CNN), this paper uses the Gramian Angular Field (GAF) algorithm to convert inertial sensor data into images and designs a multimodal fusion based on CBAM-CNN-Attention method, which can effectively improve the human activity recognition accuracy. Firstly, the time series and image features are extracted using one-dimensional convolutional kernel and two-dimensional convolutional kernel, respectively. Secondly, the most important features are obtained and optimized using the attention mechanism. Finally, the recognition results are classified using a softmax classifier. The experimental results on the UCI-HAR dataset show that our proposed method has superior performance to other methods that use only data collected by accelerometers.

(2023) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Qirui Wu, Ying Li, Hongyu Ouyang, and Yundong Xuan "A multimodal fusion human activity recognition method based on CNN and Gramian angular field", Proc. SPIE 12791, Third International Conference on Advanced Algorithms and Neural Networks (AANN 2023), 127911I (9 October 2023); https://doi.org/10.1117/12.3004953

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available