Multi-visual information fusion and aggregation for video action classification

Xuchao Gong; Zongmin Li; Xiangdong Wang

doi:10.1117/12.2644312

12 October 2022 Multi-visual information fusion and aggregation for video action classification

Xuchao Gong, Zongmin Li, Xiangdong Wang

Proceedings Volume 12342, Fourteenth International Conference on Digital Image Processing (ICDIP 2022); 123421Z (2022) https://doi.org/10.1117/12.2644312
Event: Fourteenth International Conference on Digital Image Processing (ICDIP 2022), 2022, Wuhan, China

Abstract

In order to fully mine the performance improvement of spatio-temporal features in video action classification, we propose a multi-visual information fusion time sequence prediction network (MI-TPN) which based on the feature aggregation model ActionVLAD. The method includes three parts: multi-visual information fusion, time sequence feature modeling and spatiotemporal feature aggregation. In the multi-visual information fusion, the RGB features and optical flow features are combined, the visual context and action description details are fully considered. In time sequence feature modeling, the temporal relationship is modeled by LSTM to obtain the importance measurement between temporal description features. Finally, in feature aggregation, time step feature and spatiotemporal center attention mechanism are used to aggregate features and projected them into a common feature space. This method obtains good results on three commonly used comparative datasets UCF101, HMDB51 and Something.

Citation Download Citation

Xuchao Gong, Zongmin Li, and Xiangdong Wang "Multi-visual information fusion and aggregation for video action classification", Proc. SPIE 12342, Fourteenth International Conference on Digital Image Processing (ICDIP 2022), 123421Z (12 October 2022); https://doi.org/10.1117/12.2644312

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available