Three-stream fusion networks for student engagement recognition based on TimeSformer

Yan Wang; JinWei Wang; Xue Bai

doi:10.1117/12.2671122

28 April 2023 Three-stream fusion networks for student engagement recognition based on TimeSformer

Yan Wang, JinWei Wang, Xue Bai

Proceedings Volume 12610, Third International Conference on Artificial Intelligence and Computer Engineering (ICAICE 2022); 1261037 (2023) https://doi.org/10.1117/12.2671122
Event: Third International Conference on Artificial Intelligence and Computer Engineering (ICAICE 2022), 2022, Wuhan, China

Abstract

Students' engagement is an essential indicator of students' participation in learning. This paper proposes an end-to-end three-stream fusion network students' engagement recognition model based on Spatio-temporal self-attention network. The model comprises three parallel TimeSformer networks, which extract facial information features, original video features, and video features after portrait segmentation. The calculation results of the three networks are fused for decision-making, and finally, the student's engagement level of the whole video is obtained. We trained and tested our model on the students' engagement dataset DAiSEE and obtained 57.8% four-level ACC. The recognition performance is significantly higher than the baseline of the database and better than other existing deep network models. The experimental results show that the accuracy of students' engagement recognition is effectively improved based on TimeSformer three-stream fusion network.

Citation Download Citation

Yan Wang, JinWei Wang, and Xue Bai "Three-stream fusion networks for student engagement recognition based on TimeSformer", Proc. SPIE 12610, Third International Conference on Artificial Intelligence and Computer Engineering (ICAICE 2022), 1261037 (28 April 2023); https://doi.org/10.1117/12.2671122

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
7 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Video

Video processing

Image segmentation

Feature extraction

Facial recognition systems

RELATED CONTENT

Person based video summarization and retrieval by tracking and clustering...
Proceedings of SPIE (March 21 2013)

Fusion of visual and audio features for person identification in...
Proceedings of SPIE (January 01 2001)

Bayesian-network-based soccer video event detection and retrieval
Proceedings of SPIE (September 25 2003)

ViBE: a video indexing and browsing environment
Proceedings of SPIE (August 24 1999)

Implementation of flexible-playtime video skimming
Proceedings of SPIE (December 15 2003)

Automatic home video abstraction using audio contents
Proceedings of SPIE (August 30 2002)

Motion based situation recognition in group meetings
Proceedings of SPIE (January 28 2010)

Subscribe to Digital Library

Receive Erratum Email Alert

Keywords/Phrases

Search In:

Publication Years