Expression-assisted facial action unit detection through an attention mechanism and smooth class-weighted Loss

Zhongling Liu; Ziqiang Shi; Rujie Liu; Liu Liu; Xiaoyu Mi; Kentaro Murase

doi:10.1117/12.2631478

1 May 2022 Expression-assisted facial action unit detection through an attention mechanism and smooth class-weighted Loss

Zhongling Liu, Ziqiang Shi, Rujie Liu, Liu Liu, Xiaoyu Mi, Kentaro Murase

Author Affiliations +

Proceedings Volume 12171, Thirteenth International Conference on Signal Processing Systems (ICSPS 2021); 121710R (2022) https://doi.org/10.1117/12.2631478
Event: Thirteenth International Conference on Signal Processing Systems (ICSPS 2021), 2021, Shanghai, China

Abstract

The performance of facial action unit (AU) detection is often limited owing to the lack of annotated AU data and data imbalance. The data scarcity problem can be partially mitigated by abundant expression data, as AU detection and facial expression recognition (FER) are closely related. Accordingly, in this study, FER and AU detection are trained jointly in a multi-task learning framework, in which FER serves as an auxiliary task for AU detection by providing supplementary information. Meanwhile, we propose to use an attention gate unit between the two tasks to flexibly select valuable information from each other. To address the model bias issue caused by AU data imbalance, a smooth class-weighted loss is adopted to alleviate the dominance of negative AU classes. The best average F1-score obtained using our approach on the BP4D dataset is 63.5%, which is very close to the state-of-the-art performance and exceeds the single task baseline by 3.2%.

Citation Download Citation

Zhongling Liu, Ziqiang Shi, Rujie Liu, Liu Liu, Xiaoyu Mi, and Kentaro Murase "Expression-assisted facial action unit detection through an attention mechanism and smooth class-weighted Loss", Proc. SPIE 12171, Thirteenth International Conference on Signal Processing Systems (ICSPS 2021), 121710R (1 May 2022); https://doi.org/10.1117/12.2631478

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
7 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Gold

Data modeling

Convolution

Databases

Binary data

Facial recognition systems

Feature selection

Show All Keywords

Keywords/Phrases

Search In:

Publication Years