2.1

Model overview

The overall structure of our proposed method is shown in Figure 1.

Figure 1.

An overview of the proposed model.

Firstly, the feature representation module is used to obtain sentence-level vectors of audio, text, and video. Then the feature fusion module based on cross-modal Transformer is adopted to combine three modalities. Next, a Bi-LSTM with selfattention is used to capture the temporal relationship of each single modality. Finally, the outputs of the self-attention module are fed into the low-rank fusion module for further late fusion to get the final regression result.

2.2

Feature extraction

The dataset consists of three modalities, namely text, video and audio features. For the text data, we use a pre-trained 1024-dimensional Elmo⁵ sentence embedding to encode the transcribed subjects’ responses for each question. For the audio data, 39-dimensional frame-level MFCC features are extracted for each audio segment. For the video data, 20-dimensional frame-level AU features are extracted for each video segment.

2.3

Unsupervised autoencoder

The extracted video and audio features are both frame-level. Temporal aggregations are needed to compress sentence-level vectors before feature fusion. Most previous methods encode the frame-level features into sentence-level with statistical functions which may lead to the loss of the temporal information between frames. To solve this problem, we propose an unsupervised autoencoder based on Transformer⁶. The overall structure of the autoencoder is shown in Figure 2.

Figure 2.

The structure of autoencoder.

The frame-level features are fed into the frame-sentence encoder to obtain the sentence-level vector of multi-frame features. The frame-sentence encoder consists of three layers transformer encoding units. The transformer encoding unit is illustrated in Figure 3.

Figure 3.

The structure of transformer.

The Positional Encoding module is used to add positional information to the frame-level features X. The specific calculation formula of positional embedding can be expressed by equations (1) and (2).

Here pos is the frame number, F is the dimension of the frame-level feature, I is the index of the features, and 2i represents the even index while 2i + 1 represents the odd index.

The frame-level features X are then added to the positional embedding PE to generate the input vector I. Then the Multi-Head Attention module is used to calculate the temporal relationship between frames. The relationship between vector A and input vector I can be expressed by

where W_Q, W_K, W_V denote linear matrixes,Q, K, V denote the query vector, key vector, and value vector, respectively, and d_k is the normalization coefficient.

Then vector A passes through residual connections and forward neural networks to obtain the output vector O. O is fused with the temporal and attention relationship between multi-frame features. The vector O of the last time step is the output of the frame-sentence encoder. After self-filling, a vector X′ is passed through the sentence-frame decoder to reconstruct the frame-level features. The decoder is also composed of 3-layer Transformer coding units. The output vector Y of the decoder has the same dimension as the frame-level features X. The loss function of the unsupervised autoencoder is mean square error, and the difference between X and Y is calculated to update weights of the unsupervised autoencoder. When the model is converged, the output of the encoder is stored as a sentence-level vector representation of frame-level features.

So far, we have obtained three sentence-level vectors X_t, X_a, X_v of text, audio, and video modality respectively. Before the feature fusion module for deep fusion, three one-dimensional convolutional layers are utilized to compress the dimensions of the three sentence-level vectors. Then, X_T ∈ R^S×30, X_A ∈ R^S×30, X_V ∈ R^S×30 are fed into the feature fusion module for deep fusion, where S is the question number of each subject and 30 is the dimension of compressed features.

2.4

Feature fusion

Six cross-modal transformers⁷ are adopted to achieve a deep fusion of three modalities. We take A → T cross-modal transformer as an example to introduce the application of cross-modal transformer in the feature fusion module. The structure of A → T cross-modal transformer can be seen in Figure 4.

Figure 4.

The structure of cross-modal transformer.

The relationship between the output of the i_th A → T cross-modal transformer Z_A→T^[i] and the input vector X_T, X_A can be represented by

where Positional Encoding is used to add positional information to X_T and X_A. The specific relationship can be obtained from equations (1) and (2). LayerNorm is the layer normalization operation. Z_A→T^[i–1] is the output of (i – 1)_th layer cross-modal transformer. denote linear matrixes. And Q_T, K_A, V_A denote query vector of text modality, key vector, and value vector of audio modality. d_k is the normalization coefficient. FFN is the forward neural network which is composed of linear layers.

The outputs of the cross-modal transformer are concatenated to generate Z_T ∈ R^S×60, Z_A ∈ R^S×60, Z_V ∈ R^S×60. Z_T, Z_A, Z_V deeply fuse the information of the other two modalities and are then fed into the self-attention module to capture the temporal information of each modality.

2.5

Self-attention

Bi-LSTM with attention mechanism is utilized to capture the temporal relationship of Z_T, Z_A, and Z_V. Taking Z_T for example, the Bi-LSTM model with a hidden size of 30 is used to capture the temporal relationship between each time step of Z_T. Considering that features at different time steps are of different importance to the final assessment result, an attention mechanism is introduced to weight the output of Bi-LSTM at different time steps. The relationship between the output of self-attention module S_T and the output of Bi-LSTM Z^°_T can be represented by

where w is the weighting factor and T is the question number.

Then S_T is passed through a ReLU layer and a linear layer to obtain the output O_T of self-attention module. Finally, three temporal vectors O_T, O_V, O_A are passed through the late fusion module for further fusion to obtain the final assessment result.

2.6

Late fusion

The low-rank fusion⁸ is used to further fuse O_T, O_A and O_V. The relationship between the final output H and O_T, O_V, and O_A can be represented by

where z_l, z_v, and z_a are the vectors O_T, O_V and O_A padding with 1, M is the number of modality 3, denotes the element-wise product over a sequence of tensors, and W is the weight tensor. And the value of r is 17, which is the number of rank factors.