2.1

Traditional hand-crafted features

Generally speaking, since anomaly events mainly involve two aspects of appearance and movement, in such methods, we need to extract deep features that can represent the appearance and motion of the image. Mehran et al.¹ represented objects in video frames with particle flow and introduced social forces to model the objects. Martineld et al.⁴ proposed that the target motion trajectories in the video dataset are clustered, and a clustered trajectory tree that can estimate the motion path is constructed. Hasan et al.⁵ used trajectory-like HOG and HOF features, concatenated multiple frame features in the time axis, and abnormal event detection is done using the reconstruction error between the output of the autoencoder and the ground truth.

2.2

Deep learning method

Sabokrou et al.⁶ used the fully convolutional layer of AlexNet to extract the deep features of the input video, and sending the features to cascaded Gaussian classifiers for anomaly detection. Li et al.⁷ introduced optical flow network to better predict temporal information, and anomaly detection through motion consistency. Alahi et al.⁸ proposed an unsupervised learning method to generate multiple LSTM networks for each pedestrian in a specific time frame, and predict the pedestrian’s position at the current moment based on the pedestrian’s historical position. Nguyen et al.¹⁰ improved the classic U-Net network by changing it to a combination of a reconstruction network and a generative adversarial network, and by introducing the idea of generative adversarial to generate more realistic predicted frame images. Vitjan et al. ¹¹ proposed an image inpainting method with a variant of the U-Net network, which can reconstruct the image appearance well, but lacks the participation of motion information.

3.

PROPOSED METHOD

Since abnormal event detection is the identification of behavior that does not conform to the expected behavior of a video scene, we propose to use image repair reconstruction for abnormal event detection, because the masked normal video frames have sufficient information to be successfully reconstructed, but the network cannot reconstruct well frames with anomaly events. However, we know that both temporal and spatial information are very important features. The above image repair reconstruction only considers appearance constraints, but appearance constraints can only represent spatial information, so we also add motion constraints to the objective function. Specifically, the optical flow network is introduced to ensure the temporal information of normal events, thereby improving the abnormal event detection performance. An overview of our model is shown in Figure 1. The model consists of two parts. The first part treats abnormal event detection as a repair reconstruction problem and utilizes the proposed appearance repair-based autoencoder network for multi-scale image repair to learn the appearance space structure in normal events. The second part learns the motion (temporal) structure of normal events through our proposed motion-consistent optical flow network.

Figure 1.

An overview of our network framework.

3.1

Appearance repair based autoencoder

We need to randomly sample some small regions in each frame of the input image to perform the deletion operation, that is, the randomly selected regions are set to zero in the input image and repaired by the trained network. Specifically, each frame of the input image is divided into a set of square grids of size k × k, then select several square grids to remove.

The accuracy of our image reconstruction depends on the size of the area masked during the testing phase. Since the effect of abnormal event detection depends on the similarity between the reconstructed image and the input image, the abnormal event detection performance depends on the ratio of the size k of the masked region and the size of the anomaly regions. If k is much larger than the size of the abnormal region, it cannot be reconstructed accurately; If k is too small, the reconstruction network is able to reconstruct anomalies well from neighboring regions. Because there are anomalies of different sizes, their detection must consider multiple scales. A more reliable reconstruction error map can be generated by considering multiple reconstructions of a single image generated using multiple values of k.

Our proposed appearance repair based autoencoder network is a variant of U-net, and the network architecture used is shown in Figure 2. Our encoder and decoder have a symmetric network structure, both consisting of a series of blocks. Each block contains Convolution layer, BatchNorm layer and Relu activation, skip connections are used to transfer features through different layers, thereby accurately reconstructing details.

Figure 2.

Appearance repair based autoencoder.

Our appearance repair based autoencoder learns common appearance (spatial) patterns of normal events, and our network takes the ℓ₂ distance between the ground truth I and the output of the reconstructed network as an intensity loss:

The disadvantage of using only the above strength loss is blurring in the output, so we add a gradient constraint, gradient loss is defined as:

where g_i represents the image gradient along the i-axis.

3.2

Optical flow network based on motion consistency

We construct an optical flow network, and the output frame of the appearance autoencoder and the actual frame are respectively input into the optical flow network to calculate the motion loss of the network. Our optical flow network is a variant of Flownet, which is a combination of multiple layers of convolutional neural networks. The deep features of the last four convolutional output layers are extracted in our framework to compute the network motion loss.

The motion loss function of the optical flow network is as follows:

where j represents the number of network layers, and C, H and W represent the channel number, length, and width of the corresponding convolutional layer, respectively; φ_j (p) is the output of the j-th output convolutional layer.

4.1

Abnormal event detection

An example of image restoration and reconstruction performed by our method on three benchmark datasets is shown in Figure 3. It can be seen that our method can restore the input image well.

Figure 3.

Experimental examples on three benchmark datasets.

In Table 1, we show our method achieves competitive performance compared to other popular methods. On the CUHK Avenue dataset, our anomaly event detection framework achieves the best frame-level AUC data of 85.5%; Our method is the only two methods that exceed 90% on UCSD ped1; Our detection network achieves the highest frame-level AUC of 95.9% on the UCSD ped2 dataset. In summary, our method has competitive performance compared to state-of-the-art methods.

Table 1.

AUC of different methods on the Avenue, Ped1, Ped2 datasets.

4.2

Ablation studies

In this section, we conduct experiments to explore the effect of different target constraints on the detection results of abnormal events on three benchmark datasets by removing different constraints.

Qualitative evaluation of motion constraints: Figure 4 shows the optical flow graph generated with/without motion constraint, we can see that the optical flow produced by the anomalous event detection network with motion constraints is more in line with reality. This shows that considering temporal information can help the network to better generate motion consistency and enhance the ability to detect abnormal events.

Figure 4.

Optical flow differences with and without motion constraints, input images and reconstructed images.

We remove different constraints for quantitative analysis; Furthermore, we conduct experiments to verify the impact of different objective loss functions on our anomaly event detection network, and our experimental data are presented in Table 2. We can observe that all three constraints can improve the abnormal event detection ability of our abnormal detection framework. The more constraints, the higher the frame-level AUC, and the better the abnormal event detection performance.

Table 2.

Experimental results of ablation of different losses on CUHK Avenue data