Super-resolution (SR), which aims at recovering a high-resolution (HR) image from single or sequential low-resolution (LR) images, is a widely used technology in image processing. In the field of image SR, convolutional neural networks (CNNs) have attracted increasing attention because of their high-quality performance. However, most CNN-based methods treat each channel-wise feature equally, which lacks discriminative learning ability across feature channels. Furthermore, many methods neglect to fully use information of each convolutional layer. To resolve these problems, we propose a remote sensing image SR method named dense channel attention network (DCAN). In our DCAN, a sequence of residual dense channel attention blocks (RDCABs) is cascaded with a densely connected structure. In each RDCAB, we make full use of the information from all convolution layers via densely connected convolutional layers. In addition, RDCABs utilize the channel attention mechanism to adaptively recalibrate channel-wise feature responses by explicitly modeling the interdependencies between the channels. In addition, our DCAN can make full use of the hierarchical features by densely connecting each RDCAB. Finally, to further improve the SR performance, the proposed DCAN is learned in both the pixel and wavelet domains, and a fusion layer is used to fuse the outputs of these two domains. Extensive quantitative and qualitative evaluations verify the superiority of our proposed method over several state-of-the-art methods.

We present a method for image denoising based on singular value shrinkage that fuses soft and hard thresholds. The technique simply groups similar patches from a noisy image as low-rank matrices and shrinks the singular values by the combination of soft and hard thresholds. On one hand, a hard threshold approximation method based on nonlocal self-similarity and low-rank approximation is used for fast selection of hard threshold; on the other hand, a soft threshold selection method based on random matrix and asymptotic matrix reconstruction theory is designed. In addition, we also propose an adaptive backward projection algorithm based on image phase congruency and gradient calculation so that the input images participating in the iteration are adaptive. This method improves the traditional fixed coefficient backward projection method and makes the robustness of the algorithm better. The experimental results of denoising and enhancement for a number of natural images show that the proposed algorithms have significant improvement in both subjective visual effect and objective quantization index by comparing with some related state-of-the-art denoising algorithms.

The fisheye-type camera is widely used since it has a larger view than that of a pinhole camera. But its applications have frequently been tortured by its higher distortion in imaging procedure. We propose a strategy for fisheye camera modeling based on the receptive field weighted regression algorithm. The camera’s field of view is adaptively divided into multiple grids, each of which is described by an actively learned receptive field affiliated by a regression model to denote specific imaging relation in this grid with prescribed precision. The whole model of the camera is thus achieved by weighted regressions overall receptive fields on grids. Moreover, the proposed strategy could also be applied to the simultaneous calibration of a visual system of multiple cameras since the final sensing model stems from all receptive fields over any of the camera’s field of view with the unified weighted regression scheme. Experiments on monocular and multicamera systems validate the feasibility of the proposed strategy, with performance comparisons with a state-of-the-art method.

Anomaly detection algorithms based on deep neural networks have achieved favorable performance in finding abnormal events in surveillance video. Recently, end-to-end methods that combine feature extraction, model learning, and anomaly scoring into one training procedure have become popular. However, most existing research studies have relied on a deep convolutional structure, which faces overfitting problems for a limited training set. An anomaly detection algorithm based on the state-of-the-art prediction framework is proposed, leveraging the gap between frame prediction and its ground truth to detect abnormal events. The residual block is transferred from image classification, and we modify its modules to suit the prediction application in order to tackle the difficulties in training a deeper prediction network. As far as we know, the proposed method is the first anomaly detection residual network trained from scratch, which is different from several existing ones with fixed resnet-50 layers as feature extractor. Furthermore, a new perceptual constraint focusing on high-level information is proposed and combined with the commonly used spatial–temporal constraints. Experimental results on challenging public surveillance sequences verify that our proposed framework can effectively produce state-of-the-art performance.

Recently, the correlation filter (CF)-based methods have achieved great success in the field of object tracking. In most of these methods, the CF utilizes L₂ norm as the regularization, which does not pay attention to the stability and robustness of the feature. However, there may exist some unstable points in the image because the object in the video may have different appearance changes. We propose a tracking method based on a structured robust correlation filter (SRCF), which employs the L_2,1 norm as the regularization. The robust CF can not only retain the accuracy from the regression formulation but also take into account the stability of the image region to improve the robustness of the appearance model. The alternating direction method of multipliers algorithm is used to solve the L_2,1 optimization problem in SRCF. Moreover, the multilayer convolutional features are adopted to further improve the representation accuracy. The proposed method is evaluated in several benchmark datasets, and the results demonstrate that it can achieve comparable performance with respect to the state-of-the-art tracking methods.

We propose an architecture to automatically detect widgets in mobile screenshots, considering only visual cues. Even though traditional object detection methods perform well on common objects in natural scene images, they are unable to deal with the screenshot images with complex widget layout. Therefore, we propose region-based Widget Detection Network (WDN), which introduces regularities in the screenshot images as the regularizations. First, we design a scale-aware attention structure to make the backbone network sensitive to widget scales so that the salient features of the interest regions could be captured. Second, a strategy of horizontal region generation is proposed to fully utilize the aligned property of widget arrangement, which generates all the region candidates in a horizontal line at once. Finally, a variant of online hard example mining is employed to alleviate the problem of imbalance samples, which explicitly restricts the ratio of foreground and background to achieve better balance. We conduct experiments on a proposed benchmark dataset. The quantitative results and qualitative analysis on the benchmark dataset show that WDN achieves impressive performance, which outperforms the common object detection methods in the widget detection task.

Semantic segmentation, as a dense pixelwise classification task, is of great significance to scene understanding. Many approaches based on convolutional neural network still suffer from two kinds of challenges: (1) insufficient semantic information results in semantic obfuscation between similar categories, (2) loss of spatial information leads to inaccurate location of inconspicuous objects. To tackle these challenges, we design a network with an encoder–decoder architecture based on two proposed modules: global pyramid attention module (GPAM) and pyramid decoder module (PDM). Specifically, GPAM exploits an attention mechanism as global prior knowledge to adaptively capture discriminative features for enhancing semantic representation, and PDM employs small convolutions connected in parallel to predict adjacent position relationships for refining spatial information. A series of ablation experiments are conducted to demonstrate the effectiveness of our designs, and our network achieves a mean intersection over union score of 83.4% on PASCAL VOC 2012 dataset and 78.5% on Cityscapes dataset.

Recently, many researchers have studied fiducial-based tracking of deformable nonrigid surfaces for projection mapping purposes, but efficient handling and representation of fine deformations and sharp contours in such surfaces still remain a challenge. On top of existing fiducial-based projection mapping methods, much effort is still required to achieve a projection mapping that is free from geometric errors, misalignments, and artifacts in the areas that contain such deformations. According to our observation, this is due to the fact that methods relying on fiducial markers have used markers that sparsely represent the projection surface, are large in size, and projector–camera registration is not accurate enough. We propose a method that enables us to accurately track a deformable nonrigid surface and project textures onto it, ensuring that no visible misalignments occur. We achieve this by proposing a fiducial marker design followed by an adaptive method for fiducial selection to adequately model the deformations in a nonrigid projection surface. Our proposed marker is scalable in size and provides several advantages, such as robust detection under strong deformations and support for adaptively varying fiducial marker density, which enables us to handle the aforementioned deformations with increased accuracy. We have demonstrated our results using a sheet of paper and a cloth under several geometric deformations. Our experimental results show compelling results in terms of geometric accuracy.

We propose a coarse-to-fine method automatically detecting facial landmarks for both rigid and nonrigid facial deformations. For an input 3-D face, first, we roughly detect the nose by employing 3-D local shape descriptors and further detect the tip of the nose with the tip features, e.g., the symmetry of the face. Second, we localize the eyes and mouth according to the distribution of human facial features and use a convolutional neural network to minimize the combined loss and to provide candidates for the corners of the eyes and mouth. Finally, for accurately detecting the landmarks of the eyes and mouth, we iteratively update the candidates by maximizing the similarity of the candidates and the landmarks based on the features of the candidate and its neighbors. Over Bosphorus and CASIA datasets, we evaluate the proposed method. Experiments show that compared with the state-of-the-art methods, our method detects the corners of eyes and mouth more accurately and robustly.

It is difficult to extract appropriate features of blurry digital video because of the low quality image for which traditional forgery detection methods are not effective. The video is often blurred due to various complicated conditions, such as limited bandwidth or low pixel complementary metal oxide semiconductor sensors. To address the feature extraction problem for blurry video, we explore one low rank theory to deblur video, which fuses multiple fuzzy kernels of key frame images by low rank decomposition. Further, we extract mean structural similarity features and information fidelity criterion features sequentially by the double detection mechanism to detect forgery points on blurred video frames. We can determine the forgery form of the video and locate the source replication area of the copy-move forgery. Experimental evaluation on two public video databases and videos that we recorded show that this method is robust in blurry video forgery detection, and compared with traditional video forgery detection methods, the efficiency is improved.

We proposed a vision-based methodology as an aid for an unmanned aerial vehicle (UAV) landing on a previously unsurveyed area. When the UAV was commanded to perform a landing mission in an unknown airfield, the learning procedure was activated to extract the surface features for learning the obstacle appearance. After the learning process, while hovering the UAV above the potential landing spot, the vision system would be able to predict the roughness value for confidence in a safe landing. Finally, using hybrid optical flow technology for motion estimation, we successfully carried out the UAV landing without a predefined target. Our work combines a well-equipped flight control system with the proposed vision system to yield more practical versatility for UAV applications.

Event sensors output a stream of asynchronous brightness changes (called “events”) at a very high temporal rate. Previous works on recovering the lost intensity information from the event sensor data have heavily relied on the event stream, which makes the reconstructed images nonphotorealistic and also susceptible to noise in the event stream. A method is proposed to reconstruct photorealistic intensity images from a hybrid sensor consisting of a low-frame-rate conventional camera and the event sensor. The texture-rich information from a traditional image sensor with the motion-rich information from the event sensor is exploited, producing photorealistic high-frame-rate videos. To accomplish the task, the low-frame-rate intensity images are warped to temporally dense locations of the event data. The results obtained from the proposed algorithm are more photorealistic compared to any of the previous state-of-the-art algorithms. The algorithm’s robustness to abrupt camera motion and noise in the event sensor data is also demonstrated.

Correlation filter (CF) has been widely used in visual tracking tasks due to its simplicity and high efficiency. However, the conventional CF-based trackers suffer from boundary effects and object scale changes. To address these problems, we propose an adaptive spatially regularized CF tracking method via level set segmentation (LSS). First, a fast LSS method based on region and boundary information is proposed to accurately estimate the object region. Then an adaptive spatial reliability map is constructed using the estimated object region to regularize the CF, which can significantly mitigate the boundary effects. Meanwhile, the estimated object region is also utilized to construct a series of candidate scales whose aspect ratio is variable, and a novel scale variation (SV) constraint term is proposed to restrain the abnormal scale change. Finally, the position and scale of the object is estimated by maximizing the final response, which combines the CF response, and the histogram response is calculated quickly in frequency domain with the SV constraint term. Experimental results on recent visual tracking benchmark datasets illustrate that the tracking capability of the proposed method is competitive against several state-of-the-art trackers.

Glare is always present in optical acquisition systems, such as photocameras or the eye-bulb. As a consequence, images captured by sensors do not represent an accurate reproduction of the scene, but rather a combination of scene content and glare. We discuss the reasons why this unwanted addition of spread light cannot be removed from an acquired image. To this aim, we cast the problem of glare-removal into an estimation task and focus on the aspects that make the unfolding of glare an ill-posed or ill-conditioned problem—such as nonlinearity, information loss, or eye model uncertainty. For each mechanism of glare formation, we point to the corresponding influence in terms of ill-posedness and ill-conditioning of the problem. We do not aim at proposing or reviewing solutions to the glare problem but rather at identifying more precisely the challenges it poses.

Monocular visual odometry and depth estimation play an important role in augmented reality and robot applications. Recently, deep learning technologies have been widely used in these areas. However, most existing works utilize supervised learning, which requires large amounts of labeled data and assumes that the scene is static. We propose a framework, called Un-VDNet, based on unsupervised convolutional neural networks to predict camera ego-motion and depth maps from image sequences. The framework includes three subnetworks (PoseNet, DepthNet, and FlowNet) and learns temporal motion and spatial association information in an end-to-end network. We propose a pose-consistency loss to penalize errors about the translation and rotation drifts of the pose estimated from the PoseNet. Furthermore, a geometric consistency loss, between the structure flow and scene flow learned from the FlowNet, is proposed to deal with dynamic objects in the real-world scene, which is combined with spatial and temporal photometric consistency constraints. Extensive experiments on the KITTI and TUM datasets demonstrate that our proposed Un-VDNet outperforms the state-of-the-art methods for visual odometry and depth estimation in dealing with dynamic objects of outdoor and indoor scenes.

Camera viewpoint has a great influence on reidentification (re-ID) for its diversity. However, there are few methods in response to it, which limits the development of the re-ID community. We analyze the influence of camera viewpoint on re-ID to obtain both visualization and quantitative results, indicating the potential to solve this problem. On this basis, we propose mean and variance loss function (MVL) and camera ID classifier (CIC). MVL reduces the gap among the mean and variance of the feature vectors of images acquired by different cameras, so that the distribution of image feature vectors under different cameras is similar. CIC filters out the images, which have the same camera id with the query to avoid recognition interference. The experiments are conducted on Market1501 and DukeMTMC-reID datasets. We show that MVL and CIC can mitigate the negative impact of the camera viewpoint for re-ID and further improve the accuracy of re-ID. In the Market1501 dataset, rank-1 [mean average precision score (mAP)] accuracy is improved from 88.1 (71.1) % to 90.0 (73.9) % for ResNet-50. In the DukeMTMC-reID dataset, rank-1 (mAP) accuracy is improved from 77.0 (58.9) % to 83.9 (64.1) % for ResNet-50.

Correlation filter (CF) has received increasing attention in online visual object tracking. By accelerating the correlation in the frequency domain, CF trackers have achieved superior performance. However, existing CF trackers have a common shortcoming in that tracking models are prone to drift due to error accumulation. To address this problem, we propose a two-stage cascaded framework for accurate object modeling. In the first stage, we propose a passive–aggressive correlation filter (PACF) tracker to reduce error accumulation. In the subsequent stage, an online refinement algorithm is used to calibrate the tracking model by exploiting both long-term and short-term cues. In order to achieve high efficiency, our scheme reuses the PACF tracking response in the following stage. Extensive experiments were conducted on both long-term and short-term visual tracking benchmarks. The experimental results demonstrate that our tracker outperforms the state-of-the-art online tracking schemes in both long-term and short-term settings. Finally, we present a comprehensive analysis to validate the efficacy of our proposed method.