Deep learning has achieved great success in computer vision, natural language processing, recommendation systems and other fields. However, the models of deep neural network (DNN) are very complex, which often contain millions of parameters and tens or even hundreds of layers. Optimizing weights of DNNs is easy to fall into local optima, and hard to achieve better performance. Thus, how to choose an effective optimizer which is able to obtain network with higher precision and stronger generalization ability is of great significance. In this article, we make a review of some popular historical and state-of-the-art optimizers, and conclude them into three main streams: first order optimizers that accelerate convergence speed of stochastic gradient descent or/and adaptively adjust learning rates; second order optimizers that can make use of second-order information of loss landscape which helps escape from local optima; proxy optimizers that are able to deal with non-differentiable loss functions through combining with the proxy algorithm. We also summarize the first and second order moment used in different optimizers. Moreover, we provide an insightful comparison on some optimizers through image classification. The results show that first order optimizers like AdaMod and Ranger not only have low computational cost, but also show great convergence speed. Meanwhile, the optimizers that can introduce curvature information such as Adabelief and Apollo, have a better generalization especially when optimizing complex network.
Convolution Neural Networks (CNN) have evolved to be the state-of-art technique for machine learning tasks. However, CNNs bring a significant increase in the computation and parameter storage costs, which makes it difficult to deploy on embedded devices with limited hardware resources and a tight power budget. In recent years, people focus on reducing these overheads by compressing the CNN models, such as pruning weights and pruning filters. Compared with the method of pruning weights, the method of pruning filters does not result in sparse connectivity patterns. And it is conducive to the parallel acceleration on hardware platforms. In this paper, we proposed a new method to judge the importance of filters. In order to make the judgement more accurate, we use the standard deviation to represent the amount of information extracted by the filter. In the process of pruning, the unimportant filters can be removed directly without loss in the test accuracy. We also proposed a multilayer pruning method to avoid setting the pruning rate layer by layer. This holistic pruning method can improve the pruning efficiency. In order to verify the effectiveness of our algorithm, we do experiments with simple network VGG16 and complex networks ResNet18/34. We re-trained the pruned CNNs to compensate the accuracy loss caused by the pruning process. The results showed that our pruning method can reduce inference cost by up to 50% for VGG16 and 35% for ResNet18/34 on CIFAR10 with little accuracy loss.
In this study, this paper focus on the vision-based autonomous helicopter unmanned aerial vehicle (UAV) landing problems. This paper proposed a multisensory fusion to autonomous landing of an UAV. The systems include an infrared camera, an Ultra-wideband radar that measure distance between UAV and Ground-Based system, an PAN-Tilt Unit (PTU). In order to identify all weather UAV targets, we use infrared cameras. To reduce the complexity of the stereovision or one-cameral calculating the target of three-dimensional coordinates, using the ultra-wideband radar distance module provides visual depth information, real-time Image-PTU tracking UAV and calculate the UAV threedimensional coordinates. Compared to the DGPS, the test results show that the paper is effectiveness and robustness.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.