Design of big data processing system optimization based on deep learning

Huanqin Wu; Fulin Li

doi:10.1117/12.3012256

21 December 2023 Design of big data processing system optimization based on deep learning

Huanqin Wu, Fulin Li

Proceedings Volume 12970, Fourth International Conference on Signal Processing and Computer Science (SPCS 2023); 129702J (2023) https://doi.org/10.1117/12.3012256
Event: Fourth International Conference on Signal Processing and Computer Science (SPCS 2023), 2023, Guilin, China

Abstract

In the process of optimizing the big data management system, the previous intelligent methods cannot be effectively calculated. This paper first introduces the research status of parallel optimization for Convolutional neural network model training, and explains some shortcomings. Aiming at the problems still existing in the parallel optimization of existing Convolutional neural network model training, an optimization processing method oriented to Convolutional neural network is proposed. The convolutional neural network model is optimized based on the distributed parallel processing framework. The results show that the comprehensiveness and rationality of the deep learning algorithm are greater than 83%. By using a multi parameter server, the gradient calculation parallelism is improved, the communication delay loss in parallel parameter update is reduced, and data preloading is adopted to reduce the data reading time, Improve the efficiency of network model training.

(2023) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Huanqin Wu and Fulin Li "Design of big data processing system optimization based on deep learning", Proc. SPIE 12970, Fourth International Conference on Signal Processing and Computer Science (SPCS 2023), 129702J (21 December 2023); https://doi.org/10.1117/12.3012256

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available