Design and optimization of Zstandard algorithm based on concurrent streaming of multiple hash tables

Leidong Zheng; Youyu Wu; Min Zhu; Wei Ge; Jinjiang Yang

doi:10.1117/12.2649516

14 October 2022 Design and optimization of Zstandard algorithm based on concurrent streaming of multiple hash tables

Leidong Zheng, Youyu Wu, Min Zhu, Wei Ge, Jinjiang Yang

Proceedings Volume 12343, 2nd International Conference on Laser, Optics and Optoelectronic Technology (LOPET 2022); 123432G (2022) https://doi.org/10.1117/12.2649516
Event: 2nd International Conference on Laser, Optics and Optoelectronic Technology (LOPET 2022), 2022, Qingdao, China

Abstract

When compressing massive data, using software implementation of Zstandard (Zstd) algorithm will occupy a lot of processor resources, and it will lead to performance bottleneck due to serial data dependency, hardware acceleration of Zstd algorithm is an effective solution to the above problems. Therefore, in this paper, we propose a hardware architecture for concurrent streaming of multiple hash tables applicable to the Zstd algorithm. Using this design scheme, the RTL-based VCS simulation verification shows that the compression ratio almost reaches the standard of software, the compression speed of hardware is about 5.5 times of software compression speed, and the decompression speed of hardware is about 5 times of software, where the compression speed reaches up to 1.17GB/s and decompression speed reaches up to 1.89GB/s, and compared with the existing hardware implementation of Zstd, the compression speed is about 11.4% higher than that of the existing hardware implementation of Zstd.

Citation Download Citation

Leidong Zheng, Youyu Wu, Min Zhu, Wei Ge, and Jinjiang Yang "Design and optimization of Zstandard algorithm based on concurrent streaming of multiple hash tables", Proc. SPIE 12343, 2nd International Conference on Laser, Optics and Optoelectronic Technology (LOPET 2022), 123432G (14 October 2022); https://doi.org/10.1117/12.2649516

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available