Selecting among multiple transform kernels to code prediction residuals is widely used for better compression efficiency. Conventionally, the encoder performs trials of each transform to estimate the rate-distortion (R-D) cost. However, such an exhaustive approach suffers from a significant increase of complexity. In this paper, a novel rate estimation approach is proposed to by-pass the entropy coding process for each transform type using the conditional Laplace distribution model. The proposed method estimates the Laplace distribution parameter by the context inferred by the quantization level and finds the expected rate of the coefficients for transform type selection. Furthermore, a greedy search algorithm for separable transforms is also presented to further accelerate the process. Experimental results show that the transform type selection scheme using the proposed rate estimation method achieves high accuracy and provides a satisfactory speed-performance trade-off.

© (2020) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.

Bohan Li, Jingning Han, and Yaowu Xu, "Fast transform type selection using conditional Laplace distribution based rate estimation," Proc. SPIE 11510, Applications of Digital Image Processing XLIII, 115101Z (Presented at SPIE Optical Engineering + Applications: August 27, 2020; Published: 21 August 2020); https://doi.org/10.1117/12.2568930.