The latest video coding standard, Versatile Video Coding (VVC), has been finalized in July 2020. In traditional codecs, it is not possible to refer to different resolution pictures in inter-picture prediction except for inter layer cases. However, VVC does support Reference Picture Resampling (RPR), which allows pictures to refer to different resolution pictures so that video resolution can be adaptively switched in sequence. This work’s objective is to improve compression efficiency in the case of RPR. In the preliminary experiments, the compression efficiencies of reduced size encoded case where frames are encoded in half size and up-scaled to their original size and original size encoded case are compared. The result shows that most sequences get worse BD-rate when the sequences are encoded/decoded in reduced size. Here, it is noted that up-scaling is done by VVC Test Model, VTM-10.0, which uses a linear interpolation filter. Thus, the frames cannot recover high-frequency components sufficiently and cannot use temporal correlation. Therefore, this paper proposes a multi-frame super-resolution between frames of different resolutions where some frames are encoded in high quality frames in their original size and other frames in half size by using RPR. The proposed method uses the original size frames, which contain a lot of information, to perform super-resolution to the reduced frame. By doing so, it is observed it saves the bitrate by reducing the frame resolution and increases the quality of reduced frames by using multiframe super-resolution, which leads to improving the compression efficiency.
KEYWORDS: Super resolution, Quantization, Video, Video coding, Image quality, Video compression, Video processing, Machine vision, Computer vision technology, Signal processing
In video transmission, the videos are encoded and decoded. At that time, bit control is performed by specifying the quantization parameter (QP). The video undergoes various processing to remove redundancy and then orthogonally transforms the video signal into the frequency domain. The frequency domain coefficients are then quantized and transmitted. At that time, by specifying QP, the quantization step is changed, and the amount of data can be changed. In an opinion, a codec using super-resolution is proposed. At the CNN based super-resolution of encoded images, the degradation of the input image due to encoding depends on the characteristics of the image. As a result, there is a problem that the weights of the optimal CNN for the input image changes depending on the image characteristics. In order to solve this problem, we propose a method to adaptively perform super-resolution corresponding to image degradation.
As an important subtask of video restoration, video super-resolution has attracted a lot of attention in the community as it can eventually promote a wide range of technologies, e.g., video transmission system. Recent video super resolution model1 achieves cutting-edge performance. It efficiently utilizes recurrent architecture with neural networks to gradually aggregate details from previous frames. Nevertheless, this method faces a serious drawback that it is sensitive to occlusion, blur, and large motion changes since it only takes the previous generated output as recurrent input for the super resolution model. This will lead to undesirable rapid information loss during the recurrently generating process, and performance will therefore be dramatically decreased. Our works focus on addressing the issue of rapid information loss in video super resolution model with recurrent architecture. By producing attention maps through selective fusion module, the recurrent model can adaptively aggregate necessary details across all previously generated high-resolution (HR) frames according to their informativeness. The proposed method is useful for preserving high frequency details collected progressively from each frame while being capable of removing noisy artifacts. This significantly improves the average quality of the super resolution video.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.