Voxel grid performer: efficient radiance fields generalization for human novel view synthesis

Zhongyi Fan; Ming Liu; Yuejin Zhao; Liquan Dong; Mei Hui; Lingqin Kong; Qikun Yang

doi:10.1117/12.2643856

19 December 2022 Voxel grid performer: efficient radiance fields generalization for human novel view synthesis

Zhongyi Fan, Ming Liu, Yuejin Zhao, Liquan Dong, Mei Hui, Lingqin Kong, Qikun Yang

Proceedings Volume 12319, Optical Metrology and Inspection for Industrial Applications IX; 123190M (2022) https://doi.org/10.1117/12.2643856
Event: SPIE/COS Photonics Asia, 2022, Online Only

Abstract

Novel view synthesis is a long-standing problem. Despite the rapid development of neural radiance field (nerf), in terms of rendering dynamic human body, NeRF still cannot achieve a good trade-off in precision and efficiency. In this paper, we aim at synthesizing a free-viewpoint video of an arbitrary human performers in an efficient way, only requiring a sparse number of camera views as inputs and skirting per-case fine-tuning. Recently, several works have addressed this problem by learning person-specific neural radiance fields (NeRF) to capture the appearance of a particular human. In parallel, some work proposed to use pixel-aligned features to generalize radiance fields to arbitrary new scenes and objects. Adopting these generalization approchs to human achieve reasonable rendering result. However, due to the difficulties of modeling the complex appearance of human and the dynamic sense, it is challenging to train nerf well in an efficient way. We find that the slow convergence of the human body reconstruction model is largely due to the nerf representation. In this work, we introduce a voxel grid based representation for human view synthesis, termed Voxel Grid Performer(VGP). Specifically, a sparse voxel grid is designed to represent the density and color in every space voxel, which enable better performance and less computation than conventional nerf optimization. We perform extensive experiments on both seen human performer and unseen human performer, demonstrating that our approach surpasses nerf-based methods on a wide variety of metrics. Code and data will be made available at https://github.com/fanzhongyi/vgp.

Conference Presentation

Citation Download Citation

Zhongyi Fan, Ming Liu, Yuejin Zhao, Liquan Dong, Mei Hui, Lingqin Kong, and Qikun Yang "Voxel grid performer: efficient radiance fields generalization for human novel view synthesis", Proc. SPIE 12319, Optical Metrology and Inspection for Industrial Applications IX, 123190M (19 December 2022); https://doi.org/10.1117/12.2643856

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available