The advent of neural and Gaussian-based radiance field methods have achieved great success in the field of novel view synthesis. However, specular reflection remains non-trivial, as the high frequency radiance field is notoriously difficult to fit stably and accurately. We present a deferred shading method to effectively render specular reflection with Gaussian splatting. The key challenge comes from the environment map reflection model, which requires accurate surface normal while simultaneously bottlenecks normal estimation with discontinuous gradients. We leverage the per-pixel reflection gradients generated by deferred shading to bridge the optimization process of neighboring Gaussians, allowing nearly correct normal estimations to gradually propagate and eventually spread over all reflective objects. Our method significantly outperforms state-of-the-art techniques and concurrent work in synthesizing high-quality specular reflection effects, demonstrating a consistent improvement of peak signal-to-noise ratio (PSNR) for both synthetic and real-world scenes, while running at a frame rate almost identical to vanilla Gaussian splatting.
Our deferred rendering model consists of two passes. A Gaussian splatting pass is first performed to bake reflection strength, normal, and base color to screen space maps. In the following shading pass, for each pixel, we use the normal map to compute a reflection direction and query an environment map for a reflected color. The reflection strength is then used to blend base color and reflection color into the final result. An image loss is used to back-propagate gradients. Note that there exist many gradient propagation paths. Here we only illustrate the gradient flow most relevant to reflection fitting.
We present a training algorithm featuring normal propagation. Specifically, based on the observation that Gaussians with relative large reflective strength values have near-correct normal vectors, we expand these reflective Gaussians to propagate their normal vectors to nearby Gaussians. In this way, after one Gaussian with near-correct normal overlaps a different Gaussian without one, some shared pixels can also have near-correct normal, which will get meaningful normal gradients, helping to optimize the normal of the later Gaussian.
[Kerbl et al. 2023] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 2023. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics (SIGGRAPH Conference Proceedings) 42, 4 (July 2023).
[Jiang et al. 2023] Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long, Wenping Wang, and Yuexin Ma. 2023. GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfaces. arXiv:2311.17977 [cs.CV]
[Liang et al. 2023] Ruofan Liang, Huiting Chen, Chunlin Li, Fan Chen, Selvakumar Panneer, and Nandita Vijaykumar. 2023. Envidr: Implicit differentiable renderer with neural environment lighting. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 79-89.
[Verbin et al. 2022] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan. 2022. Ref-nerf: Structured view-dependent appearance for neural radiance fields. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 5481-5490.
@article{ye2024gsdr,
author = {Keyang, Ye and Qiming, Hou and Kun, Zhou},
title = {3D Gaussian Splatting with Deferred Reflection},
booktitle = {ACM SIGGRAPH Conference Proceedings, Denver, CO, United States, July 28 - August 1, 2024},
year = {2024},
}