Abstract
Generalizable NeRF aims to synthesize novel views for unseen scenes. Common practices involve constructing variance-based cost volumes for geometry reconstruction and encoding 3D descriptors for decoding novel views. However, existing methods show limited generalization ability in challenging conditions due to inaccurate geometry, sub-optimal descriptors, and decoding strategies. We address these issues point by point. First, we find the variance-based cost volume exhibits failure patterns as the features of pixels corresponding to the same point can be inconsistent across different views due to occlusions or reflections. We introduce an Adaptive Cost Aggregation (ACA) approach to amplify the contribution of consistent pixel pairs and suppress inconsistent ones. Unlike previous methods that solely fuse 2D features into descriptors, our approach introduces a Spatial-View Aggregator (SVA) to incorporate 3D context into descriptors through spatial and inter-view interaction. When decoding the descriptors, we observe the two existing decoding strategies excel in different areas, which are complementary. A Consistency-Aware Fusion (CAF) strategy is proposed to leverage the advantages of both. We incorporate the above ACA, SVA, and CAF into a coarse-to-fine framework, termed Geometry-aware Reconstruction and Fusion-refined Rendering (GeFu). GeFu attains state-of-the-art performance across multiple datasets. Code is available at this https URL .
Abstract (translated)
可扩展的 NeRF 的目标是生成未曾见过的场景的新视角。常见的做法包括基于方差构建形状重构的成本体积和为解码新视图编码 3D 描述符。然而，由于不准确的几何、次优描述符和解码策略，现有方法在具有挑战性的条件下表现出有限的泛化能力。我们逐一解决这些问题。首先，我们发现基于方差的成本体积表现出失败模式，因为对应于同一点的像素特征在不同视图中可能是不一致的。我们引入了一种自适应成本聚合（ACA）方法来放大一致像素对描述符的贡献并抑制不一致的像素。与之前方法仅将 2D 特征融合到描述符中不同，我们的方法引入了一种空间视图聚合器（SVA），通过空间和视图交互将 3D 上下文融入描述符中。在解码描述符时，我们观察到两种现有解码策略在不同的领域表现出优异性能，这些策略是互补的。我们提出了一个一致性感知融合（CAF）策略，利用两种策略的优势。我们将上述的 ACA、SVA 和 CAF 融入到粗-到-细的框架中，称之为几何感知重构和优化渲染（GeFu）。GeFu 在多个数据集上取得最先进的性能。代码可以从该链接下载：https://www.researchgate.net/publication/333003621_Geometry-aware_Reconstruction_and_Fusion_Rendering
URL
https://arxiv.org/abs/2404.17528
