Abstract
Generalized metric depth understanding is critical for precise vision-guided robotics, which current state-of-the-art (SOTA) vision-encoders do not support. To address this, we propose Vanishing Depth, a self-supervised training approach that extends pretrained RGB encoders to incorporate and align metric depth into their feature embeddings. Based on our novel positional depth encoding, we enable stable depth density and depth distribution invariant feature extraction. We achieve performance improvements and SOTA results across a spectrum of relevant RGBD downstream tasks - without the necessity of finetuning the encoder. Most notably, we achieve 56.05 mIoU on SUN-RGBD segmentation, 88.3 RMSE on Void's depth completion, and 83.8 Top 1 accuracy on NYUv2 scene classification. In 6D-object pose estimation, we outperform our predecessors of DinoV2, EVA-02, and Omnivore and achieve SOTA results for non-finetuned encoders in several related RGBD downstream tasks.
Abstract (translated)
通用度量深度理解对于精确的视觉引导机器人技术至关重要，而当前最先进的（SOTA）视觉编码器无法支持这一点。为了解决这个问题，我们提出了Vanishing Depth，这是一种自监督训练方法，它扩展了预训练的RGB编码器，以便将度量深度整合到其特征嵌入中，并进行对齐处理。基于我们的新型位置深度编码技术，我们能够实现稳定且与深度密度和分布无关的特征提取能力。我们在一系列相关的RGBD下游任务上实现了性能改进和SOTA结果——并且不需要微调编码器。最值得注意的是，在SUN-RGBD分割任务中我们达到了56.05 mIoU，在Void的深度完成任务中我们达到了88.3 RMSE，在NYUv2场景分类任务中我们达到了83.8 Top 1准确率。在6D物体姿态估计方面，我们超越了DinoV2、EVA-02和Omnivore等前人，并且对于几种相关的RGBD下游任务实现了未经微调编码器的SOTA结果。
URL
https://arxiv.org/abs/2503.19947
