Abstract
Indoor environments evolve as objects move, appear, or disappear. Capturing these dynamics requires maintaining temporally consistent instance identities across intermittently captured 3D scans, even when changes are unobserved. We introduce and formalize the task of temporally sparse 4D indoor semantic instance segmentation (SIS), which jointly segments, identifies, and temporally associates object instances. This setting poses a challenge for existing 3DSIS methods, which require a discrete matching step due to their lack of temporal reasoning, and for 4D LiDAR approaches, which perform poorly due to their reliance on high-frequency temporal measurements that are uncommon in the longer-horizon evolution of indoor environments. We propose ReScene4D, a novel method that adapts 3DSIS architectures for 4DSIS without needing dense observations. It explores strategies to share information across observations, demonstrating that this shared context not only enables consistent instance tracking but also improves standard 3DSIS quality. To evaluate this task, we define a new metric, t-mAP, that extends mAP to reward temporal identity consistency. ReScene4D achieves state-of-the-art performance on the 3RScan dataset, establishing a new benchmark for understanding evolving indoor scenes.
Abstract (translated)
室内环境随着物体的移动、出现或消失而演变。捕捉这些动态变化需要在间歇性捕获的三维扫描中保持时间上一致的对象实例身份，即使是在未被观察到的变化情况下也是如此。我们提出了并形式化了稀疏4D室内语义实例分割（SIS）的任务，该任务同时对物体实例进行分割、标识和时序关联。这一设置对于现有的3DSIS方法构成了挑战，因为这些方法由于缺乏时间推理能力而需要一个离散匹配步骤；同样，依赖高频时间测量的4DLiDAR方法在室内环境长期演变中的应用效果较差，因为在长时间尺度上获取高频率数据并不常见。 为此，我们提出了一种新的方法——ReScene4D，该方法可以将3DSIS架构适应到4DSIS中，而无需密集观察。它探索了跨不同观测值分享信息的策略，并证明这种共享上下文不仅支持一致性的实例追踪，还能提升标准3DSIS的质量。 为了评估这一任务，我们定义了一个新的度量标准——t-mAP，该指标扩展了mAP来奖励时间身份一致性。ReScene4D在3RScan数据集上达到了最先进的性能水平，为理解不断演变的室内场景设立了新基准。
URL
https://arxiv.org/abs/2601.11508
