Abstract
With the rapid advancement and widespread adoption of VR/AR technologies, there is a growing demand for the creation of high-quality, immersive dynamic scenes. However, existing generation works predominantly concentrate on the creation of static scenes or narrow perspective-view dynamic scenes, falling short of delivering a truly 360-degree immersive experience from any viewpoint. In this paper, we introduce \textbf{TiP4GEN}, an advanced text-to-dynamic panorama scene generation framework that enables fine-grained content control and synthesizes motion-rich, geometry-consistent panoramic 4D scenes. TiP4GEN integrates panorama video generation and dynamic scene reconstruction to create 360-degree immersive virtual environments. For video generation, we introduce a \textbf{Dual-branch Generation Model} consisting of a panorama branch and a perspective branch, responsible for global and local view generation, respectively. A bidirectional cross-attention mechanism facilitates comprehensive information exchange between the branches. For scene reconstruction, we propose a \textbf{Geometry-aligned Reconstruction Model} based on 3D Gaussian Splatting. By aligning spatial-temporal point clouds using metric depth maps and initializing scene cameras with estimated poses, our method ensures geometric consistency and temporal coherence for the reconstructed scenes. Extensive experiments demonstrate the effectiveness of our proposed designs and the superiority of TiP4GEN in generating visually compelling and motion-coherent dynamic panoramic scenes. Our project page is at this https URL.
Abstract (translated)
随着虚拟现实（VR）和增强现实（AR）技术的迅速发展及其广泛应用，对高质量、沉浸式动态场景的需求日益增长。然而，现有的生成工作主要集中在静态场景或窄视角动态场景的创建上，无法提供从任意视点出发的真实360度沉浸式体验。在本文中，我们介绍了**TiP4GEN**，这是一个先进的文本到动态全景场景生成框架，能够实现精细的内容控制，并合成富于动作、几何一致性的全景4D场景。TiP4GEN结合了全景视频生成和动态场景重建技术，以创建360度的沉浸式虚拟环境。 对于视频生成部分，我们提出了一种**双分支生成模型**，包括一个用于全局视图生成的全景支路和一个用于局部视图生成的透视支路。双向交叉注意机制促进了两个分支之间全面的信息交换。 在场景重建方面，我们提出了基于三维高斯点云的**几何对齐重构模型**。通过使用度量深度地图来对齐时空点云，并利用估计的姿态初始化场景摄像机，我们的方法确保了重建场景的几何一致性和时间连贯性。 广泛的实验验证了我们所提出的方案的有效性和TiP4GEN在生成视觉吸引力强、动作协调的动态全景场景方面的优越性。我们的项目页面位于此[URL]（请将此处的"this https URL"替换为实际链接）。
URL
https://arxiv.org/abs/2508.12415
