Abstract
Generating realistic 3D hand motion from natural language is vital for VR, robotics, and human-computer interaction. Existing methods either focus on full-body motion, overlooking detailed hand gestures, or require explicit 3D object meshes, limiting generality. We propose TSHaMo, a model-agnostic teacher-student diffusion framework for text-driven hand motion generation. The student model learns to synthesize motions from text alone, while the teacher leverages auxiliary signals (e.g., MANO parameters) to provide structured guidance during training. A co-training strategy enables the student to benefit from the teacher's intermediate predictions while remaining text-only at inference. Evaluated using two diffusion backbones on GRAB and H2O, TSHaMo consistently improves motion quality and diversity. Ablations confirm its robustness and flexibility in using diverse auxiliary inputs without requiring 3D objects at test time.
Abstract (translated)
从自然语言生成逼真的3D手部运动对于虚拟现实、机器人技术和人机交互至关重要。现有方法要么专注于全身运动，忽略了精细的手势细节；要么需要显式的3D物体网格，限制了通用性。我们提出了TSHaMo，一个与模型无关的教师-学生扩散框架，用于文本驱动的手部运动生成。学生模型仅从文本中学习合成运动，而教师则利用辅助信号（例如MANO参数）在训练期间提供结构化指导。一种协同训练策略使学生能够从教师的中期预测中获益，同时在推理时仍保持仅依赖文本。在GRAB和H2O数据集上使用两种扩散骨干网络进行评估，TSHaMo持续提升了运动质量和多样性。消融实验证实了其鲁棒性和灵活性，能够利用多样化的辅助输入，且无需在测试时使用3D物体。
URL
https://arxiv.org/abs/2603.24407
