Abstract
Most sign language handshape datasets are severely limited and unbalanced, posing significant challenges to effective model training. In this paper, we explore the effectiveness of augmenting the training data of a handshape classifier by generating synthetic data. We use an EfficientNet classifier trained on the RWTH German sign language handshape dataset, which is small and heavily unbalanced, applying different strategies to combine generated and real images. We compare two Generative Adversarial Networks (GAN) architectures for data generation: ReACGAN, which uses label information to condition the data generation process through an auxiliary classifier, and SPADE, which utilizes spatially-adaptive normalization to condition the generation on pose information. ReACGAN allows for the generation of realistic images that align with specific handshape labels, while SPADE focuses on generating images with accurate spatial handshape configurations. Our proposed techniques improve the current state-of-the-art accuracy on the RWTH dataset by 5%, addressing the limitations of small and unbalanced datasets. Additionally, our method demonstrates the capability to generalize across different sign language datasets by leveraging pose-based generation trained on the extensive HaGRID dataset. We achieve comparable performance to single-source trained classifiers without the need for retraining the generator.
Abstract (translated)
大多数手语手势数据集在规模和平衡性上都存在严重限制，这给有效的模型训练带来了重大挑战。本文探讨了通过生成合成数据来增强手势分类器训练数据有效性的方法。我们使用了一个基于RWTH德国手语手势数据集（该数据集较小且不平衡）训练的EfficientNet分类器，并应用不同的策略将生成的数据与真实图像结合。我们比较了两种用于数据生成的生成对抗网络（GAN）架构：ReACGAN，通过辅助分类器利用标签信息来调节数据生成过程；以及SPADE，它采用空间自适应归一化来根据姿势信息调整生成过程。ReACGAN能够生成与特定手势标签相符的真实图像，而SPADE则专注于生成具有准确手部姿态配置的图像。 我们提出的方法在RWTH数据集上将当前最先进的准确性提高了5%，解决了小规模和不平衡数据集的问题。此外，我们的方法还展示了跨不同手语数据集泛化的能力，通过基于姿势的生成技术，在庞大的HaGRID数据集上进行训练后实现这一点。我们在无需重新训练生成器的情况下，达到了与单一来源训练分类器相当的表现。 综上所述，本文提出的方法为解决手势识别中数据集限制问题提供了一种有效途径，并展示了其在不同手语数据集中泛化的潜力。
URL
https://arxiv.org/abs/2507.17008
