Abstract
Social group detection, or the identification of humans involved in reciprocal interpersonal interactions (e.g., family members, friends, and customers and merchants), is a crucial component of social intelligence needed for agents transacting in the world. The few existing benchmarks for social group detection are limited by low scene diversity and reliance on third-person camera sources (e.g., surveillance footage). Consequently, these benchmarks generally lack real-world evaluation on how groups form and evolve in diverse cultural contexts and unconstrained settings. To address this gap, we introduce EgoGroups, a first-person view dataset that captures social dynamics in cities around the world. EgoGroups spans 65 countries covering low, medium, and high-crowd settings under four weather/time-of-day conditions. We include dense human annotations for person and social groups, along with rich geographic and scene metadata. Using this dataset, we performed an extensive evaluation of state-of-the-art VLM/LLMs and supervised models on their group detection capabilities. We found several interesting findings, including VLMs and LLMs can outperform supervised baselines in a zero-shot setting, while crowd density and cultural regions clearly influence model performance.
Abstract (translated)
社交群体检测，即识别参与相互人际互动（如家庭成员、朋友、顾客与商家）的人类，是智能体在现实世界中开展交互所需社交智能的关键组成部分。现有少数社交群体检测基准数据集受限于场景多样性不足以及过度依赖第三人称视角影像源（如监控录像），因此这些基准通常缺乏在多元文化背景和无约束环境下对群体形成与演变过程的真实世界评估。为填补此空白，我们推出了EgoGroups——一个捕捉全球城市社交动态的第一人称视角数据集。该数据集覆盖65个国家，包含低、中、高人流密度场景，并涵盖四种天气/时段条件。数据包含密集的人员与社交群体人工标注，以及丰富的地理与场景元数据。基于此数据集，我们对当前最先进的视觉语言模型/大语言模型及有监督模型进行了广泛的群体检测能力评估。研究发现若干有趣结论：视觉语言模型与大语言模型在零样本设置下可超越有监督基线模型，而人群密度与文化区域则显著影响模型性能。
URL
https://arxiv.org/abs/2603.22249
