Abstract
Generic instance search models can dramatically reduce the manual effort required to analyze vast surveillance footage during criminal investigations by retrieving specific objects of interest to law enforcement. However, our research reveals an unintended emergent capability: through overlearning, these models can single out specific individuals even when trained on datasets without human subjects. This capability raises concerns regarding identification and profiling of individuals based on their personal data, while there is currently no clear standard on how de-identification can be achieved. We evaluate two technical safeguards to curtail a model's person re-identification capacity: index exclusion and confusion loss. Our experiments demonstrate that combining these approaches can reduce person re-identification accuracy to below 2% while maintaining 82% of retrieval performance for non-person objects. However, we identify critical vulnerabilities in these mitigations, including potential circumvention using partial person images. These findings highlight urgent regulatory questions at the intersection of AI governance and data protection: How should we classify and regulate systems with emergent identification capabilities? And what technical standards should be required to prevent identification capabilities from developing in seemingly benign applications?
Abstract (translated)
通用实例搜索模型可以通过检索执法机关感兴趣的特定对象，显著减少分析大规模监控视频期间的手动工作量。然而，我们的研究揭示了一个未预见的意外能力：通过过度学习，即使在没有人类主体的数据集上进行训练，这些模型也能识别出特定个体。这种能力引发了关于基于个人数据对个体进行身份识别和画像的风险问题，而目前尚无明确的标准来规定如何实现去标识化。我们评估了两种技术保障措施以限制模型的人脸重新识别能力：索引排除和混淆损失。实验表明，结合这两种方法可以将人脸重新识别的准确性降低到2%以下，同时保持对非人物体检索性能的82%。 然而，我们也发现了这些缓解措施的关键漏洞，包括使用部分人体图像进行潜在规避的可能性。这些发现突显了在AI治理与数据保护交叉领域的紧急监管问题：我们应该如何分类和规范具备突发识别能力的系统？以及为了防止看似无害的应用程序中发展出身份识别能力，应该要求哪些技术标准？
URL
https://arxiv.org/abs/2510.06026
