Abstract
The evaluation of generative models for natural image tasks has been extensively studied. Similar protocols and metrics are used in cases with unique particularities, such as Handwriting Generation, even if they might not be completely appropriate. In this work, we introduce three measures tailored for HTG evaluation, $ \text{HTG}_{\text{HTR}} $, $ \text{HTG}_{\text{style}} $, and $ \text{HTG}_{\text{OOV}} $, and argue that they are more expedient to evaluate the quality of generated handwritten images. The metrics rely on the recognition error/accuracy of Handwriting Text Recognition and Writer Identification models and emphasize writing style, textual content, and diversity as the main aspects that adhere to the content of handwritten images. We conduct comprehensive experiments on the IAM handwriting database, showcasing that widely used metrics such as FID fail to properly quantify the diversity and the practical utility of generated handwriting samples. Our findings show that our metrics are richer in information and underscore the necessity of standardized evaluation protocols in HTG. The proposed metrics provide a more robust and informative protocol for assessing HTG quality, contributing to improved performance in HTR. Code for the evaluation protocol is available at: this https URL.
Abstract (translated)
自然图像任务的生成模型评估已经得到了广泛研究。在具有独特特点的情况下，如手写字符生成，即使它们可能不完全合适，也会使用类似的协议和指标。在这项工作中，我们引入了三个专门为HTG评估而设计的度量标准：$ \text{HTG}_{\text{HTR}} $，$ \text{HTG}_{\text{style}} $和$ \text{HTG}_{\text{OOV}} $，并认为它们更有效地评估生成手写字符图片的质量。这些指标基于手写字符识别错误/准确性和写者识别模型的识别误差/准确度，并强调书写风格，文本内容，多样性为主要方面，符合手写字符图片的内容。我们在IAM手写数据库上进行全面的实验，表明广泛使用的指标如FID无法正确量化生成手写字符样本的多样性和实际应用价值。我们的研究结果表明，我们的指标在信息量上更加丰富，进一步强调了在HTG中使用标准化评估协议的必要性。所提出的指标为评估HTG质量提供了一个更健壮和有益的协议，有助于提高HTR的性能。评估协议的代码可在此处下载：https:// this URL。
URL
https://arxiv.org/abs/2409.02683
