Abstract
Recent advancements in Machine Unlearning (MU) have introduced solutions to selectively remove certain training samples, such as those with outdated or sensitive information, from trained models. Despite these advancements, evaluation of MU methods have been inconsistent, employing different trained models and architectures, and sample removal strategies, which hampers accurate comparison. In addition, prior MU approaches have mainly focused on singular tasks or modalities, which is not comprehensive. To address these limitations, we develop MU-Bench, the first comprehensive benchmark for MU that (i) unifies the sets of deleted samples and trained models, and (ii) provides broad coverage of tasks and data modalities, including previously unexplored domains such as speech and video classification. Our evaluation show that RandLabel and SalUn are the most effective general MU approaches on MU-Bench, and BadT and SCRUB are capable of achieving random performance on the deletion set. We analyze several under-investigated aspects of unlearning, including scalability, the impacts of parameter-efficient fine-tuning and curriculum learning, and susceptibility to dataset biases. MU-Bench provides an easy-to-use package that includes dataset splits, models, and implementations, together with a leader board to enable unified and scalable MU research.
Abstract (translated)
近年来，机器学习（ML）中的进展为选择性地从训练模型中移除某些训练样本，如过时或敏感信息，提供了解决方案。尽管这些进步，但ML方法的评估存在不一致性，使用了不同的训练模型和架构以及样本删除策略，这阻碍了准确比较。此外，之前的多元方法主要集中在单一任务或维度上，不全面。为了克服这些限制，我们开发了MU-Bench，第一个全面的MU基准，它统一了删除的样本和训练模型，并为任务和数据维度提供广泛的覆盖，包括之前未探索的领域，如语音和视频分类。我们的评估显示，RandLabel和SalUn是MU-Bench中最有效的通用ML方法，而BadT和SCRUB在删除集上具有随机性能。我们分析了几个尚未进行深入研究的问题，包括可扩展性、参数高效微调以及正则化学习，以及易受数据偏差的影响。MU-Bench提供了一个易于使用的软件包，包括数据集划分、模型和实现，以及领导者板，以实现统一和可扩展的ML研究。
URL
https://arxiv.org/abs/2406.14796
