Abstract
One crucial factor behind the success of deep learning lies in the implicit bias induced by noise inherent in gradient-based training algorithms. Motivated by empirical observations that training with noisy labels improves model generalization, we delve into the underlying mechanisms behind stochastic gradient descent (SGD) with label noise. Focusing on a two-layer over-parameterized linear network, we analyze the learning dynamics of label noise SGD, unveiling a two-phase learning behavior. In \emph{Phase I}, the magnitudes of model weights progressively diminish, and the model escapes the lazy regime; enters the rich regime. In \emph{Phase II}, the alignment between model weights and the ground-truth interpolator increases, and the model eventually converges. Our analysis highlights the critical role of label noise in driving the transition from the lazy to the rich regime and minimally explains its empirical success. Furthermore, we extend these insights to Sharpness-Aware Minimization (SAM), showing that the principles governing label noise SGD also apply to broader optimization algorithms. Extensive experiments, conducted under both synthetic and real-world setups, strongly support our theory. Our code is released at this https URL.
Abstract (translated)
深度学习成功的一个关键因素在于基于梯度的训练算法中固有的噪声所引起的隐式偏差。受使用带噪标签进行训练可以提高模型泛化能力这一经验观察的启发，我们深入研究了带有标签噪声的随机梯度下降（SGD）背后的机制。重点分析了一个两层过参数化的线性网络的学习动态，在这个过程中揭示了标签噪声SGD的双阶段学习行为。 在**第一阶段（Phase I）**中，模型权重的大小逐渐减小，并且模型从懒惰模式逃逸进入丰富模式。 在**第二阶段（Phase II）**中，模型权重与真实插值器之间的对齐度增加，最终达到收敛状态。 我们的分析强调了标签噪声在推动从懒惰模式向丰富模式转变中的关键作用，并至少部分解释了其经验上的成功。此外，我们将这些见解扩展到了Sharpness-Aware Minimization（SAM）中，表明指导带有标签噪声的SGD的原则也适用于更广泛的优化算法。我们通过在合成和真实世界设置下进行的一系列实验强烈支持我们的理论。我们的代码可在[此链接](https://this https URL)处获取。
URL
https://arxiv.org/abs/2603.10397
