Gradient_Descent
-
ADMM-Based Training for Spiking Neural Networks
2025-05-08 10:20:33Giovanni Perin, Cesare Bidini, Riccardo Mazzieri, Michele RossiAbstract
In recent years, spiking neural networks (SNNs) have gained momentum due to their high potential in time-series processing combined with minimal energy consumption. However, they still lack a dedicated and efficient training algorithm. The popular backpropagation with surrogate gradients, adapted from stochastic gradient descent (SGD)-derived algorithms, has several drawbacks when used as an optimizer for SNNs. Specifically, it suffers from low scalability and numerical imprecision. In this paper, we propose a novel SNN training method based on the alternating direction method of multipliers (ADMM). Our ADMM-based training aims to solve the problem of the SNN step function's non-differentiability. We formulate the problem, derive closed-form updates, and empirically show the optimizer's convergence properties, great potential, and possible new research directions to improve the method in a simulated proof-of-concept.
Abstract (translated)
近年来，由于脉冲神经网络（SNN）在时间序列处理方面具有巨大潜力，并且能耗极低，因此它们得到了迅速的发展。然而，目前这些网络仍缺乏一个专门和高效的训练算法。从随机梯度下降（SGD）衍生出来的流行反向传播方法与替代梯度，在作为SNN优化器使用时存在一些缺点，特别是可扩展性和数值精度方面的问题。在本文中，我们提出了一种基于交替方向乘子法（ADMM）的新颖的SNN训练方法。我们的基于ADMM的训练旨在解决SNN步函数不可微分的问题。我们对问题进行了建模，推导了封闭形式的更新公式，并通过模拟的概念验证实验展示了优化器的收敛特性、巨大的潜力以及可能的研究新方向来改进这种方法。
URL
https://arxiv.org/abs/2505.05527
PDF
-
Precise gradient descent training dynamics for finite-width multi-layer neural networks
Abstract
In this paper, we provide the first precise distributional characterization of gradient descent iterates for general multi-layer neural networks under the canonical single-index regression model, in the `finite-width proportional regime' where the sample size and feature dimension grow proportionally while the network width and depth remain bounded. Our non-asymptotic state evolution theory captures Gaussian fluctuations in first-layer weights and concentration in deeper-layer weights, and remains valid for non-Gaussian features. Our theory differs from existing neural tangent kernel (NTK), mean-field (MF) theories and tensor program (TP) in several key aspects. First, our theory operates in the finite-width regime whereas these existing theories are fundamentally infinite-width. Second, our theory allows weights to evolve from individual initializations beyond the lazy training regime, whereas NTK and MF are either frozen at or only weakly sensitive to initialization, and TP relies on special initialization schemes. Third, our theory characterizes both training and generalization errors for general multi-layer neural networks beyond the uniform convergence regime, whereas existing theories study generalization almost exclusively in two-layer settings. As a statistical application, we show that vanilla gradient descent can be augmented to yield consistent estimates of the generalization error at each iteration, which can be used to guide early stopping and hyperparameter tuning. As a further theoretical implication, we show that despite model misspecification, the model learned by gradient descent retains the structure of a single-index function with an effective signal determined by a linear combination of the true signal and the initialization.
Abstract (translated)
在这篇论文中，我们首次提供了多层神经网络在标准单索引回归模型下的梯度下降迭代的精确分布特征，在“有限宽度比例制度”下进行研究，即样本数量和特征维度按比例增长的同时，网络的宽度和深度保持不变。我们的非渐近状态演化理论捕捉了第一层权重中的高斯波动和更深层级权重中的集中特性，并且对非高斯特征依然有效。 该理论在几个关键方面与现有的神经切片核（NTK）、均值场（MF）理论以及张量程序（TP）有所不同。首先，我们的理论适用于有限宽度的情形，而这些现有理论本质上是无限宽度的。其次，我们的理论允许权重从个体初始化演变超出懒惰训练阶段，而NTK和MF要么在初始化时冻结不变，要么对初始化仅有微弱敏感性，并且TP依赖于特殊的初始化方案。第三，我们的理论描述了广泛多层神经网络的训练误差和泛化误差，超出了均匀收敛范围，而现有的理论几乎完全专注于两层设置下的泛化研究。 作为统计学的应用实例，我们展示了普通的梯度下降可以被增强以产生一致性的估计值，这些估计值在每次迭代时反映了泛化误差，并可用于指导提前停止和超参数调整。此外，从理论上讲，即使存在模型规格不当的情况，由梯度下降学习到的模型仍保留了单索引函数的形式，其有效信号由真实信号与初始化的线性组合决定。
URL
https://arxiv.org/abs/2505.04898
PDF
-
Adversarial Robustness Analysis of Vision-Language Models in Medical Image Segmentation
2025-05-05 18:54:41Anjila Budathoki, Manish DhakalAbstract
Adversarial attacks have been fairly explored for computer vision and vision-language models. However, the avenue of adversarial attack for the vision language segmentation models (VLSMs) is still under-explored, especially for medical image analysis. Thus, we have investigated the robustness of VLSMs against adversarial attacks for 2D medical images with different modalities with radiology, photography, and endoscopy. The main idea of this project was to assess the robustness of the fine-tuned VLSMs specially in the medical domain setting to address the high risk scenario. First, we have fine-tuned pre-trained VLSMs for medical image segmentation with adapters. Then, we have employed adversarial attacks -- projected gradient descent (PGD) and fast gradient sign method (FGSM) -- on that fine-tuned model to determine its robustness against adversaries. We have reported models' performance decline to analyze the adversaries' impact. The results exhibit significant drops in the DSC and IoU scores after the introduction of these adversaries. Furthermore, we also explored universal perturbation but were not able to find for the medical images. \footnote{this https URL}
Abstract (translated)
对抗攻击在计算机视觉和视觉-语言模型领域已得到了相当程度的研究。然而，针对视觉语言分割模型（VLSMs）的对抗攻击研究仍然较少，特别是在医学图像分析方面。因此，我们调查了不同模态2D医学图像上经过微调的VLSMs面对对抗攻击时的鲁棒性，尤其是在放射学、摄影和内窥镜检查领域。该项目的主要目的是评估在特定医疗场景下针对高风险情况下的微调VLSM的稳健性。 首先，我们在医学图像分割任务中使用适配器对预训练的VLSMs进行了微调。然后，我们对该模型应用了对抗攻击——包括投影梯度下降（PGD）和快速梯度符号方法（FGSM），以评估其在面对对手时的鲁棒性。我们报告了模型性能的下降情况来分析这些对抗样本的影响。实验结果显示，在引入这些对抗样本后，Dice相似系数（DSC）和交并比（IoU）得分有显著下降。 此外，我们也探索了一种通用扰动方法但未能在医学图像上找到有效应用。\footnote{此链接指向的是原始研究或相关资源的URL，但由于没有提供具体的URL地址，这里仅以说明性文本形式表达。} 这段翻译详细描述了项目的目标、方法和结果，并指出对抗攻击对VLSMs在医疗图像分析中的影响显著，但通用扰动法的应用尚未成功实现。
URL
https://arxiv.org/abs/2505.02971
PDF
-
Minimisation of Quasar-Convex Functions Using Random Zeroth-Order Oracles
Abstract
This study explores the performance of a random Gaussian smoothing zeroth-order (ZO) scheme for minimising quasar-convex (QC) and strongly quasar-convex (SQC) functions in both unconstrained and constrained settings. For the unconstrained problem, we establish the ZO algorithm's convergence to a global minimum along with its complexity when applied to both QC and SQC functions. For the constrained problem, we introduce the new notion of proximal-quasar-convexity and prove analogous results to the unconstrained case. Specifically, we show the complexity bounds and the convergence of the algorithm to a neighbourhood of a global minimum whose size can be controlled under a variance reduction scheme. Theoretical findings are illustrated through investigating the performance of the algorithm applied to a range of problems in machine learning and optimisation. Specifically, we observe scenarios where the ZO method outperforms gradient descent. We provide a possible explanation for this phenomenon.
Abstract (translated)
这项研究探讨了随机高斯平滑零阶(ZO)方案在无约束和有约束条件下最小化类夸克凸(QC)和强类夸克凸(SQC)函数时的性能。对于无约束问题，我们建立了ZO算法应用于QC和SQC函数时收敛于全局最优解及其复杂度的理论基础。对于有约束问题，我们引入了新的近似-类夸克凸性概念，并证明了与无约束情况类似的结果。具体来说，我们展示了算法在方差减少方案下的复杂度界线以及该算法收敛到一个可以控制大小的全局最优点邻域。 通过研究ZO方法应用于机器学习和优化问题中的性能表现，本文理论发现得到了验证。特别是，在一些场景中，ZO方法的表现优于梯度下降法。对此现象，我们提供了一种可能的解释。
URL
https://arxiv.org/abs/2505.02281
PDF
-
Focus on the Likely: Test-time Instance-based Uncertainty Removal
Abstract
We propose two novel test-time fine-tuning methods to improve uncertain model predictions. Our methods require no auxiliary data and use the given test instance only. Instead of performing a greedy selection of the most likely class to make a prediction, we introduce an additional focus on the likely classes step during inference. By applying a single-step gradient descent, we refine predictions when an initial forward pass indicates high uncertainty. This aligns predictions more closely with the ideal of assigning zero probability to less plausible outcomes. Our theoretical discussion provides a deeper understanding highlighting the impact on shared and non-shared features among (focus) classes. The experimental evaluation highlights accuracy gains on samples exhibiting high decision uncertainty for a diverse set of models from both the text and image domain using the same hyperparameters.
Abstract (translated)
我们提出了两种新颖的测试时间微调方法，旨在改进模型在不确定情况下的预测。这些方法不需要额外的数据支持，并且仅使用给定的测试实例。不同于执行贪婪选择最可能类别的策略以进行预测，我们在推理阶段引入了一个额外关注可能类别步骤的方法。通过应用单步梯度下降法，在初始前向传递表明高不确定性时，我们可以优化预测结果。这种方法使预测更加接近理想状态：即对不太可能的结果赋予零概率。 理论讨论深入探讨了共享和非共享特征在各类别中的影响，并提供了更深层次的理解。实验评估显示，使用相同的超参数设置，对于来自文本和图像领域多种模型中表现出高度决策不确定性的样本，我们的方法能够显著提高准确性。
URL
https://arxiv.org/abs/2505.03819
PDF
-
Towards the Resistance of Neural Network Watermarking to Fine-tuning
2025-05-02 05:11:17Ling Tang, Yuefeng Chen, Hui Xue, Quanshi ZhangAbstract
This paper proves a new watermarking method to embed the ownership information into a deep neural network (DNN), which is robust to fine-tuning. Specifically, we prove that when the input feature of a convolutional layer only contains low-frequency components, specific frequency components of the convolutional filter will not be changed by gradient descent during the fine-tuning process, where we propose a revised Fourier transform to extract frequency components from the convolutional filter. Additionally, we also prove that these frequency components are equivariant to weight scaling and weight permutations. In this way, we design a watermark module to encode the watermark information to specific frequency components in a convolutional filter. Preliminary experiments demonstrate the effectiveness of our method.
Abstract (translated)
本文提出了一种新的水印技术，可以将所有权信息嵌入到深度神经网络（DNN）中，并且这种水印在微调过程中是鲁棒的。具体而言，我们证明了当卷积层的输入特征仅包含低频分量时，在梯度下降过程中的微调阶段，该层的卷积滤波器的特定频率成分不会发生变化。为此，我们提出了一种修订版的傅里叶变换方法来从卷积滤波器中提取这些频率成分。此外，我们也证明了这些频率成分对于权重缩放和权重置换是等变（equivariant）的。通过这种方式，我们设计了一个水印模块，可以将水印信息编码到卷积滤波器的特定频率分量中。初步实验表明我们的方法的有效性。
URL
https://arxiv.org/abs/2505.01007
PDF
-
How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias
2025-05-02 00:07:35Ruiquan Huang, Yingbin Liang, Jing YangAbstract
Language recognition tasks are fundamental in natural language processing (NLP) and have been widely used to benchmark the performance of large language models (LLMs). These tasks also play a crucial role in explaining the working mechanisms of transformers. In this work, we focus on two representative tasks in the category of regular language recognition, known as `even pairs' and `parity check', the aim of which is to determine whether the occurrences of certain subsequences in a given sequence are even. Our goal is to explore how a one-layer transformer, consisting of an attention layer followed by a linear layer, learns to solve these tasks by theoretically analyzing its training dynamics under gradient descent. While even pairs can be solved directly by a one-layer transformer, parity check need to be solved by integrating Chain-of-Thought (CoT), either into the inference stage of a transformer well-trained for the even pairs task, or into the training of a one-layer transformer. For both problems, our analysis shows that the joint training of attention and linear layers exhibits two distinct phases. In the first phase, the attention layer grows rapidly, mapping data sequences into separable vectors. In the second phase, the attention layer becomes stable, while the linear layer grows logarithmically and approaches in direction to a max-margin hyperplane that correctly separates the attention layer outputs into positive and negative samples, and the loss decreases at a rate of $O(1/t)$. Our experiments validate those theoretical results.
Abstract (translated)
语言识别任务是自然语言处理（NLP）中的基本任务，广泛用于评估大型语言模型（LLMs）的性能。这些任务在解释变压器的工作机制方面也扮演着重要角色。在这项工作中，我们专注于两类常规语言识别任务的代表性任务：即“偶数对”和“奇偶校验”，其目的是确定给定序列中某些子序列出现次数是否为偶数。我们的目标是通过理论分析单层变压器（由注意力层和线性层组成）在梯度下降训练过程中的学习动态，来探索它如何解决这些任务。“偶数对”可以直接使用单层变压器解决，“奇偶校验”则需要结合“链式思维”(Chain-of-Thought, CoT) 方法来解决。对于已经经过“偶数对”任务良好训练的变压器推理阶段中集成CoT，或是在单层变压器的训练过程中直接整合。 针对这两个问题，我们的分析表明注意力和线性层的联合训练展示了两个不同的阶段。在第一阶段，注意力层迅速增长，并将数据序列映射为可分向量。而在第二阶段，注意力层变得稳定，而线性层则以对数方式增长，在方向上逐渐接近于能够正确分离注意力层输出（将样本分为正负两类）的最大间隔超平面，并且损失函数的减少速度是$O(1/t)$。 实验结果验证了这些理论成果。
URL
https://arxiv.org/abs/2505.00926
PDF
-
The Dark Side of Digital Twins: Adversarial Attacks on AI-Driven Water Forecasting
2025-04-28 22:34:11Mohammadhossein Homaei, Victor Gonzalez Morales, Oscar Mogollon-Gutierrez, Andres CaroAbstract
Digital twins (DTs) are improving water distribution systems by using real-time data, analytics, and prediction models to optimize operations. This paper presents a DT platform designed for a Spanish water supply network, utilizing Long Short-Term Memory (LSTM) networks to predict water consumption. However, machine learning models are vulnerable to adversarial attacks, such as the Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD). These attacks manipulate critical model parameters, injecting subtle distortions that degrade forecasting accuracy. To further exploit these vulnerabilities, we introduce a Learning Automata (LA) and Random LA-based approach that dynamically adjusts perturbations, making adversarial attacks more difficult to detect. Experimental results show that this approach significantly impacts prediction reliability, causing the Mean Absolute Percentage Error (MAPE) to rise from 26% to over 35%. Moreover, adaptive attack strategies amplify this effect, highlighting cybersecurity risks in AI-driven DTs. These findings emphasize the urgent need for robust defenses, including adversarial training, anomaly detection, and secure data pipelines.
Abstract (translated)
数字孪生（DT）通过使用实时数据、分析和预测模型来优化运营，正在改善水分配系统。本文介绍了一个专为西班牙供水网络设计的数字孪生平台，并利用长短期记忆（LSTM）网络来预测用水量。然而，机器学习模型容易受到诸如快速梯度符号法（FGSM）和投影梯度下降（PGD）等对抗性攻击的影响。这些攻击操纵关键的模型参数，在不影响数据外观的情况下注入细微扰动以降低预测准确性。为了进一步利用这些漏洞，我们引入了一种基于学习自动机（LA）和随机学习自动机的方法，该方法能够动态调整扰动，使对抗性攻击更难被检测到。实验结果表明，这种方法显著影响了预测的可靠性，导致平均绝对百分比误差（MAPE）从26%上升至超过35%。此外，适应性的攻击策略放大了这种效果，突显了AI驱动的数字孪生中的网络安全风险。这些发现强调了建立强大的防御措施的迫切需要，包括对抗性训练、异常检测和安全的数据管道。
URL
https://arxiv.org/abs/2504.20295
PDF
-
FedAvgen: Metadata for Model Aggregation In Communication Systems
2025-04-28 13:11:32Anthony Kiggundu, Dennis Krummacker, Hans D. SchottenAbstract
To improve business efficiency and minimize costs, Artificial Intelligence (AI) practitioners have adopted a shift from formulating models from scratch towards sharing pretrained models. The pretrained models are then aggregated into a global model with higher generalization capabilities, which is afterwards distributed to the client devices. This approach is known as federated learning and inherently utilizes different techniques to select the candidate client models averaged to obtain the global model. This approach, in the case of communication systems, faces challenges arising from the existential diversity in device profiles. The multiplicity in profiles motivates our conceptual assessment of a metaheuristic algorithm (FedAvgen), which relates each pretrained model with its weight space as metadata, to a phenotype and genotype, respectively. This parent-child genetic evolution characterizes the global averaging step in federated learning. We then compare the results of our approach to two widely adopted baseline federated learning algorithms like Federated Averaging (FedAvg) and Federated Stochastic Gradient Descent (FedSGD).
Abstract (translated)
为了提高业务效率并减少成本，人工智能（AI）从业者已从零开始构建模型转向共享预训练模型。这些预训练模型会被聚合为一个具有更强泛化能力的全局模型，然后分发到客户端设备上。这种方法被称为联邦学习，并且内在地利用不同的技术来选择用于获得全局模型的候选客户端模型的平均值。在通信系统中，该方法面临着由于设备配置文件存在多样性而产生的挑战。这种配置文件的多样化促使我们对一种元启发式算法（FedAvgen）进行概念评估，该算法将每个预训练模型与其权重空间分别视为元数据中的表型和基因型。这种父代-子代遗传进化特征化了联邦学习中的全局平均步骤。然后，我们将我们的方法的结果与两个广泛采用的基线联邦学习算法，如联邦平均（FedAvg）和联邦随机梯度下降（FedSGD），进行比较。
URL
https://arxiv.org/abs/2505.05486
PDF
-
Smart Placement, Faster Robots -- A Comparison of Algorithms for Robot Base-Pose Optimization
2025-04-28 08:36:17Matthias Mayer, Matthias AlthoffAbstract
Robotic automation is a key technology that increases the efficiency and flexibility of manufacturing processes. However, one of the challenges in deploying robots in novel environments is finding the optimal base pose for the robot, which affects its reachability and deployment cost. Yet, the existing research for automatically optimizing the base pose of robots has not been compared. We address this problem by optimizing the base pose of industrial robots with Bayesian optimization, exhaustive search, genetic algorithms, and stochastic gradient descent and find that all algorithms can reduce the cycle time for various evaluated tasks in synthetic and real-world environments. Stochastic gradient descent shows superior performance with regard to success rate solving over 90% of our real-world tasks, while genetic algorithms show the lowest final costs. All benchmarks and implemented methods are available as baselines against which novel approaches can be compared.
Abstract (translated)
机器人自动化是提高制造过程效率和灵活性的关键技术。然而，在新的环境中部署机器人时，一个挑战在于找到机器人的最优基座姿态（base pose），这会影响其可达性和部署成本。不过，现有的关于自动优化机器人基座姿态的研究尚未进行过比较研究。我们通过使用贝叶斯优化、穷举搜索、遗传算法和随机梯度下降来优化工业机器人的基座姿态，并发现所有这些算法都能在合成环境和真实世界环境中减少各种任务的循环时间。随机梯度下降法表现出色，成功解决了超过90%的真实世界任务；而遗传算法则显示出最低的最终成本。所有的基准测试和实现方法都可以作为新方法进行比较的基础。
URL
https://arxiv.org/abs/2504.19577
PDF
-
Sharp higher order convergence rates for the Adam optimizer
2025-04-28 02:17:50Steffen Dereich, Arnulf Jentzen, Adrian RiekertAbstract
Gradient descent based optimization methods are the methods of choice to train deep neural networks in machine learning. Beyond the standard gradient descent method, also suitable modified variants of standard gradient descent involving acceleration techniques such as the momentum method and/or adaptivity techniques such as the RMSprop method are frequently considered optimization methods. These days the most popular of such sophisticated optimization schemes is presumably the Adam optimizer that has been proposed in 2014 by Kingma and Ba. A highly relevant topic of research is to investigate the speed of convergence of such optimization methods. In particular, in 1964 Polyak showed that the standard gradient descent method converges in a neighborhood of a strict local minimizer with rate (x - 1)(x + 1)^{-1} while momentum achieves the (optimal) strictly faster convergence rate (\sqrt{x} - 1)(\sqrt{x} + 1)^{-1} where x \in (1,\infty) is the condition number (the ratio of the largest and the smallest eigenvalue) of the Hessian of the objective function at the local minimizer. It is the key contribution of this work to reveal that Adam also converges with the strictly faster convergence rate (\sqrt{x} - 1)(\sqrt{x} + 1)^{-1} while RMSprop only converges with the convergence rate (x - 1)(x + 1)^{-1}.
Abstract (translated)
基于梯度下降的优化方法是训练深度神经网络在机器学习中的首选方法。除了标准的梯度下降方法外，还包括适合加速技术（如动量法）和适应性技术（如RMSprop方法）的标准梯度下降的各种变体，这些技术也常被考虑作为优化方法。近年来，最流行的此类复杂优化方案可能是Kingma和Ba在2014年提出的Adam优化器。研究这样一个优化方法的收敛速度是一个非常重要的课题。特别地，在1964年Polyak证明了标准梯度下降方法在一个严格局部极小值附近的收敛率为(x - 1)(x + 1)^{-1}，而动量法则实现了最优且更快速的收敛率(\sqrt{x} - 1)(\sqrt{x} + 1)^{-1}，其中x \in (1,\infty)是目标函数在局部极小值点处Hessian矩阵的最大特征值与最小特征值比（即条件数）。这项工作的关键贡献在于揭示了Adam优化器也以严格更快的收敛率(\sqrt{x} - 1)(\sqrt{x} + 1)^{-1}进行收敛，而RMSprop仅以(x - 1)(x + 1)^{-1}的收敛速率进行收敛。
URL
https://arxiv.org/abs/2504.19426
PDF
-
NANO-SLAM : Natural Gradient Gaussian Approximation for Vehicle SLAM
2025-04-27 11:05:15Tianyi Zhang, Wenhan Cao, Chang Liu, Feihong Zhang, Wei Wu, Shengbo Eben LiAbstract
Accurate localization is a challenging task for autonomous vehicles, particularly in GPS-denied environments such as urban canyons and tunnels. In these scenarios, simultaneous localization and mapping (SLAM) offers a more robust alternative to GPS-based positioning, enabling vehicles to determine their position using onboard sensors and surrounding environment's landmarks. Among various vehicle SLAM approaches, Rao-Blackwellized particle filter (RBPF) stands out as one of the most widely adopted methods due to its efficient solution with logarithmic complexity relative to the map size. RBPF approximates the posterior distribution of the vehicle pose using a set of Monte Carlo particles through two main steps: sampling and importance weighting. The key to effective sampling lies in solving a distribution that closely approximates the posterior, known as the sampling distribution, to accelerate convergence. Existing methods typically derive this distribution via linearization, which introduces significant approximation errors due to the inherent nonlinearity of the system. To address this limitation, we propose a novel vehicle SLAM method called \textit{N}atural Gr\textit{a}dient Gaussia\textit{n} Appr\textit{o}ximation (NANO)-SLAM, which avoids linearization errors by modeling the sampling distribution as the solution to an optimization problem over Gaussian parameters and solving it using natural gradient descent. This approach improves the accuracy of the sampling distribution and consequently enhances localization performance. Experimental results on the long-distance Sydney Victoria Park vehicle SLAM dataset show that NANO-SLAM achieves over 50\% improvement in localization accuracy compared to the most widely used vehicle SLAM algorithms, with minimal additional computational cost.
Abstract (translated)
准确的定位是自主车辆面临的挑战，特别是在没有GPS信号的环境中，如城市峡谷和隧道。在这种情况下，同时定位与地图构建（SLAM）提供了一个比基于GPS的位置服务更为稳健的选择，使车辆能够利用车载传感器和周围环境中的地标来确定自己的位置。在众多车辆SLAM方法中，Rao-Blackwellized粒子滤波器（RBPF）因其对地图大小具有对数复杂度的高效解决方案而被广泛采用。RBPF通过两个主要步骤——采样和重要性加权——利用一组蒙特卡洛粒子来近似车辆姿态的后验分布。有效采样的关键在于求解一个能够紧密逼近后验分布的分布，即采样分布，从而加速收敛过程。现有方法通常通过线性化导出该分布，但由于系统固有的非线性特性，这会导致显著的近似误差。 为了解决这一局限性，我们提出了一种新的车辆SLAM方法——自然梯度高斯逼近（NANO）-SLAM，这种方法通过将采样分布建模为一个关于高斯参数优化问题的解，并利用自然梯度下降法进行求解来避免线性化误差。这种做法提高了采样分布的准确性，从而提升了定位性能。 在长距离悉尼维多利亚公园车辆SLAM数据集上的实验结果显示，NANO-SLAM相较于最广泛使用的车辆SLAM算法，在定位精度上实现了超过50%的改进，并且几乎不需要增加额外的计算成本。
URL
https://arxiv.org/abs/2504.19195
PDF
-
Generalization Capability for Imitation Learning
2025-04-25 17:59:59Yixiao WangAbstract
Imitation learning holds the promise of equipping robots with versatile skills by learning from expert demonstrations. However, policies trained on finite datasets often struggle to generalize beyond the training distribution. In this work, we present a unified perspective on the generalization capability of imitation learning, grounded in both information theorey and data distribution property. We first show that the generalization gap can be upper bounded by (i) the conditional information bottleneck on intermediate representations and (ii) the mutual information between the model parameters and the training dataset. This characterization provides theoretical guidance for designing effective training strategies in imitation learning, particularly in determining whether to freeze, fine-tune, or train large pretrained encoders (e.g., vision-language models or vision foundation models) from scratch to achieve better generalization. Furthermore, we demonstrate that high conditional entropy from input to output induces a flatter likelihood landscape, thereby reducing the upper bound on the generalization gap. In addition, it shortens the stochastic gradient descent (SGD) escape time from sharp local minima, which may increase the likelihood of reaching global optima under fixed optimization budgets. These insights explain why imitation learning often exhibits limited generalization and underscore the importance of not only scaling the diversity of input data but also enriching the variability of output labels conditioned on the same input.
Abstract (translated)
模仿学习通过从专家演示中学习，承诺赋予机器人多种技能。然而，基于有限数据集训练的策略往往难以超越训练分布进行泛化。在这项工作中，我们提出了一种统一的观点来探讨模仿学习的泛化能力，该观点既以信息理论为依据，也考虑了数据分布特性。首先，我们表明泛化差距可以被上界限定为（i）中间表示上的条件信息瓶颈和（ii）模型参数与训练数据集之间的互信息。这一特征提供了设计有效训练策略的理论指导，在模仿学习中特别有助于确定是否冻结、微调或从头开始训练大规模预训练编码器（例如，视觉-语言模型或视觉基础模型），以实现更好的泛化能力。 此外，我们还证明了输入到输出方向上的高条件熵会诱导出更平坦的概率景观，从而减少泛化差距的上界。同时，它缩短了随机梯度下降（SGD）从尖锐局部最小值逃逸的时间，在固定优化预算下可能增加达到全局最优解的可能性。 这些见解解释了为什么模仿学习通常表现出有限的泛化能力，并强调不仅需要扩大输入数据的多样性，而且还需要丰富给定相同输入时输出标签的变化性。
URL
https://arxiv.org/abs/2504.18538
PDF
-
Gradient Descent as a Shrinkage Operator for Spectral Bias
Abstract
We generalize the connection between activation function and spline regression/smoothing and characterize how this choice may influence spectral bias within a 1D shallow network. We then demonstrate how gradient descent (GD) can be reinterpreted as a shrinkage operator that masks the singular values of a neural network's Jacobian. Viewed this way, GD implicitly selects the number of frequency components to retain, thereby controlling the spectral bias. An explicit relationship is proposed between the choice of GD hyperparameters (learning rate & number of iterations) and bandwidth (the number of active components). GD regularization is shown to be effective only with monotonic activation functions. Finally, we highlight the utility of non-monotonic activation functions (sinc, Gaussian) as iteration-efficient surrogates for spectral bias.
Abstract (translated)
我们扩展了激活函数与样条回归/平滑之间的联系，并描述了这种选择如何可能影响一维浅层网络中的谱偏差（spectral bias）。然后，我们展示了梯度下降（GD）可以重新解释为一种收缩算子，该算子屏蔽神经网络雅可比矩阵的奇异值。从这个角度来看，GD隐式地选择了要保留的频率分量的数量，从而控制了谱偏差。提出了梯度下降超参数的选择（学习率和迭代次数）与带宽（活动组件数量）之间的显式关系。证明了GD正则化仅在使用单调激活函数时有效。最后，我们强调非单调激活函数（如sinc和高斯）作为具有较高迭代效率的谱偏差替代方案的实用性。
URL
https://arxiv.org/abs/2504.18207
PDF
-
Gradient-Optimized Fuzzy Classifier: A Benchmark Study Against State-of-the-Art Models
2025-04-22 20:47:06Magnus Sieverding, Nathan Steffen, Kelly CohenAbstract
This paper presents a performance benchmarking study of a Gradient-Optimized Fuzzy Inference System (GF) classifier against several state-of-the-art machine learning models, including Random Forest, XGBoost, Logistic Regression, Support Vector Machines, and Neural Networks. The evaluation was conducted across five datasets from the UCI Machine Learning Repository, each chosen for their diversity in input types, class distributions, and classification complexity. Unlike traditional Fuzzy Inference Systems that rely on derivative-free optimization methods, the GF leverages gradient descent to significantly improving training efficiency and predictive performance. Results demonstrate that the GF model achieved competitive, and in several cases superior, classification accuracy while maintaining high precision and exceptionally low training times. In particular, the GF exhibited strong consistency across folds and datasets, underscoring its robustness in handling noisy data and variable feature sets. These findings support the potential of gradient optimized fuzzy systems as interpretable, efficient, and adaptable alternatives to more complex deep learning models in supervised learning tasks.
Abstract (translated)
本文介绍了一项针对梯度优化模糊推理系统（GF）分类器的性能基准研究，将其与几种最先进的机器学习模型进行了对比，包括随机森林、XGBoost、逻辑回归、支持向量机和神经网络。评估在来自UCI机器学习仓库的五个不同数据集上进行，这些数据集因输入类型多样、类别分布各异及分类难度的不同而被选中。 不同于依赖无导数优化方法的传统模糊推理系统，GF利用梯度下降显著提升了训练效率和预测性能。实验结果显示，在保持高精度的同时，GF模型实现了与其它基准模型相当甚至更优的分类准确率，并且具备极低的训练时间。此外，GF在各个折集（folds）和数据集上表现出了强一致性，这突显了其处理噪声数据及变化特征集合时的鲁棒性。 这些发现支持了梯度优化模糊系统作为解释性强、高效且适应性强的替代方案，可应用于监督学习任务中复杂的深度学习模型。
URL
https://arxiv.org/abs/2504.16263
PDF
-
Efficient Temporal Consistency in Diffusion-Based Video Editing with Adaptor Modules: A Theoretical Framework
2025-04-22 16:28:35Xinyuan Song, Yangfan He, Sida Li, Jianhui Wang, Hongyang He, Xinhang Yuan, Ruoyu Wang, Jiaqi Chen, Keqin Li, Kuan Lu, Menghao Huo, Binxu Li, Pei LiuAbstract
Adapter-based methods are commonly used to enhance model performance with minimal additional complexity, especially in video editing tasks that require frame-to-frame consistency. By inserting small, learnable modules into pretrained diffusion models, these adapters can maintain temporal coherence without extensive retraining. Approaches that incorporate prompt learning with both shared and frame-specific tokens are particularly effective in preserving continuity across frames at low training cost. In this work, we want to provide a general theoretical framework for adapters that maintain frame consistency in DDIM-based models under a temporal consistency loss. First, we prove that the temporal consistency objective is differentiable under bounded feature norms, and we establish a Lipschitz bound on its gradient. Second, we show that gradient descent on this objective decreases the loss monotonically and converges to a local minimum if the learning rate is within an appropriate range. Finally, we analyze the stability of modules in the DDIM inversion procedure, showing that the associated error remains controlled. These theoretical findings will reinforce the reliability of diffusion-based video editing methods that rely on adapter strategies and provide theoretical insights in video generation tasks.
Abstract (translated)
基于适配器的方法通常用于在最小增加复杂度的情况下增强模型性能，特别是在需要帧到帧一致性的视频编辑任务中。通过将小型可学习模块插入预训练的扩散模型中，这些适配器可以在无需大量重新训练的情况下保持时间一致性。结合提示学习（使用共享和特定于每一帧的标记）的方法特别有效，在低训练成本下跨帧维持连续性。 在这项工作中，我们希望为在DDIM（Denoising Diffusion Implicit Models）基于模型中保持帧一致性的适配器提供一个通用理论框架，并在此基础上引入时间一致性损失。首先，我们将证明当特征范数有界时，时间一致性目标是可微的，并且我们建立了其梯度上的Lipschitz边界。其次，我们展示在适当的学习率范围内进行梯度下降会使该目标的损失单调减少并最终收敛到局部最小值。最后，我们分析了DDIM反演过程中的模块稳定性，证明相关误差保持受控。 这些理论发现将增强基于扩散方法依赖适配器策略进行视频编辑技术的可靠性，并为视频生成任务提供理论见解。
URL
https://arxiv.org/abs/2504.16016
PDF
-
Cascade IPG Observer for Underwater Robot State Estimation
2025-04-21 17:10:10Kaustubh Joshi, Tianchen Liu, Nikhil ChopraAbstract
This paper presents a novel cascade nonlinear observer framework for inertial state estimation. It tackles the problem of intermediate state estimation when external localization is unavailable or in the event of a sensor outage. The proposed observer comprises two nonlinear observers based on a recently developed iteratively preconditioned gradient descent (IPG) algorithm. It takes the inputs via an IMU preintegration model where the first observer is a quaternion-based IPG. The output for the first observer is the input for the second observer, estimating the velocity and, consequently, the position. The proposed observer is validated on a public underwater dataset and a real-world experiment using our robot platform. The estimation is compared with an extended Kalman filter (EKF) and an invariant extended Kalman filter (InEKF). Results demonstrate that our method outperforms these methods regarding better positional accuracy and lower variance.
Abstract (translated)
本文提出了一种新的级联非线性观测器框架，用于惯性状态估计。该方法解决了在外部定位不可用或传感器故障情况下中间状态估计的问题。所提出的观测器由两个基于最近开发的迭代预条件梯度下降（IPG）算法的非线性观测器组成。它通过IMU（惯性测量单元）预积分模型接收输入，其中第一个观测器是基于四元数的IPG。第一个观测器的输出作为第二个观测器的输入，用于估计速度和位置。该提出的观测器在公开的水下数据集和我们机器人平台上的真实世界实验中进行了验证，并与扩展卡尔曼滤波（EKF）和不变扩展卡尔曼滤波（InEKF）方法进行比较。结果显示，我们的方法在位置精度和方差方面优于这些方法。
URL
https://arxiv.org/abs/2504.15235
PDF
-
HyperFlow: Gradient-Free Emulation of Few-Shot Fine-Tuning
2025-04-21 03:04:38Donggyun Kim, Chanwoo Kim, Seunghoon HongAbstract
While test-time fine-tuning is beneficial in few-shot learning, the need for multiple backpropagation steps can be prohibitively expensive in real-time or low-resource scenarios. To address this limitation, we propose an approach that emulates gradient descent without computing gradients, enabling efficient test-time adaptation. Specifically, we formulate gradient descent as an Euler discretization of an ordinary differential equation (ODE) and train an auxiliary network to predict the task-conditional drift using only the few-shot support set. The adaptation then reduces to a simple numerical integration (e.g., via the Euler method), which requires only a few forward passes of the auxiliary network -- no gradients or forward passes of the target model are needed. In experiments on cross-domain few-shot classification using the Meta-Dataset and CDFSL benchmarks, our method significantly improves out-of-domain performance over the non-fine-tuned baseline while incurring only 6\% of the memory cost and 0.02\% of the computation time of standard fine-tuning, thus establishing a practical middle ground between direct transfer and fully fine-tuned approaches.
Abstract (translated)
虽然测试时的微调在少量样本学习中是有益的，但在实时或资源受限场景下，需要进行多次反向传播步骤的成本可能过高。为了解决这一限制，我们提出了一种方法，该方法能够在不计算梯度的情况下模拟梯度下降，从而实现高效的测试时适应性调整。具体而言，我们将梯度下降表述为普通微分方程（ODE）的欧拉离散化，并训练一个辅助网络仅使用少量样本支持集来预测任务条件下的漂移。随后的适应过程简化为简单的数值积分（例如通过欧拉方法），这只需要进行几次辅助网络的前向传递——不需要计算梯度或目标模型的前向传递。 在跨域少量样本分类实验中，我们使用了Meta-Dataset和CDFSL基准测试，并且我们的方法显著提高了跨域性能，相比于不进行微调的基础线而言，在仅占用标准微调内存成本6%和0.02%计算时间的情况下实现了这一改进。因此，该方法在直接迁移与完全微调之间建立了一种实用的中间途径。 简言之，这项研究提供了一个有效的方法来优化少量样本学习中的测试时适应性调整过程，在减少资源消耗的同时提高跨域模型性能。
URL
https://arxiv.org/abs/2504.15323
PDF
-
Visual Prompting for One-shot Controllable Video Editing without Inversion
2025-04-19 16:00:47Zhengbo Zhang, Yuxi Zhou, Duo Peng, Joo-Hwee Lim, Zhigang Tu, De Wen Soh, Lin Geng FooAbstract
One-shot controllable video editing (OCVE) is an important yet challenging task, aiming to propagate user edits that are made -- using any image editing tool -- on the first frame of a video to all subsequent frames, while ensuring content consistency between edited frames and source frames. To achieve this, prior methods employ DDIM inversion to transform source frames into latent noise, which is then fed into a pre-trained diffusion model, conditioned on the user-edited first frame, to generate the edited video. However, the DDIM inversion process accumulates errors, which hinder the latent noise from accurately reconstructing the source frames, ultimately compromising content consistency in the generated edited frames. To overcome it, our method eliminates the need for DDIM inversion by performing OCVE through a novel perspective based on visual prompting. Furthermore, inspired by consistency models that can perform multi-step consistency sampling to generate a sequence of content-consistent images, we propose a content consistency sampling (CCS) to ensure content consistency between the generated edited frames and the source frames. Moreover, we introduce a temporal-content consistency sampling (TCS) based on Stein Variational Gradient Descent to ensure temporal consistency across the edited frames. Extensive experiments validate the effectiveness of our approach.
Abstract (translated)
一次性可控视频编辑（OCVE）是一项重要但具有挑战性的任务，旨在将用户在视频第一帧上使用任何图像编辑工具进行的修改传播到所有后续帧，并确保编辑后的帧与原始帧之间保持内容一致性。为了实现这一目标，先前的方法采用DDIM逆向过程将源帧转换为潜在噪声，然后将其输入预训练的扩散模型，在此模型中以用户编辑的第一帧作为条件来生成编辑后的视频。然而，DDIM逆向过程中会累积错误，这阻碍了潜在噪声准确重建原始帧的能力，最终影响了生成的编辑帧的内容一致性。 为了克服这一问题，我们的方法通过一种新的基于视觉提示的方法消除了对DDIM逆向过程的需求，并在此基础上执行OCVE。此外，受到能够进行多步一致性采样以生成一系列内容一致图像的一致性模型的启发，我们提出了内容一致性采样（CCS），以确保生成的编辑帧与源帧之间的内容一致性。另外，我们引入了一种基于Stein变分梯度下降的时间-内容一致性采样（TCS）方法，以确保编辑后的帧之间的时间连续性。大量的实验验证了我们这种方法的有效性。
URL
https://arxiv.org/abs/2504.14335
PDF
-
A mean teacher algorithm for unlearning of language models
2025-04-18 00:34:19Yegor KlochkovAbstract
One of the goals of language model unlearning is to reduce memorization of selected text instances while retaining the model's general abilities. Despite various proposed methods, reducing memorization of large datasets without noticeable degradation in model utility remains challenging. In this paper, we investigate the mean teacher algorithm (Tarvainen & Valpola, 2017), a simple proximal optimization method from continual learning literature that gradually modifies the teacher model. We show that the mean teacher can approximate a trajectory of a slow natural gradient descent (NGD), which inherently seeks low-curvature updates that are less likely to degrade the model utility. While slow NGD can suffer from vanishing gradients, we introduce a new unlearning loss called "negative log-unlikelihood" (NLUL) that avoids this problem. We show that the combination of mean teacher and NLUL improves some metrics on the MUSE benchmarks (Shi et al., 2024).
Abstract (translated)
语言模型“遗忘”（unlearning）的一个目标是减少对选定文本实例的记忆，同时保持模型的一般能力。尽管已提出了多种方法，但在不显著降低模型实用性的情况下减少大型数据集的内存仍是一个挑战。在本文中，我们研究了均值教师算法（Tarvainen & Valpola, 2017），这是一种源自连续学习文献中的简单邻近优化方法，它逐步修改教师模型。我们展示了均值教师可以逼近慢速自然梯度下降（NGD）的轨迹，而这种更新本质上寻求低曲率更新，这些更新不太可能降低模型实用性。虽然慢速NGD可能会遇到消失的梯度问题，但我们引入了一种新的遗忘损失方法，称为“负对数非可能性”（NLUL），它可以避免这个问题。我们展示了均值教师与NLUL结合后，在MUSE基准测试（Shi et al., 2024）中的某些指标有所提高。
URL
https://arxiv.org/abs/2504.13388
PDF
