Gradient_Descent
-
Decoupling Search and Learning in Neural Net Training
Abstract
Gradient descent typically converges to a single minimum of the training loss without mechanisms to explore alternative minima that may generalize better. Searching for diverse minima directly in high-dimensional parameter space is generally intractable. To address this, we propose a framework that performs training in two distinct phases: search in a tractable representation space (the space of intermediate activations) to find diverse representational solutions, and gradient-based learning in parameter space by regressing to those searched representations. Through evolutionary search, we discover representational solutions whose fitness and diversity scale with compute--larger populations and more generations produce better and more varied solutions. These representations prove to be learnable: networks trained by regressing to searched representations approach SGD's performance on MNIST, CIFAR-10, and CIFAR-100. Performance improves with search compute up to saturation. The resulting models differ qualitatively from networks trained with gradient descent, following different representational trajectories during training. This work demonstrates how future training algorithms could overcome gradient descent's exploratory limitations by decoupling search in representation space from efficient gradient-based learning in parameter space.
Abstract (translated)
梯度下降通常会收敛到训练损失中的单一最小值，而没有探索可能更好泛化的替代极小值的机制。在高维参数空间中直接搜索多样性的极小值通常是不可行的。为了解决这个问题，我们提出了一种框架，在两个不同的阶段进行训练：首先在可处理的表现形式空间（中间激活的空间）中寻找多样化的表示解；然后通过基于梯度的学习方法在参数空间中回归到这些找到的表示上进行学习。 通过进化搜索，我们可以发现那些其适应性和多样性与计算资源成比例增长的表示解——更大的种群和更多的世代会产生更好且更加多样的解决方案。这些表示证明是可学得的：通过回归至搜索到的表示来训练的网络在MNIST、CIFAR-10和CIFAR-100数据集上的性能与随机梯度下降法（SGD）相当。随着搜索计算资源的增加，性能会提升直至饱和。 由此产生的模型在训练过程中遵循不同的表示轨迹，这区别于通过梯度下降方法训练得到的网络。这项工作展示了未来训练算法如何能够通过将表示空间中的搜索和参数空间中基于梯度的学习解耦来克服梯度下降探索上的限制。
URL
https://arxiv.org/abs/2509.10973
PDF
-
Sound Matching an Analogue Levelling Amplifier Using the Newton-Raphson Method
Abstract
Automatic differentiation through digital signal processing algorithms for virtual analogue modelling has recently gained popularity. These algorithms are typically more computationally efficient than black-box neural networks that rely on dense matrix multiplications. Due to their differentiable nature, they can be integrated with neural networks and jointly trained using gradient descent algorithms, resulting in more efficient systems. Furthermore, signal processing algorithms have significantly fewer parameters than neural networks, allowing the application of the Newton-Raphson method. This method offers faster and more robust convergence than gradient descent at the cost of quadratic storage. This paper presents a method to emulate analogue levelling amplifiers using a feed-forward digital compressor with parameters optimised via the Newton-Raphson method. We demonstrate that a digital compressor can successfully approximate the behaviour of our target unit, the Teletronix LA-2A. Different strategies for computing the Hessian matrix are benchmarked. We leverage parallel algorithms for recursive filters to achieve efficient training on modern GPUs. The resulting model is made into a VST plugin and is open-sourced at this https URL.
Abstract (translated)
最近，通过数字信号处理算法进行自动微分以实现虚拟模拟建模的方法变得越来越受欢迎。这些算法通常比依赖密集矩阵乘法的黑盒神经网络更具有计算效率。由于它们是可微的，因此可以与神经网络集成，并使用梯度下降算法联合训练，从而生成更为高效的系统。此外，信号处理算法相较于神经网络拥有较少的参数量，这使得能够应用牛顿-拉夫森方法。此方法虽然以二次存储为代价，但比梯度下降法提供更快且更稳健的收敛速度。 本文提出了一种使用前馈数字压缩器来模拟类比电平放大器的方法，并利用牛顿-拉夫森方法优化参数。我们展示了通过这种方式，一个数字压缩器可以成功地近似目标单元——Teletronix LA-2A的行为表现。文章还对计算海森矩阵的不同策略进行了基准测试。我们利用递归滤波器的并行算法来实现在现代GPU上的高效训练。最终模型被制作成VST插件，并在以下网址开源：[此链接](https://这个URL需要你根据实际情况提供)。 请注意，上述翻译中的最后一句话提到的链接应当由原文作者或发布者提供一个具体的、可访问的实际地址。
URL
https://arxiv.org/abs/2509.10706
PDF
-
Privacy-Preserving Decentralized Federated Learning via Explainable Adaptive Differential Privacy
2025-09-12 20:52:41Fardin Jalil Piran, Zhiling Chen, Yang Zhang, Qianyu Zhou, Jiong Tang, Farhad ImaniAbstract
Decentralized federated learning faces privacy risks because model updates can leak data through inference attacks and membership inference, a concern that grows over many client exchanges. Differential privacy offers principled protection by injecting calibrated noise so confidential information remains secure on resource-limited IoT devices. Yet without transparency, black-box training cannot track noise already injected by previous clients and rounds, which forces worst-case additions and harms accuracy. We propose PrivateDFL, an explainable framework that joins hyperdimensional computing with differential privacy and keeps an auditable account of cumulative noise so each client adds only the difference between the required noise and what has already been accumulated. We evaluate on MNIST, ISOLET, and UCI-HAR to span image, signal, and tabular modalities, and we benchmark against transformer-based and deep learning-based baselines trained centrally with Differentially Private Stochastic Gradient Descent (DP-SGD) and Renyi Differential Privacy (RDP). PrivateDFL delivers higher accuracy, lower latency, and lower energy across IID and non-IID partitions while preserving formal (epsilon, delta) guarantees and operating without a central server. For example, under non-IID partitions, PrivateDFL achieves 24.42% higher accuracy than the Vision Transformer on MNIST while using about 10x less training time, 76x lower inference latency, and 11x less energy, and on ISOLET it exceeds Transformer accuracy by more than 80% with roughly 10x less training time, 40x lower inference latency, and 36x less training energy. Future work will extend the explainable accounting to adversarial clients and adaptive topologies with heterogeneous privacy budgets.
Abstract (translated)
去中心化的联邦学习由于模型更新可能通过推理攻击和成员推断等方式泄露数据，因此面临着隐私风险。随着多个客户端之间的交互增加，这种担忧也在加剧。差分隐私（Differential Privacy）提供了一种原则性的保护方法，通过注入校准噪声来确保在资源受限的物联网设备上保密信息的安全性。然而，在缺乏透明度的情况下，黑盒训练无法追踪之前客户端和轮次中已经注入的噪声，这迫使每次必须增加最坏情况下的噪音量，并损害了准确性。 我们提出了一种名为PrivateDFL的可解释框架，该框架结合了超维度计算（Hyperdimensional Computing）与差分隐私技术，并维护了一个可以审计的累计噪声记录，使得每个客户端只需添加所需噪声与其之前累积噪声之间的差异部分。我们在MNIST、ISOLET和UCI-HAR数据集上进行了测试，涵盖图像、信号和表格模态，并将PrivateDFL与其他基于转换器（Transformer）和深度学习且使用中心化训练的Differentially Private Stochastic Gradient Descent (DP-SGD) 和 Renyi Differential Privacy (RDP) 方法进行基准比较。结果表明，在保持正式的(epsilon, delta)保证的同时，PrivateDFL在IID和非IID数据分区上都提供了更高的准确率、更低的延迟和能耗，并且无需中央服务器支持。 具体而言，在处理非IID数据分区时，与视觉转换器相比，PrivateDFL在MNIST数据集上的准确性提高了24.42%，训练时间减少了约10倍，推理延迟降低了76倍，训练能量消耗减少了11倍；而在ISOLET数据集上，PrivateDFL的准确率超出转换器模型80%以上，并且训练时间减少大约10倍、推理延迟降低40倍以及训练能耗减少36倍。 未来的工作计划将可解释性账目扩展到对抗客户端和自适应拓扑结构中具有异构隐私预算的情况。
URL
https://arxiv.org/abs/2509.10691
PDF
-
zkUnlearner: A Zero-Knowledge Framework for Verifiable Unlearning with Multi-Granularity and Forgery-Resistance
2025-09-08 23:50:35Nan Wang, Nan Wu, Xiangyu Hui, Jiafan Wang, Xin YuanAbstract
As the demand for exercising the "right to be forgotten" grows, the need for verifiable machine unlearning has become increasingly evident to ensure both transparency and accountability. We present {\em zkUnlearner}, the first zero-knowledge framework for verifiable machine unlearning, specifically designed to support {\em multi-granularity} and {\em forgery-resistance}. First, we propose a general computational model that employs a {\em bit-masking} technique to enable the {\em selectivity} of existing zero-knowledge proofs of training for gradient descent algorithms. This innovation enables not only traditional {\em sample-level} unlearning but also more advanced {\em feature-level} and {\em class-level} unlearning. Our model can be translated to arithmetic circuits, ensuring compatibility with a broad range of zero-knowledge proof systems. Furthermore, our approach overcomes key limitations of existing methods in both efficiency and privacy. Second, forging attacks present a serious threat to the reliability of unlearning. Specifically, in Stochastic Gradient Descent optimization, gradients from unlearned data, or from minibatches containing it, can be forged using alternative data samples or minibatches that exclude it. We propose the first effective strategies to resist state-of-the-art forging attacks. Finally, we benchmark a zkSNARK-based instantiation of our framework and perform comprehensive performance evaluations to validate its practicality.
Abstract (translated)
随着人们对于“被遗忘权”的需求增加，确保透明度和问责制的可验证机器遗忘技术的重要性也日益凸显。我们提出了一种名为{\em zkUnlearner}的零知识框架，这是第一个专门用于支持{\em多粒度}和{\em防伪造}的可验证机器遗忘系统。 首先，我们提出了一个通用计算模型，并使用一种称为{\em位掩码}的技术来使现有的针对梯度下降算法训练过程的零知识证明具备选择性。这一创新不仅能够实现传统的基于样本级别的遗忘，还支持更为高级的特征级别和类别级别遗忘。我们的模型可以转换为算术电路，确保与广泛范围内的零知识证明系统兼容。此外，我们方法在效率和隐私方面克服了现有技术的关键限制。 其次，伪造攻击对遗忘过程的可靠性构成严重威胁。特别是在随机梯度下降优化过程中，来自被遗忘数据或包含此类数据的小批量中的梯度可以通过使用不包含这些数据样本或小批量的数据来伪造。我们提出了有效抵抗当前最先进的伪造攻击的战略。 最后，我们采用基于zkSNARK（零知识简洁非交互式知识论证）的实例化框架，并进行了全面性能评估以验证其实际应用价值。
URL
https://arxiv.org/abs/2509.07290
PDF
-
Intraoperative 2D/3D Registration via Spherical Similarity Learning and Inference-Time Differentiable Levenberg-Marquardt Optimization
2025-09-08 17:10:43Minheng Chen, Youyong KongAbstract
Intraoperative 2D/3D registration aligns preoperative 3D volumes with real-time 2D radiographs, enabling accurate localization of instruments and implants. A recent fully differentiable similarity learning framework approximates geodesic distances on SE(3), expanding the capture range of registration and mitigating the effects of substantial disturbances, but existing Euclidean approximations distort manifold structure and slow convergence. To address these limitations, we explore similarity learning in non-Euclidean spherical feature spaces to better capture and fit complex manifold structure. We extract feature embeddings using a CNN-Transformer encoder, project them into spherical space, and approximate their geodesic distances with Riemannian distances in the bi-invariant SO(4) space. This enables a more expressive and geometrically consistent deep similarity metric, enhancing the ability to distinguish subtle pose differences. During inference, we replace gradient descent with fully differentiable Levenberg-Marquardt optimization to accelerate convergence. Experiments on real and synthetic datasets show superior accuracy in both patient-specific and patient-agnostic scenarios.
Abstract (translated)
术中2D/3D配准将术前的三维体积与实时二维影像对齐，使得手术器械和植入物的位置能够被精确地定位。最近提出的一种完全可微相似性学习框架，在SE(3)上近似测地距离，从而扩大了注册范围并减轻了重大干扰的影响；然而，现有的欧几里得近似方法会扭曲流形结构，并减缓收敛速度。为了解决这些问题，我们探索在非欧几里得球面特征空间中进行相似性学习，以更好地捕捉和拟合复杂的流形结构。我们使用CNN-Transformer编码器提取特征嵌入，将其投影到球面上，并通过SO(4)双不变量空间中的黎曼距离来近似它们的测地距离。这使得深度相似度度量更具表现力且几何一致性更强，从而增强了区分细微姿态差异的能力。在推理过程中，我们用完全可微的Levenberg-Marquardt优化方法替代梯度下降法，以加快收敛速度。实验证明，在真实和合成数据集上进行测试时，无论是患者特定的情况还是无差别情况，该方法均表现出更高的准确性。
URL
https://arxiv.org/abs/2509.06890
PDF
-
Evaluating the Impact of Adversarial Attacks on Traffic Sign Classification using the LISA Dataset
2025-09-08 16:06:41Nabeyou Tadessa, Balaji Iyangar, Mashrur ChowdhuryAbstract
Adversarial attacks pose significant threats to machine learning models by introducing carefully crafted perturbations that cause misclassification. While prior work has primarily focused on MNIST and similar datasets, this paper investigates the vulnerability of traffic sign classifiers using the LISA Traffic Sign dataset. We train a convolutional neural network to classify 47 different traffic signs and evaluate its robustness against Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) attacks. Our results show a sharp decline in classification accuracy as the perturbation magnitude increases, highlighting the models susceptibility to adversarial examples. This study lays the groundwork for future exploration into defense mechanisms tailored for real-world traffic sign recognition systems.
Abstract (translated)
对抗性攻击通过引入精心设计的扰动来误导机器学习模型，从而对这些模型构成重大威胁。虽然之前的研究主要集中在MNIST和类似的数据库上，但本文探讨了使用LISA交通标志数据集评估交通标志分类器的安全漏洞问题。我们训练了一个卷积神经网络来识别47种不同的交通标志，并用快速梯度符号方法（FGSM）和投影梯度下降法（PGD）攻击对其进行鲁棒性测试。我们的结果显示，随着扰动大小的增加，分类准确率急剧下降，这凸显了模型对于对抗样本的脆弱性。这项研究为未来开发针对现实世界交通标志识别系统的防御机制奠定了基础。
URL
https://arxiv.org/abs/2509.06835
PDF
-
MachineLearningLM: Continued Pretraining Language Models on Millions of Synthetic Tabular Prediction Tasks Scales In-Context ML
2025-09-08 15:38:31Haoyu Dong, Pengkun Zhang, Mingzhe Lu, Yanzhen Shen, Guolin KeAbstract
Large language models (LLMs) possess broad world knowledge and strong general-purpose reasoning ability, yet they struggle to learn from many in-context examples on standard machine learning (ML) tasks, that is, to leverage many-shot demonstrations purely via in-context learning (ICL) without gradient descent. We introduce MachineLearningLM, a portable continued-pretraining framework that equips a general-purpose LLM with robust in-context ML capability while preserving its general knowledge and reasoning for broader chat workflows. Our pretraining procedure synthesizes ML tasks from millions of structural causal models (SCMs), spanning shot counts up to 1,024. We begin with a random-forest teacher, distilling tree-based decision strategies into the LLM to strengthen robustness in numerical modeling. All tasks are serialized with a token-efficient prompt, enabling 3x to 6x more examples per context window and delivering up to 50x amortized throughput via batch inference. Despite a modest setup (Qwen-2.5-7B-Instruct with LoRA rank 8), MachineLearningLM outperforms strong LLM baselines (e.g., GPT-5-mini) by an average of about 15% on out-of-distribution tabular classification across finance, physics, biology, and healthcare domains. It exhibits a striking many-shot scaling law: accuracy increases monotonically as in-context demonstrations grow from 8 to 1,024. Without any task-specific training, it attains random-forest-level accuracy across hundreds of shots. General chat capabilities, including knowledge and reasoning, are preserved: it achieves 75.4% on MMLU.
Abstract (translated)
大型语言模型（LLMs）拥有广泛的世界知识和强大的通用推理能力，但它们在标准机器学习（ML）任务上难以从许多上下文示例中进行学习，即无法通过纯上下文学习（ICL）而不使用梯度下降方法来利用多示例演示。我们引入了MachineLearningLM，这是一个便携式的持续预训练框架，它为通用的LLM配备了稳健的上下文机器学习能力，同时保留了其广泛的常识和推理能力，适用于更广泛的任务流程。 我们的预训练过程从数百万个结构因果模型（SCMs）中合成ML任务，并扩展到1,024个示例数量。我们首先使用随机森林教师将基于树的决策策略提炼进LLM，以增强数值建模中的鲁棒性。所有任务都使用一个高效的提示进行序列化，这使得每个上下文窗口内的示例数量最多可增加3至6倍，并通过批量推理实现了最高50倍的平均吞吐量。 尽管采用了适度配置（如Qwen-2.5-7B-Instruct和LoRA秩8），MachineLearningLM在跨金融、物理、生物及医疗领域分布外表格分类任务上，相对于强大的LLM基线模型（例如GPT-5-mini）表现出约15%的平均性能提升。它还展示了显著的多示例扩展法则：随着上下文演示从8个增加到1,024个，准确率单调上升；无需特定任务训练，在数百个示例的情况下也能达到随机森林级别的精度。 此外，其通用聊天能力（包括知识和推理）得到了保持：在MMLU基准测试中，它达到了75.4%的分数。
URL
https://arxiv.org/abs/2509.06806
PDF
-
Reasoning-Intensive Regression
2025-08-29 16:37:42Diane Tchuindjo, Omar KhattabAbstract
AI researchers and practitioners increasingly apply large language models (LLMs) to what we call reasoning-intensive regression (RiR), i.e. deducing subtle numerical properties from text. Unlike standard language regression tasks, e.g. for sentiment or similarity, RiR often appears instead in ad-hoc problems like rubric-based scoring or domain-specific retrieval, where much deeper analysis of text is required while only limited task-specific training data and computation are available. We cast three realistic problems as RiR tasks to establish an initial benchmark, and use that to test our hypothesis that prompting frozen LLMs and finetuning Transformer encoders via gradient descent will both often struggle in RiR. We then propose MENTAT, a simple and lightweight method that combines batch-reflective prompt optimization with neural ensemble learning. MENTAT achieves up to 65% improvement over both baselines, though substantial room remains for future advances in RiR.
Abstract (translated)
AI研究人员和实践者越来越多地将大型语言模型（LLMs）应用于我们所说的推理密集型回归（RiR），即从文本中推断出细微的数值属性。与标准的语言回归任务，如情感分析或相似度评估不同，RiR通常出现在一些临时性问题上，例如基于评分细则的评分或特定领域的检索，在这些问题中需要对文本进行更深入的分析，而同时可用的任务特定训练数据和计算资源却非常有限。我们把三个现实中的问题作为RiR任务来建立初步基准，并利用这些任务测试我们的假设：即通过提示冻结的语言模型以及通过梯度下降微调变换器编码器在处理RiR时常常会遇到困难。然后，我们提出了MENTAT方法，这是一种简单而轻量级的方法，结合了批量自省的提示优化与神经集成学习。相比基准线，MENTAT实现了高达65%的改进，尽管未来在RiR领域仍然存在很大的进步空间。
URL
https://arxiv.org/abs/2508.21762
PDF
-
Unfolding Framework with Complex-Valued Deformable Attention for High-Quality Computer-Generated Hologram Generation
2025-08-29 14:21:22Haomiao Zhang, Zhangyuan Li, Yanling Piao, Zhi Li, Xiaodong Wang, Miao Cao, Xiongfei Su, Qiang Song, Xin YuanAbstract
Computer-generated holography (CGH) has gained wide attention with deep learning-based algorithms. However, due to its nonlinear and ill-posed nature, challenges remain in achieving accurate and stable reconstruction. Specifically, ($i$) the widely used end-to-end networks treat the reconstruction model as a black box, ignoring underlying physical relationships, which reduces interpretability and flexibility. ($ii$) CNN-based CGH algorithms have limited receptive fields, hindering their ability to capture long-range dependencies and global context. ($iii$) Angular spectrum method (ASM)-based models are constrained to finite this http URL this paper, we propose a Deep Unfolding Network (DUN) that decomposes gradient descent into two modules: an adaptive bandwidth-preserving model (ABPM) and a phase-domain complex-valued denoiser (PCD), providing more flexibility. ABPM allows for wider working distances compared to ASM-based methods. At the same time, PCD leverages its complex-valued deformable self-attention module to capture global features and enhance performance, achieving a PSNR over 35 dB. Experiments on simulated and real data show state-of-the-art results.
Abstract (translated)
计算机生成全息图（CGH）由于基于深度学习的算法而受到广泛关注。然而，由于其非线性和不适定特性，实现准确且稳定的重建仍面临挑战。具体来说： (i) 广泛使用的端到端网络将重建模型视为黑盒模型，忽略了潜在的物理关系，降低了可解释性与灵活性。 (ii) 基于卷积神经网络（CNN）的CGH算法具有有限的感受野，难以捕捉长距离依赖和全局上下文信息。 (iii) 基于角谱法（ASM）的方法受到计算域限制。本文中，我们提出了一种深度展开网络（DUN），将梯度下降分解为两个模块：自适应带宽保持模型（ABPM）和相位域复值去噪器（PCD）。这种方法提供了更高的灵活性。 ABPM 允许与基于 ASM 的方法相比具有更宽的工作距离。同时，PCD 通过其复数可变形的自我注意机制来捕获全局特征并增强性能，实现了超过35dB的PSNR。在模拟和真实数据上的实验显示了最先进的结果。
URL
https://arxiv.org/abs/2508.21657
PDF
-
Adaptive Heavy-Tailed Stochastic Gradient Descent
2025-08-29 06:32:26Bodu Gong, Gustavo Enrique Batista, Pierre Lafaye de MicheauxAbstract
In the era of large-scale neural network models, optimization algorithms often struggle with generalization due to an overreliance on training loss. One key insight widely accepted in the machine learning community is the idea that wide basins (regions around a local minimum where the loss increases gradually) promote better generalization by offering greater stability to small changes in input data or model parameters. In contrast, sharp minima are typically more sensitive and less stable. Motivated by two key empirical observations - the inherent heavy-tailed distribution of gradient noise in stochastic gradient descent and the Edge of Stability phenomenon during neural network training, in which curvature grows before settling at a plateau, we introduce Adaptive Heavy Tailed Stochastic Gradient Descent (AHTSGD). The algorithm injects heavier-tailed noise into the optimizer during the early stages of training to enhance exploration and gradually transitions to lighter-tailed noise as sharpness stabilizes. By dynamically adapting to the sharpness of the loss landscape throughout training, AHTSGD promotes accelerated convergence to wide basins. AHTSGD is the first algorithm to adjust the nature of injected noise into an optimizer based on the Edge of Stability phenomenon. AHTSGD consistently outperforms SGD and other noise-based methods on benchmarks like MNIST and CIFAR-10, with marked gains on noisy datasets such as SVHN. It ultimately accelerates early training from poor initializations and improves generalization across clean and noisy settings, remaining robust to learning rate choices.
Abstract (translated)
在大规模神经网络模型的时代，优化算法由于过度依赖训练损失而常常难以实现泛化。机器学习社区广泛接受的一个关键见解是：宽盆地（即围绕局部最小值的区域，在该区域内损失逐渐增加）通过提供对输入数据或模型参数的小变化更稳定的特性来促进更好的泛化能力；相比之下，尖锐极小值通常更加敏感且不够稳定。 受到两个重要经验观察结果的启发——随机梯度下降中的梯度噪声呈现固有的重尾分布以及在神经网络训练过程中边缘稳定性现象的存在，在这一现象中曲率会在达到平台期之前先增长。我们提出了自适应重尾随机梯度下降（AHTSGD）。该算法在训练初期向优化器注入更重尾的噪声以增强探索，并随着尖锐性的稳定逐渐过渡到较轻尾的噪声。通过在整个训练过程中动态地调整注入噪声的性质，AHTSGD促进了加速收敛于宽盆地。 AHTSGD是首个根据边缘稳定性现象来调整输入给优化器中的噪声特性的算法。在MNIST和CIFAR-10等基准测试中，AHTSGD始终优于随机梯度下降（SGD）和其他基于噪声的方法，在SVHN这样的嘈杂数据集上表现出显著的改进效果。它最终加快了从较差初始化开始时的早期训练速度，并在整个清洁与嘈杂环境设置下提高了泛化能力，同时对学习率的选择保持鲁棒性。
URL
https://arxiv.org/abs/2508.21353
PDF
-
Observer Design for Optical Flow-Based Visual-Inertial Odometry with Almost-Global Convergence
2025-08-28 18:55:16Tarek Bouazza, Soulaimane Berkane, Minh-Duc Hua, Tarek HamelAbstract
This paper presents a novel cascaded observer architecture that combines optical flow and IMU measurements to perform continuous monocular visual-inertial odometry (VIO). The proposed solution estimates body-frame velocity and gravity direction simultaneously by fusing velocity direction information from optical flow measurements with gyro and accelerometer data. This fusion is achieved using a globally exponentially stable Riccati observer, which operates under persistently exciting translational motion conditions. The estimated gravity direction in the body frame is then employed, along with an optional magnetometer measurement, to design a complementary observer on $\mathbf{SO}(3)$ for attitude estimation. The resulting interconnected observer architecture is shown to be almost globally asymptotically stable. To extract the velocity direction from sparse optical flow data, a gradient descent algorithm is developed to solve a constrained minimization problem on the unit sphere. The effectiveness of the proposed algorithms is validated through simulation results.
Abstract (translated)
这篇论文提出了一种新颖的级联观测器架构，该架构结合了光流和惯性测量单元（IMU）的数据来执行连续单目视觉惯性里程计（VIO）。所提出的解决方案通过融合来自光流测量的速度方向信息与陀螺仪和加速度计数据，在同一个框架内同时估计机体坐标系中的速度和重力方向。这种融合是利用一个在持续激励平移运动条件下全局指数稳定的里卡蒂观测器来实现的。 然后，以机体系中估算出的重力方向为基础，并可选择性地结合磁力计测量值，设计了一个定义在$\mathbf{SO}(3)$上的互补观测器来进行姿态估计。所得到的互联观测器架构被证明是几乎全局渐近稳定的。 为了从稀疏光流数据中提取速度方向，开发了一种梯度下降算法来解决单位球面上的一个约束最小化问题。通过仿真结果验证了所提出的算法的有效性。
URL
https://arxiv.org/abs/2508.21163
PDF
-
Self-Supervised Pre-Training with Equilibrium Constraints
2025-08-27 15:48:50Xiaodong Cui, A F M Saif, Brian Kingsbury, Tianyi ChenAbstract
Self-supervised pre-training using unlabeled data is widely used in machine learning. In this paper, we propose a new self-supervised pre-training approach to dealing with heterogeneous data. Instead of mixing all the data and minimizing the averaged global loss in the conventional way, we impose additional equilibrium constraints to ensure that the models optimizes each source of heterogeneous data to its local optima after $K$-step gradient descent initialized from the model. We formulate this as a bilevel optimization problem, and use the first-order approximation method to solve the problem. We discuss its connection to model-agnostic meta learning (MAML). Experiments are carried out on self-supervised pre-training using multi-domain and multilingual datasets, demonstrating that the proposed approach can significantly improve the adaptivity of the self-supervised pre-trained model for the downstream supervised fine-tuning tasks.
Abstract (translated)
无监督预训练在使用未标记数据的机器学习中广泛应用。本文提出了一种新的自监督预训练方法，用于处理异构数据。不同于传统方法将所有数据混合并最小化平均全局损失，我们引入了额外的平衡约束条件，以确保模型从初始化开始经过 $K$ 步梯度下降后能够优化每一种异构数据到其局部最优解。我们将此问题形式化为双层优化问题，并使用一阶近似方法来求解。讨论了该方法与无模型元学习（MAML）的联系。在多领域和多语言数据集上的自监督预训练实验表明，所提出的这种方法可以显著提高自监督预训练模型对下游有监督微调任务的适应性。
URL
https://arxiv.org/abs/2508.19990
PDF
-
Attention is also needed for form design
2025-08-27 09:15:21B. Sankar, Dibakar SenAbstract
Conventional product design is a cognitively demanding process, limited by its time-consuming nature, reliance on subjective expertise, and the opaque translation of inspiration into tangible concepts. This research introduces a novel, attention-aware framework that integrates two synergistic systems: EUPHORIA, an immersive Virtual Reality environment using eye-tracking to implicitly capture a designer's aesthetic preferences, and RETINA, an agentic AI pipeline that translates these implicit preferences into concrete design outputs. The foundational principles were validated in a two-part study. An initial study correlated user's implicit attention with explicit preference and the next one correlated mood to attention. A comparative study where 4 designers solved challenging design problems using 4 distinct workflows, from a manual process to an end-to-end automated pipeline, showed the integrated EUPHORIA-RETINA workflow was over 4 times more time-efficient than the conventional method. A panel of 50 design experts evaluated the 16 final renderings. Designs generated by the fully automated system consistently received the highest Worthiness (calculated by an inverse Plackett-Luce model based on gradient descent optimization) and Design Effectiveness scores, indicating superior quality across 8 criteria: novelty, visual appeal, emotional resonance, clarity of purpose, distinctiveness of silhouette, implied materiality, proportional balance, & adherence to the brief. This research presents a validated paradigm shift from traditional Computer-Assisted Design (CAD) to a collaborative model of Designer-Assisting Computers (DAC). By automating logistical and skill-dependent generative tasks, the proposed framework elevates the designer's role to that of a creative director, synergizing human intuition with the generative power of agentic AI to produce higher-quality designs more efficiently.
Abstract (translated)
传统的商品设计是一个认知负担重的过程，受限于其耗时性、依赖主观专业知识以及将灵感转化为具体概念的不透明过程。这项研究介绍了一种新颖且具有注意力感知能力的框架，整合了两个协同系统：EUPHORIA和RETINA。其中，EUPHORIA是一个利用眼动追踪技术捕捉设计师审美偏好的沉浸式虚拟现实环境；而RETINA则是一条代理人工智能管道，能够将这些隐性的偏好转化为具体的设 计输出。 该基础原理在两项研究中得到了验证：一项初步研究将用户隐性注意力与显性偏好进行了关联，另一项研究将情绪和注意力相关联。在一项比较研究中，4位设计师使用了四种不同的工作流程来解决设计难题，从手动过程到端到端的自动化管道，结果显示集成的EUPHORIA-RETINA工作流比传统方法的时间效率高出四倍以上。 一个由50名设计专家组成的评审团对16个最终渲染进行了评估。全自动化系统生成的设计在新颖性、视觉吸引力、情感共鸣、目标清晰度、轮廓独特性、材料暗示、比例平衡以及符合设计简报的八个标准上，持续获得最高的“价值”（通过基于梯度下降优化的逆Plackett-Luce模型计算）和“设计有效性”评分，表明其质量明显优于传统方法。 这项研究提出了一种从传统的计算机辅助设计(CAD)转向设计师与计算机协作模式的设计范式转变。通过自动化物流性和技能依赖性的生成任务，该框架提升了设计师的角色至创意总监，并将人类的直觉与代理人工智能的生成能力相结合，以更高效地创造出更高品质的设计。
URL
https://arxiv.org/abs/2508.19708
PDF
-
A Symbolic Adversarial Learning Framework for Evolving Fake News Generation and Detection
2025-08-27 07:14:17Chong Tian, Qirong Ho, Xiuying ChenAbstract
Rapid LLM advancements heighten fake news risks by enabling the automatic generation of increasingly sophisticated misinformation. Previous detection methods, including fine-tuned small models or LLM-based detectors, often struggle with its dynamically evolving nature. In this work, we propose a novel framework called the Symbolic Adversarial Learning Framework (SALF), which implements an adversarial training paradigm by an agent symbolic learning optimization process, rather than relying on numerical updates. SALF introduces a paradigm where the generation agent crafts deceptive narratives, and the detection agent uses structured debates to identify logical and factual flaws for detection, and they iteratively refine themselves through such adversarial interactions. Unlike traditional neural updates, we represent agents using agent symbolic learning, where learnable weights are defined by agent prompts, and simulate back-propagation and gradient descent by operating on natural language representations of weights, loss, and gradients. Experiments on two multilingual benchmark datasets demonstrate SALF's effectiveness, showing it generates sophisticated fake news that degrades state-of-the-art detection performance by up to 53.4% in Chinese and 34.2% in English on average. SALF also refines detectors, improving detection of refined content by up to 7.7%. We hope our work inspires further exploration into more robust, adaptable fake news detection systems.
Abstract (translated)
快速发展的大型语言模型（LLM）技术提高了虚假新闻的风险，因为它可以自动产生越来越复杂的误导信息。以往的检测方法，包括微调的小型模型或基于LLM的检测器，往往难以应对这种动态变化的特性。在这项工作中，我们提出了一种新的框架——符号对抗学习框架（SALF），该框架通过一个代理符号学习优化过程实施了对抗性训练范式，而不是依赖数值更新。在SALF中，生成者创建误导性的叙述，而检测器则使用结构化的辩论来识别逻辑和事实上的错误进行检测；两者通过这种对抗性互动不断自我完善。 与传统的神经网络更新不同，我们用代理符号学习表示代理，其中可学习的权重由代理提示定义，并通过操作自然语言表达形式的权重、损失和梯度来模拟反向传播和梯度下降。在两个多语言基准数据集上的实验表明，SALF表现出色：它生成了复杂且具有欺骗性的虚假新闻，在中文上降低了最先进的检测性能平均高达53.4%，在英文上则为34.2%。同时，SALF还改进了探测器的准确性，提高了对经过优化的内容进行识别的能力最多可达7.7%。 我们希望我们的工作能够激发更多关于更强大、适应性更强的虚假新闻检测系统的研究探索。
URL
https://arxiv.org/abs/2508.19633
PDF
-
Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Models
2025-08-27 04:46:56Yuhang Liu, Tao Li, Zhehao Huang, Zuopeng Yang, Xiaolin HuangAbstract
Fine-tuning large-scale pre-trained models with limited data presents significant challenges for generalization. While Sharpness-Aware Minimization (SAM) has proven effective in improving generalization by seeking flat minima, its substantial extra memory and computation overhead make it impractical for large models. Integrating SAM with parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA) is a promising direction. However, we find that directly applying SAM to LoRA parameters limits the sharpness optimization to a restricted subspace, hindering its effectiveness. To address this limitation, we propose Bi-directional Low-Rank Adaptation (Bi-LoRA), which introduces an auxiliary LoRA module to model SAM's adversarial weight perturbations. It decouples SAM's weight perturbations from LoRA optimization: the primary LoRA module adapts to specific tasks via standard gradient descent, while the auxiliary module captures the sharpness of the loss landscape through gradient ascent. Such dual-module design enables Bi-LoRA to capture broader sharpness for achieving flatter minima while remaining memory-efficient. Another important benefit is that the dual design allows for simultaneous optimization and perturbation, eliminating SAM's doubled training costs. Extensive experiments across diverse tasks and architectures demonstrate Bi-LoRA's efficiency and effectiveness in enhancing generalization.
Abstract (translated)
在数据有限的情况下，对大规模预训练模型进行微调以实现泛化是一项重大挑战。虽然敏锐度感知最小化（Sharpness-Aware Minimization, SAM）通过寻找平坦的极小值来提高泛化的有效性已经得到了证明，但其显著增加的记忆和计算开销使其在大型模型上的应用变得不切实际。将SAM与低秩适应（Low-Rank Adaptation, LoRA）等参数高效微调方法结合是一种有前景的方向。然而，我们发现直接将SAM应用于LoRA参数会限制尖锐度优化到一个受限的子空间中，从而影响其效果。 为了解决这一局限性，我们提出了一种双向低秩适应（Bi-directional Low-Rank Adaptation, Bi-LoRA）方法，通过引入辅助的LoRA模块来模拟SAM的对抗权重扰动。这种方法将SAM的权重扰动与LoRA优化过程解耦：主LoRA模块通过标准梯度下降法针对特定任务进行调整，而辅助模块则通过梯度上升捕捉损失地形图的锐利性。这种双模块设计使Bi-LoRA能够在保持内存效率的同时捕获更广泛的锐度以实现更平坦的极小值。此外，双重设计还允许同时进行优化和扰动，消除了SAM加倍训练成本的问题。 跨多种任务和架构的大量实验表明了Bi-LoRA在提高泛化能力方面既高效又有效。
URL
https://arxiv.org/abs/2508.19564
PDF
-
Robustness Feature Adapter for Efficient Adversarial Training
2025-08-25 05:23:50Quanwei Wu, Jun Guo, Wei Wang, Yi WangAbstract
Adversarial training (AT) with projected gradient descent is the most popular method to improve model robustness under adversarial attacks. However, computational overheads become prohibitively large when AT is applied to large backbone models. AT is also known to have the issue of robust overfitting. This paper contributes to solving both problems simultaneously towards building more trustworthy foundation models. In particular, we propose a new adapter-based approach for efficient AT directly in the feature space. We show that the proposed adapter-based approach can improve the inner-loop convergence quality by eliminating robust overfitting. As a result, it significantly increases computational efficiency and improves model accuracy by generalizing adversarial robustness to unseen attacks. We demonstrate the effectiveness of the new adapter-based approach in different backbone architectures and in AT at scale.
Abstract (translated)
URL
https://arxiv.org/abs/2508.17680
PDF
-
Consistent Opponent Modeling of Static Opponents in Imperfect-Information Games
Abstract
The goal of agents in multi-agent environments is to maximize total reward against the opposing agents that are encountered. Following a game-theoretic solution concept, such as Nash equilibrium, may obtain a strong performance in some settings; however, such approaches fail to capitalize on historical and observed data from repeated interactions against our opponents. Opponent modeling algorithms integrate machine learning techniques to exploit suboptimal opponents utilizing available data; however, the effectiveness of such approaches in imperfect-information games to date is quite limited. We show that existing opponent modeling approaches fail to satisfy a simple desirable property even against static opponents drawn from a known prior distribution; namely, they do not guarantee that the model approaches the opponent's true strategy even in the limit as the number of game iterations approaches infinity. We develop a new algorithm that is able to achieve this property and runs efficiently by solving a convex minimization problem based on the sequence-form game representation using projected gradient descent. The algorithm is guaranteed to efficiently converge to the opponent's true strategy given observations from gameplay and possibly additional historical data if it is available.
Abstract (translated)
在多代理环境中的代理目标是最大化与遇到的对立代理之间的总奖励。根据博弈论解决方案概念，如纳什均衡，在某些情况下可以获得强大的性能；然而，这类方法未能充分利用反复互动中从对手那里获得的历史和观察数据。对手建模算法通过整合机器学习技术来利用已有的数据对次优对手进行优化；然而，这些方法在不完全信息游戏中迄今为止的有效性相当有限。 我们展示了现有的对手建模方法甚至对于来自已知先验分布的静态对手都无法满足一个简单的理想属性：即它们无法保证模型会随着游戏轮数趋近于无穷时逐渐接近对手的真实策略。为此，我们开发了一种新算法，该算法能够解决这一问题，并通过基于序列形式博弈表示求解凸优化问题（使用投影梯度下降）高效运行。 该算法确保在给定游戏观察数据和可能的额外历史数据的情况下，能有效地收敛到对手的真实策略。
URL
https://arxiv.org/abs/2508.17671
PDF
-
In-Context Algorithm Emulation in Fixed-Weight Transformers
2025-08-24 23:20:31Jerry Yao-Chieh Hu, Hude Liu, Jennifer Yuntong Zhang, Han LiuAbstract
We prove that a minimal Transformer architecture with frozen weights is capable of emulating a broad class of algorithms by in-context prompting. In particular, for any algorithm implementable by a fixed-weight attention head (e.g. one-step gradient descent or linear/ridge regression), there exists a prompt that drives a two-layer softmax attention module to reproduce the algorithm's output with arbitrary precision. This guarantee extends even to a single-head attention layer (using longer prompts if necessary), achieving architectural minimality. Our key idea is to construct prompts that encode an algorithm's parameters into token representations, creating sharp dot-product gaps that force the softmax attention to follow the intended computation. This construction requires no feed-forward layers and no parameter updates. All adaptation happens through the prompt alone. These findings forge a direct link between in-context learning and algorithmic emulation, and offer a simple mechanism for large Transformers to serve as prompt-programmable libraries of algorithms. They illuminate how GPT-style foundation models may swap algorithms via prompts alone, establishing a form of algorithmic universality in modern Transformer models.
Abstract (translated)
我们证明，一个权重固定化的最小Transformer架构能够通过上下文提示来模拟一大类算法。具体来说，对于任何可以通过固定权重的注意力头实现的算法（例如一步梯度下降或线性/岭回归），都存在一种提示，可以驱动两层softmax注意力模块以任意精度重现该算法的输出。这一保证甚至适用于单个头部注意层（如果需要的话使用更长的提示），实现了架构上的最小化。我们的核心思想是构造能够将算法参数编码为标记表示形式的提示，创造出明确的点积差距，迫使softmax注意机制按照预期进行计算。这种构建不需要前向传播层和参数更新。所有适应性都通过提示单独完成。 这些发现直接建立了上下文学习与算法模拟之间的联系，并提供了一种简单的机制，使大型Transformer可以充当可通过提示编程的算法库。它们揭示了GPT风格的基础模型如何仅通过提示即可交换算法，从而在现代Transformer模型中建立了一种形式的算法普适性。
URL
https://arxiv.org/abs/2508.17550
PDF
-
ONG: Orthogonal Natural Gradient Descent
2025-08-24 00:27:23Yajat Yadav, Jathin Korrapati, Patrick MendozaAbstract
Orthogonal gradient descent has emerged as a powerful method for continual learning tasks. However, its Euclidean projections overlook the underlying information-geometric structure of the space of distributions parametrized by neural networks, which can lead to suboptimal convergence in learning tasks. To counteract this, we combine it with the idea of the natural gradient and present ONG (Orthogonal Natural Gradient Descent). ONG preconditions each new task gradient with an efficient EKFAC approximation of the inverse Fisher information matrix, yielding updates that follow the steepest descent direction under a Riemannian metric. To preserve performance on previously learned tasks, ONG projects these natural gradients onto the orthogonal complement of prior task gradients. We provide a theoretical justification for this procedure, introduce the ONG algorithm, and benchmark its performance on the Permuted and Rotated MNIST datasets. All code for our experiments/reproducibility can be found at this https URL.
Abstract (translated)
正交梯度下降（Orthogonal gradient descent）已成为连续学习任务中的一种强大方法。然而，其欧几里得投影忽视了由神经网络参数化分布空间所具有的底层信息几何结构，这可能导致在学习任务中的次优收敛。为解决这一问题，我们将正交梯度与自然梯度的概念相结合，并提出了ONG（Orthogonal Natural Gradient Descent）。ONG通过高效地使用EKFAC近似逆费雪信息矩阵来预处理每个新任务的梯度，从而生成遵循黎曼度量下最陡下降方向的更新。为了保持在之前学习的任务上的性能，ONG将这些自然梯度投影到先前任务梯度正交补集上。我们为这一过程提供了理论依据，并引入了ONG算法，在Permuted和Rotated MNIST数据集上对其进行了基准测试以展示其效果。 所有实验代码及重现性所需的内容都可以在以下网址找到：[此处插入具体URL]。
URL
https://arxiv.org/abs/2508.17169
PDF
-
Enhanced NIRMAL Optimizer With Damped Nesterov Acceleration: A Comparative Analysis
2025-08-22 17:16:06Nirmal Gaud, Prasad Krishna Murthy, Mostaque Md. Morshedur Hassan, Abhijit Ganguly, Vinay Mali, Ms Lalita Bhagwat Randive, Abhaypratap SinghAbstract
This study introduces the Enhanced NIRMAL (Novel Integrated Robust Multi-Adaptation Learning with Damped Nesterov Acceleration) optimizer, an improved version of the original NIRMAL optimizer. By incorporating an $(\alpha, r)$-damped Nesterov acceleration mechanism, Enhanced NIRMAL improves convergence stability while retaining chess-inspired strategies of gradient descent, momentum, stochastic perturbations, adaptive learning rates, and non-linear transformations. We evaluate Enhanced NIRMAL against Adam, SGD with Momentum, Nesterov, and the original NIRMAL on four benchmark image classification datasets: MNIST, FashionMNIST, CIFAR-10, and CIFAR-100, using tailored convolutional neural network (CNN) architectures. Enhanced NIRMAL achieves a test accuracy of 46.06\% and the lowest test loss (1.960435) on CIFAR-100, surpassing the original NIRMAL (44.34\% accuracy) and closely rivaling SGD with Momentum (46.43\% accuracy). These results underscore Enhanced NIRMAL's superior generalization and stability, particularly on complex datasets.
Abstract (translated)
这项研究介绍了增强型NIRMAL（集成鲁棒多适应学习与阻尼Nesterov加速）优化器，它是原始NIRMAL优化器的一个改进版本。通过引入$(\alpha, r)$-阻尼的Nesterov加速机制，Enhanced NIRMAL在保持棋盘启发策略的同时提高了收敛稳定性，这些策略包括梯度下降、动量、随机扰动、自适应学习率和非线性变换。我们在四个基准图像分类数据集（MNIST、FashionMNIST、CIFAR-10 和 CIFAR-100）上使用定制的卷积神经网络（CNN）架构对Enhanced NIRMAL进行了评估，并将其与Adam、带有动量的SGD、Nesterov以及原始NIRMAL进行对比。在CIFAR-100数据集上，Enhanced NIRMAL达到了46.06%的测试准确率和最低的测试损失（1.960435），超过了原版NIRMAL（44.34%准确率）并且与带有动量的SGD（46.43%准确率）非常接近。这些结果强调了Enhanced NIRMAL在复杂数据集上的优越泛化能力和稳定性。
URL
https://arxiv.org/abs/2508.16550
PDF
