Gradient_Descent
-
Fine-grained Analysis of Stability and Generalization for Stochastic Bilevel Optimization
2026-04-05 12:12:58Xuelin Zhang, Hong Chen, Bin Gu, Tieliang Gong, Feng ZhengAbstract
Stochastic bilevel optimization (SBO) has been integrated into many machine learning paradigms recently, including hyperparameter optimization, meta learning, and reinforcement learning. Along with the wide range of applications, there have been numerous studies on the computational behavior of SBO. However, the generalization guarantees of SBO methods are far less understood from the lens of statistical learning theory. In this paper, we provide a systematic generalization analysis of the first-order gradient-based bilevel optimization methods. Firstly, we establish the quantitative connections between the on-average argument stability and the generalization gap of SBO methods. Then, we derive the upper bounds of on-average argument stability for single-timescale stochastic gradient descent (SGD) and two-timescale SGD, where three settings (nonconvex-nonconvex (NC-NC), convex-convex (C-C), and strongly-convex-strongly-convex (SC-SC)) are considered respectively. Experimental analysis validates our theoretical findings. Compared with the previous algorithmic stability analysis, our results do not require reinitializing the inner-level parameters at each iteration and are applicable to more general objective functions.
Abstract (translated)
随机双层优化（SBO）近期已被整合进多种机器学习范式，包括超参数优化、元学习与强化学习。随着应用范围的不断拓展，学界对SBO的计算行为展开了大量研究。然而，从统计学习理论视角审视，SBO方法的泛化保证机制仍远未被充分理解。本文对基于一阶梯度的双层优化方法进行了系统性泛化分析。首先，我们建立了SBO方法平均论证稳定性与泛化间隙之间的定量关联。随后，我们在非凸-非凸（NC-NC）、凸-凸（C-C）及强凸-强凸（SC-SC）三种设定下，分别推导了单时间尺度随机梯度下降（SGD）与双时间尺度SGD的平均论证稳定性上界。实验分析验证了我们的理论发现。相较于既有算法稳定性分析，本研究结果无需在每轮迭代中重新初始化内层参数，且适用于更广泛的目标函数形式。
URL
https://arxiv.org/abs/2604.04090
PDF
-
An Improved Last-Iterate Convergence Rate for Anchored Gradient Descent Ascent
2026-04-04 16:08:55Anja Surina, Arun Suggala, George Tsoukalas, Anton Kovsharov, Sergey Shirobokov, Francisco J. R. Ruiz, Pushmeet Kohli, Swarat ChaudhuriAbstract
We analyze the last-iterate convergence of the Anchored Gradient Descent Ascent algorithm for smooth convex-concave min-max problems. While previous work established a last-iterate rate of $\mathcal{O}(1/t^{2-2p})$ for the squared gradient norm, where $p \in (1/2, 1)$, it remained an open problem whether the improved exact $\mathcal{O}(1/t)$ rate is achievable. In this work, we resolve this question in the affirmative. This result was discovered autonomously by an AI system capable of writing formal proofs in Lean. The Lean proof can be accessed at this https URL
Abstract (translated)
我们分析了锚定梯度下降上升算法在光滑凸凹极小极大问题中的最后迭代收敛性。先前工作已证明，当 $p \in (1/2, 1)$ 时，该算法关于平方梯度范数的最后迭代收敛率为 $\mathcal{O}(1/t^{2-2p})$，但更优的精确 $\mathcal{O}(1/t)$ 收敛率是否可达仍是一个未解问题。本研究肯定地回答了这一问题。该成果由一个能够用 Lean 语言撰写形式化证明的 AI 系统自主发现。Lean 证明可在此 https 链接获取。
URL
https://arxiv.org/abs/2604.03782
PDF
-
Learning the Signature of Memorization in Autoregressive Language Models
2026-04-03 17:17:51David Ili\'c, Kostadin Cvejoski, David Stanojevi\'c, Evgeny GrigorenkoAbstract
All prior membership inference attacks for fine-tuned language models use hand-crafted heuristics (e.g., loss thresholding, Min-K\%, reference calibration), each bounded by the designer's intuition. We introduce the first transferable learned attack, enabled by the observation that fine-tuning any model on any corpus yields unlimited labeled data, since membership is known by construction. This removes the shadow model bottleneck and brings membership inference into the deep learning era: learning what matters rather than designing it, with generalization through training diversity and scale. We discover that fine-tuning language models produces an invariant signature of memorization detectable across architectural families and data domains. We train a membership inference classifier exclusively on transformer-based models. It transfers zero-shot to Mamba (state-space), RWKV-4 (linear attention), and RecurrentGemma (gated recurrence), achieving 0.963, 0.972, and 0.936 AUC respectively. Each evaluation combines an architecture and dataset never seen during training, yet all three exceed performance on held-out transformers (0.908 AUC). These four families share no computational mechanisms, their only commonality is gradient descent on cross-entropy loss. Even simple likelihood-based methods exhibit strong transfer, confirming the signature exists independently of the detection method. Our method, Learned Transfer MIA (LT-MIA), captures this signal most effectively by reframing membership inference as sequence classification over per-token distributional statistics. On transformers, LT-MIA achieves 2.8$\times$ higher TPR at 0.1\% FPR than the strongest baseline. The method also transfers to code (0.865 AUC) despite training only on natural language texts. Code and trained classifier available at this https URL.
Abstract (translated)
此前针对微调语言模型的所有成员推断攻击均依赖手工设计的启发式规则（例如损失阈值法、Min-K%、参考校准），其性能受设计者直觉的限制。我们提出了首个可迁移的习得攻击，其核心发现是：任何模型在任何语料库上微调时，由于成员身份在构造时已知，因此能产生无限标记数据。这消除了影子模型瓶颈，将成员推断带入深度学习时代——从“设计特征”转向“学习特征”，通过训练多样性和规模实现泛化。我们发现，微调语言模型会产生一个跨架构家族和数据域可检测的恒定记忆签名。我们仅在基于Transformer的模型上训练成员推断分类器，并实现零样本迁移至Mamba（状态空间模型）、RWKV-4（线性注意力）和RecurrentGemma（门控循环网络）， respective AUC分别达到0.963、0.972和0.936。每次评估均结合训练中未见的架构与数据集，但三者性能均超过在保留Transformer上的表现（0.908 AUC）。这四类架构共享无计算机制，其唯一共同点是基于交叉熵损失的梯度下降。即使简单基于似然的方法也展现出强迁移性，证实该签名独立于检测方法存在。我们的方法——习得迁移成员推断（LT-MIA）——通过将成员推断重构为基于每标记分布统计的序列分类任务，最有效地捕获该信号。在Transformer上，LT-MIA在0.1%假阳性率下实现比最强基线高2.8倍的真阳性率。该方法即使仅使用自然语言文本训练，也能迁移至代码领域（0.865 AUC）。代码与训练好的分类器可通过此链接获取。
URL
https://arxiv.org/abs/2604.03199
PDF
-
Hybrid Energy-Based Models for Physical AI: Provably Stable Identification of Port-Hamiltonian Dynamics
Abstract
Energy-based models (EBMs) implement inference as gradient descent on a learned Lyapunov function, yielding interpretable, structure-preserving alternatives to black-box neural ODEs and aligning naturally with physical AI. Yet their use in system identification remains limited, and existing architectures lack formal stability guarantees that globally preclude unstable modes. We address this gap by introducing an EBM framework for system identification with stable, dissipative, absorbing invariant dynamics. Unlike classical global Lyapunov stability, absorbing invariance expands the class of stability-preserving architectures, enabling more flexible and expressive EBMs. We extend EBM theory to nonsmooth activations by establishing negative energy dissipation via Clarke derivatives and deriving new conditions for radial unboundedness, exposing a stability-expressivity tradeoff in standard EBMs. To overcome this, we introduce a hybrid architecture with a dynamical visible layer and static hidden layers, prove absorbing invariance under mild assumptions, and show that these guarantees extend to port-Hamiltonian EBMs. Experiments on metric-deformed multi-well and ring systems validate the approach, showcasing how our hybrid EBM architecture combines expressivity with sound and provable safety guarantees by design.
Abstract (translated)
基于能量的模型（EBMs）通过在学习得到的李雅普诺夫函数上执行梯度下降来实现推断，为黑箱神经常微分方程提供了可解释、结构保持的替代方案，并与物理人工智能自然契合。然而，其在系统辨识中的应用仍较为有限，且现有架构缺乏全局排除不稳定模式的正式稳定性保证。本文通过引入一个具备稳定、耗散、吸收不变动力学的EBM框架来弥补这一空白。与经典的全局李雅普诺夫稳定性不同，吸收不变性扩展了稳定性保持架构的范畴，从而支持更灵活、更具表达力的EBMs。我们通过克拉克导数建立负能量耗散，并推导径向无界性的新条件，将EBM理论扩展至非光滑激活函数，揭示了标准EBMs中存在的稳定性-表达性权衡。为克服此局限，我们提出一种混合架构：动态可见层与静态隐藏层，在温和假设下证明其吸收不变性，并表明此类保证可延伸至端口哈密顿EBMs。在度量变形多阱与环系统上的实验验证了该方法，展示了我们的混合EBM架构如何通过设计将表达力与可靠且可证明的安全性保证相结合。
URL
https://arxiv.org/abs/2604.00277
PDF
-
Reasoning as Energy Minimization over Structured Latent Trajectories
2026-03-30 10:20:56David K. JohanssonAbstract
Single-shot neural decoders commit to answers without iterative refinement, while chain-of-thought methods introduce discrete intermediate steps but lack a scalar measure of reasoning progress. We propose Energy-Based Reasoning via Structured Latent Planning (EBRM), which models reasoning as gradient-based optimization of a multi-step latent trajectory $z_{1:T}$ under a learned energy function $E(h_x, z)$. The energy decomposes into per-step compatibility, transition consistency, and trajectory smoothness terms. Training combines supervised encoder-decoder learning with contrastive energy shaping using hard negatives, while inference performs gradient descent or Langevin dynamics over $z$ and decodes from $z_T$. We identify a critical failure mode: on CNF logic satisfaction, latent planning reduces accuracy from $\approx 95\%$ to $\approx 56\%$. This degradation arises from a distribution mismatch, where the decoder is trained on encoder outputs $h_x$ but evaluated on planner outputs $z_T$ that drift into unseen latent regions. We analyze this behavior through per-step decoding, latent drift tracking, and gradient decomposition. To address it, we propose dual-path decoder training and latent anchoring. We further introduce a six-part ablation protocol covering component contributions, trajectory length, planner dynamics, initialization, decoder training distribution, and anchor weight. Experiments on three synthetic tasks show that energy decreases monotonically and induces structured latent trajectories on graph and logic tasks, while remaining flat on arithmetic ($r = 0.073$), indicating a negative result. Code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2603.28248
PDF
-
OSA: Echocardiography Video Segmentation via Orthogonalized State Update and Anatomical Prior-aware Feature Enhancement
2026-03-27 09:02:16Rui Wang, Huisi Wu, Jing QinAbstract
Accurate and temporally consistent segmentation of the left ventricle from echocardiography videos is essential for estimating the ejection fraction and assessing cardiac function. However, modeling spatiotemporal dynamics remains difficult due to severe speckle noise and rapid non-rigid deformations. Existing linear recurrent models offer efficient in-context associative recall for temporal tracking, but rely on unconstrained state updates, which cause progressive singular value decay in the state matrix, a phenomenon known as rank collapse, resulting in anatomical details being overwhelmed by noise. To address this, we propose OSA, a framework that constrains the state evolution on the Stiefel manifold. We introduce the Orthogonalized State Update (OSU) mechanism, which formulates the memory evolution as Euclidean projected gradient descent on the Stiefel manifold to prevent rank collapse and maintain stable temporal transitions. Furthermore, an Anatomical Prior-aware Feature Enhancement module explicitly separates anatomical structures from speckle noise through a physics-driven process, providing the temporal tracker with noise-resilient structural cues. Comprehensive experiments on the CAMUS and EchoNet-Dynamic datasets show that OSA achieves state-of-the-art segmentation accuracy and temporal stability, while maintaining real-time inference efficiency for clinical deployment. Codes are available at this https URL.
Abstract (translated)
准确且时序一致地从超声心动图视频中分割左心室，对于估算射血分数和评估心脏功能至关重要。然而，由于严重的斑点噪声和快速的非刚性形变，对时空动态的建模仍然困难。现有的线性循环模型虽能提供高效的上下文关联记忆以实现时序追踪，但其依赖无约束的状态更新，导致状态矩阵出现渐进奇异值衰减——即秩崩溃现象，使解剖细节被噪声淹没。针对此问题，我们提出了OSA框架，该框架将状态演化约束在Stiefel流形上。我们引入了正交状态更新（OSU）机制，将记忆演化表述为Stiefel流形上的欧几里得投影梯度下降，以防止秩崩溃并维持稳定的时序过渡。此外，解剖先验感知特征增强模块通过物理驱动过程明确分离解剖结构与斑点噪声，为时序追踪器提供抗噪声的结构化线索。在CAMUS和EchoNet-Dynamic数据集上的综合实验表明，OSA在实现最先进分割精度与时序稳定性的同时，保持了临床部署所需的实时推理效率。代码已公开于此https URL。
URL
https://arxiv.org/abs/2603.26188
PDF
-
EvoForest: A Novel Machine-Learning Paradigm via Open-Ended Evolution of Computational Graphs
2026-03-26 00:07:45Kamer Ali Yuksel, Hassan SawafAbstract
Modern machine learning is still largely organized around a single recipe: choose a parameterized model family and optimize its weights. Although highly successful, this paradigm is too narrow for many structured prediction problems, where the main bottleneck is not parameter fitting but discovering what should be computed from the data. Success often depends on identifying the right transformations, statistics, invariances, interaction structures, temporal summaries, gates, or nonlinear compositions, especially when objectives are non-differentiable, evaluation is cross-validation-based, interpretability matters, or continual adaptation is required. We present EvoForest, a hybrid neuro-symbolic system for end-to-end open-ended evolution of computation. Rather than merely generating features, EvoForest jointly evolves reusable computational structure, callable function families, and trainable low-dimensional continuous components inside a shared directed acyclic graph. Intermediate nodes store alternative implementations, callable nodes encode reusable transformation families such as projections, gates, and activations, output nodes define candidate predictive computations, and persistent global parameters can be refined by gradient descent. For each graph configuration, EvoForest evaluates the discovered computation and uses a lightweight Ridge-based readout to score the resulting representation against a non-differentiable cross-validation target. The evaluator also produces structured feedback that guides future LLM-driven mutations. In the 2025 ADIA Lab Structural Break Challenge, EvoForest reached 94.13% ROC-AUC after 600 evolution steps, exceeding the publicly reported winning score of 90.14% under the same evaluation protocol.
Abstract (translated)
URL
https://arxiv.org/abs/2604.19761
PDF
-
Amplified Patch-Level Differential Privacy for Free via Random Cropping
2026-03-25 18:15:06Kaan Durmaz, Jan Schuchardt, Sebastian Schmidt, Stephan G\"unnemannAbstract
Random cropping is one of the most common data augmentation techniques in computer vision, yet the role of its inherent randomness in training differentially private machine learning models has thus far gone unexplored. We observe that when sensitive content in an image is spatially localized, such as a face or license plate, random cropping can probabilistically exclude that content from the model's input. This introduces a third source of stochasticity in differentially private training with stochastic gradient descent, in addition to gradient noise and minibatch sampling. This additional randomness amplifies differential privacy without requiring changes to model architecture or training procedure. We formalize this effect by introducing a patch-level neighboring relation for vision data and deriving tight privacy bounds for differentially private stochastic gradient descent (DP-SGD) when combined with random cropping. Our analysis quantifies the patch inclusion probability and shows how it composes with minibatch sampling to yield a lower effective sampling rate. Empirically, we validate that patch-level amplification improves the privacy-utility trade-off across multiple segmentation architectures and datasets. Our results demonstrate that aligning privacy accounting with domain structure and additional existing sources of randomness can yield stronger guarantees at no additional cost.
Abstract (translated)
随机裁剪是计算机视觉中最常见的数据增强技术之一，但其内在随机性在训练差分隐私机器学习模型中的作用迄今尚未被探索。我们观察到，当图像中的敏感内容（如面部或车牌）呈空间局部化分布时，随机裁剪可以概率性地将这些内容排除在模型输入之外。这为采用随机梯度下降进行差分隐私训练引入了除梯度噪声和小批量采样之外的第三种随机性来源。这种额外的随机性无需更改模型架构或训练流程即可增强差分隐私。我们通过引入视觉数据的块级邻近关系来形式化这一效应，并为结合随机裁剪的差分隐私随机梯度下降（DP-SGD）推导出紧致的隐私界限。我们的分析量化了块包含概率，并展示了其如何与小批量采样组合以产生更低的有效采样率。经验上，我们验证了块级增强在多个分割架构和数据集上改善了隐私-效用权衡。我们的结果表明，将隐私核算与领域结构及现有随机性来源对齐，可以无需额外成本地提供更强的保障。
URL
https://arxiv.org/abs/2603.24695
PDF
-
SafeSeek: Universal Attribution of Safety Circuits in Language Models
2026-03-24 14:32:53Miao Yu, Siyuan Fu, Moayad Aloqaily, Zhenhong Zhou, Safa Otoum, Xing fan, Kun Wang, Yufei Guo, Qingsong WenAbstract
Mechanistic interpretability reveals that safety-critical behaviors (e.g., alignment, jailbreak, backdoor) in Large Language Models (LLMs) are grounded in specialized functional components. However, existing safety attribution methods struggle with generalization and reliability due to their reliance on heuristic, domain-specific metrics and search algorithms. To address this, we propose \ourmethod, a unified safety interpretability framework that identifies functionally complete safety circuits in LLMs via optimization. Unlike methods focusing on isolated heads or neurons, \ourmethod introduces differentiable binary masks to extract multi-granular circuits through gradient descent on safety datasets, while integrates Safety Circuit Tuning to utilize these sparse circuits for efficient safety fine-tuning. We validate \ourmethod in two key scenarios in LLM safety: \textbf{(1) backdoor attacks}, identifying a backdoor circuit with 0.42\% sparsity, whose ablation eradicates the Attack Success Rate (ASR) from 100\% $\to$ 0.4\% while retaining over 99\% general utility; \textbf{(2) safety alignment}, localizing an alignment circuit with 3.03\% heads and 0.79\% neurons, whose removal spikes ASR from 0.8\% $\to$ 96.9\%, whereas excluding this circuit during helpfulness fine-tuning maintains 96.5\% safety retention.
Abstract (translated)
机制可解释性揭示，大型语言模型（LLMs）中的安全关键行为（如对齐、越狱、后门）根植于特定的功能组件。然而，现有安全归因方法因依赖启发式、领域特定的指标与搜索算法，在泛化性与可靠性方面存在局限。为此，我们提出\ourmethod\——一个通过优化识别LLMs中功能完整安全电路的统一安全可解释性框架。不同于聚焦孤立注意力头或神经元的方法，\ourmethod\引入可微分二值掩码，通过在安全数据集上进行梯度下降提取多粒度电路，同时整合安全电路调优（Safety Circuit Tuning）以利用这些稀疏电路实现高效安全微调。我们在LLM安全的两大关键场景中验证\ourmethod\：\textbf{(1) 后门攻击}，识别出稀疏度仅0.42%的后门电路，其消融将攻击成功率（ASR）从100%降至0.4%，同时保留超过99%的通用性能；\textbf{(2) 安全对齐}，定位出包含3.03%注意力头与0.79%神经元的对齐电路，移除该电路使ASR从0.8%飙升至96.9%，而在助人性微调中排除此电路仍能保持96.5%的安全保留率。
URL
https://arxiv.org/abs/2603.23268
PDF
-
PA-SFM: Tracker-free differentiable acoustic radiation for freehand 3D photoacoustic imaging
2026-03-23 08:19:45Shuang Li, Jian Gao, Chulhong Kim, Seongwook Choi, Qian Chen, Yibing Wang, Shuang Wu, Yu Zhang, Tingting Huang, Yucheng Zhou, Boxin Yao, Yao Yao, Changhui LiAbstract
Three-dimensional (3D) handheld photoacoustic tomography typically relies on bulky and expensive external positioning sensors to correct motion artifacts, which severely limits its clinical flexibility and accessibility. To address this challenge, we present PA-SFM, a tracker-free framework that leverages exclusively single-modality photoacoustic data for both sensor pose recovery and high-fidelity 3D reconstruction via differentiable acoustic radiation modeling. Unlike traditional structure-from-motion (SFM) methods based on visual features, PA-SFM integrates the acoustic wave equation into a differentiable programming pipeline. By leveraging a high-performance, GPU-accelerated acoustic radiation kernel, the framework simultaneously optimizes the 3D photoacoustic source distribution and the sensor array pose via gradient descent. To ensure robust convergence in freehand scenarios, we introduce a coarse-to-fine optimization strategy that incorporates geometric consistency checks and rigid-body constraints to eliminate motion outliers. We validated the proposed method through both numerical simulations and in-vivo rat experiments. The results demonstrate that PA-SFM achieves sub-millimeter positioning accuracy and restores high-resolution 3D vascular structures comparable to ground-truth benchmarks, offering a low-cost, software-defined solution for clinical freehand photoacoustic imaging. The source code is publicly available at \href{this https URL}{this https URL}.
Abstract (translated)
URL
https://arxiv.org/abs/2604.09643
PDF
-
Optimizing Multi-Agent Weather Captioning via Text Gradient Descent: A Training-Free Approach with Consensus-Aware Gradient Fusion
2026-03-23 07:55:45Shixu LiuAbstract
Generating interpretable natural language captions from weather time series data remains a significant challenge at the intersection of meteorological science and natural language processing. While recent advances in Large Language Models (LLMs) have demonstrated remarkable capabilities in time series forecasting and analysis, existing approaches either produce numerical predictions without human-accessible explanations or generate generic descriptions lacking domain-specific depth. We introduce WeatherTGD, a training-free multi-agent framework that reinterprets collaborative caption refinement through the lens of Text Gradient Descent (TGD). Our system deploys three specialized LLM agents including a Statistical Analyst, a Physics Interpreter, and a Meteorology Expert that generate domain-specific textual gradients from weather time series observations. These gradients are aggregated through a novel Consensus-Aware Gradient Fusion mechanism that extracts common signals while preserving unique domain perspectives. The fused gradients then guide an iterative refinement process analogous to gradient descent, where each LLM-generated feedback signal updates the caption toward an optimal solution. Experiments on real-world meteorological datasets demonstrate that WeatherTGD achieves significant improvements in both LLM-based evaluation and human expert evaluation, substantially outperforming existing multi-agent baselines while maintaining computational efficiency through parallel agent execution.
Abstract (translated)
URL
https://arxiv.org/abs/2603.21673
PDF
-
Natural Gradient Descent for Online Continual Learning
2026-03-21 18:08:28Joe Khawand, David ColliauxAbstract
Online Continual Learning (OCL) for image classification represents a challenging subset of Continual Learning, focusing on classifying images from a stream without assuming data independence and identical distribution (i.i.d). The primary challenge in this context is to prevent catastrophic forgetting, where the model's performance on previous tasks deteriorates as it learns new ones. Although various strategies have been proposed to address this issue, achieving rapid convergence remains a significant challenge in the online setting. In this work, we introduce a novel approach to training OCL models that utilizes the Natural Gradient Descent optimizer, incorporating an approximation of the Fisher Information Matrix (FIM) through Kronecker Factored Approximate Curvature (KFAC). This method demonstrates substantial improvements in performance across all OCL methods, particularly when combined with existing OCL tricks, on datasets such as Split CIFAR-100, CORE50, and Split miniImageNet.
Abstract (translated)
在线持续学习（OCL）图像分类是持续学习中的一个具有挑战性的子领域，其核心在于处理来自数据流的图像分类任务，且不假设数据满足独立同分布（i.i.d.）。该场景下的主要挑战是防止灾难性遗忘——即模型在学习新任务时对先前任务的性能会逐渐退化。尽管已有多种策略被提出以解决此问题，但在在线学习场景中实现快速收敛仍是一项重大挑战。本研究提出了一种训练OCL模型的新方法，该方法采用自然梯度下降（Natural Gradient Descent）优化器，并通过克罗内克因子近似曲率（KFAC）对费雪信息矩阵（FIM）进行近似。实验表明，该方法在Split CIFAR-100、CORE50和Split miniImageNet等数据集上，能够显著提升所有OCL方法的性能，尤其在与现有OCL技巧结合时效果更为突出。
URL
https://arxiv.org/abs/2603.20898
PDF
-
Growing Networks with Autonomous Pruning
2026-03-20 08:44:46Charles De Lambilly, Stefan DuffnerAbstract
This paper introduces Growing Networks with Autonomous Pruning (GNAP) for image classification. Unlike traditional convolutional neural networks, GNAP change their size, as well as the number of parameters they are using, during training, in order to best fit the data while trying to use as few parameters as possible. This is achieved through two complementary mechanisms: growth and pruning. GNAP start with few parameters, but their size is expanded periodically during training to add more expressive power each time the network has converged to a saturation point. Between these growing phases, model parameters are trained for classification and pruned simultaneously, with complete autonomy by gradient descent. Growing phases allow GNAP to improve their classification performance, while autonomous pruning allows them to keep as few parameters as possible. Experimental results on several image classification benchmarks show that our approach can train extremely sparse neural networks with high accuracy. For example, on MNIST, we achieved 99.44% accuracy with as few as 6.2k parameters, while on CIFAR10, we achieved 92.2\ accuracy with 157.8k parameters.
Abstract (translated)
本文介绍了用于图像分类的自主修剪生长网络（GNAP）。与传统卷积神经网络不同，GNAP在训练过程中会动态调整其规模及所用参数数量，以在尽可能减少参数的同时最佳适配数据。这通过两个互补机制实现：生长与修剪。GNAP初始参数较少，但在每次网络收敛至饱和点后，会定期扩展规模以增强表达能力。在生长阶段之间，模型参数在分类训练中同步进行自主修剪，完全由梯度下降驱动。生长阶段提升分类性能，而自主修剪则确保参数精简。在多个图像分类基准测试中，该方法能训练出高精度且极度稀疏的神经网络。例如，在MNIST数据集上，仅用6.2k参数即达到99.44%准确率；在CIFAR10数据集上，以157.8k参数实现92.2%准确率。
URL
https://arxiv.org/abs/2603.19759
PDF
-
AdaRadar: Rate Adaptive Spectral Compression for Radar-based Perception
2026-03-18 17:42:34Jinho Park, Se Young Chun, Mingoo SeokAbstract
Radar is a critical perception modality in autonomous driving systems due to its all-weather characteristics and ability to measure range and Doppler velocity. However, the sheer volume of high-dimensional raw radar data saturates the communication link to the computing engine (e.g., an NPU), which is often a low-bandwidth interface with data rate provisioned only for a few low-resolution range-Doppler frames. A generalized codec for utilizing high-dimensional radar data is notably absent, while existing image-domain approaches are unsuitable, as they typically operate at fixed compression ratios and fail to adapt to varying or adversarial conditions. In light of this, we propose radar data compression with adaptive feedback. It dynamically adjusts the compression ratio by performing gradient descent from the proxy gradient of detection confidence with respect to the compression rate. We employ a zeroth-order gradient approximation as it enables gradient computation even with non-differentiable core operations--pruning and quantization. This also avoids transmitting the gradient tensors over the band-limited link, which, if estimated, would be as large as the original radar data. In addition, we have found that radar feature maps are heavily concentrated on a few frequency components. Thus, we apply the discrete cosine transform to the radar data cubes and selectively prune out the coefficients effectively. We preserve the dynamic range of each radar patch through scaled quantization. Combining those techniques, our proposed online adaptive compression scheme achieves over 100x feature size reduction at minimal performance drop (~1%p). We validate our results on the RADIal, CARRADA, and Radatron datasets.
Abstract (translated)
URL
https://arxiv.org/abs/2603.17979
PDF
-
Auto-Unrolled Proximal Gradient Descent: An AutoML Approach to Interpretable Waveform Optimization
Abstract
This study explores the combination of automated machine learning (AutoML) with model-based deep unfolding (DU) for optimizing wireless beamforming and waveforms. We convert the iterative proximal gradient descent (PGD) algorithm into a deep neural network, wherein the parameters of each layer are learned instead of being predetermined. Additionally, we enhance the architecture by incorporating a hybrid layer that performs a learnable linear gradient transformation prior to the proximal projection. By utilizing AutoGluon with a tree-structured parzen estimator (TPE) for hyperparameter optimization (HPO) across an expanded search space, which includes network depth, step-size initialization, optimizer, learning rate scheduler, layer type, and post-gradient activation, the proposed auto-unrolled PGD (Auto-PGD) achieves 98.8% of the spectral efficiency of a traditional 200-iteration PGD solver using only five unrolled layers, while requiring only 100 training samples. We also address a gradient normalization issue to ensure consistent performance during training and evaluation, and we illustrate per-layer sum-rate logging as a tool for transparency. These contributions highlight a notable reduction in the amount of training data and inference cost required, while maintaining high interpretability compared to conventional black-box architectures.
Abstract (translated)
URL
https://arxiv.org/abs/2603.17478
PDF
-
Asymmetric Nash Seeking via Best Response Maps: Global Linear Convergence and Robustness to Inexact Reaction Models
2026-03-17 18:45:56Mahdis Rabbani, Navid Mojahed, Shima NazariAbstract
Nash equilibria provide a principled framework for modeling interactions in multi-agent decision-making and control. However, many equilibrium-seeking methods implicitly assume that each agent has access to the other agents' objectives and constraints, an assumption that is often unrealistic in practice. This letter studies a class of asymmetric-information two-player constrained games with decoupled feasible sets, in which Player 1 knows its own objective and constraints while Player 2 is available only through a best-response map. For this class of games, we propose an asymmetric projected gradient descent-best response iteration that does not require full mutual knowledge of both players' optimization problems. Under suitable regularity conditions, we establish the existence and uniqueness of the Nash equilibrium and prove global linear convergence of the proposed iteration when the best-response map is exact. Recognizing that best-response maps are often learned or estimated, we further analyze the inexact case and show that, when the approximation error is uniformly bounded by $\varepsilon$, the iterates enter an explicit $O(\varepsilon)$ neighborhood of the true Nash equilibrium. Numerical results on a benchmark game corroborate the predicted convergence behavior and error scaling.
Abstract (translated)
URL
https://arxiv.org/abs/2603.17058
PDF
-
Adversarial attacks against Modern Vision-Language Models
2026-03-17 04:55:10Alejandro Paredes La TorreAbstract
We study adversarial robustness of open-source vision-language model (VLM) agents deployed in a self-contained e-commerce environment built to simulate realistic pre-deployment conditions. We evaluate two agents, LLaVA-v1.5-7B and Qwen2.5-VL-7B, under three gradient-based attacks: the Basic Iterative Method (BIM), Projected Gradient Descent (PGD), and a CLIP-based spectral attack. Against LLaVA, all three attacks achieve substantial attack success rates (52.6%, 53.8%, and 66.9% respectively), demonstrating that simple gradient-based methods pose a practical threat to open-source VLM agents. Qwen2.5-VL proves significantly more robust across all attacks (6.5%, 7.7%, and 15.5%), suggesting meaningful architectural differences in adversarial resilience between open-source VLM families. These findings have direct implications for the security evaluation of VLM agents prior to commercial deployment.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16960
PDF
-
INSTRUMENTAL: Automatic Synthesizer Parameter Recovery from Audio via Evolutionary Optimization
Abstract
Existing audio-to-MIDI tools extract notes but discard the timbral characteristics that define an instrument's identity. We present Instrumental, a system that recovers continuous synthesizer parameters from audio by coupling a differentiable 28-parameter subtractive synthesizer with CMA-ES, a derivative-free evolutionary optimizer. We optimize a composite perceptual loss combining mel-scaled STFT, spectral centroid, and MFCC divergence, achieving a matching loss of 2.09 on real recorded audio. We systematically evaluate eight hypotheses for improving convergence and find that only parametric EQ boosting yields meaningful improvement. Our results show that CMA-ES outperforms gradient descent on this non-convex landscape, that more parameters do not monotonically improve matching, and that spectral analysis initialization accelerates convergence over random starts.
Abstract (translated)
URL
https://arxiv.org/abs/2603.15905
PDF
-
Surgical Robot, Path Planning, Joint Space, Riemannian Manifolds
2026-03-16 05:50:50Yoshiki Yamamoto, Maina Sogabe, Shunichi Hirahara, Toshiki Kaisaki, Tetsuro Miyazaki, Kenji KawashimaAbstract
Robotic surgery for minimally invasive surgery can reduce the surgeon's workload by autonomously guiding robotic forceps. Movement of the robot is restricted around a fixed insertion port. The robot often encounters angle limitations during operation. Also, the surface of the abdominal cavity is non-concave, making it computationally expensive to find the desired this http URL this work, to solve these problems, we propose a method for path planning in joint space by transforming the position into a Riemannian manifold. An edge cost function is defined to search for a desired path in the joint space and reduce the range of motion of the joints. We found that the organ is mostly non-concave, making it easy to find the optimal path using gradient descent method. Experimental results demonstrated that the proposed method reduces the range of joint angle movement compared to calculations in position space.
Abstract (translated)
URL
https://arxiv.org/abs/2603.14852
PDF
-
GradMem: Learning to Write Context into Memory with Test-Time Gradient Descent
2026-03-14 10:17:33Yuri Kuratov, Matvey Kairov, Aydar Bulatov, Ivan Rodkin, Mikhail BurtsevAbstract
Many large language model applications require conditioning on long contexts. Transformers typically support this by storing a large per-layer KV-cache of past activations, which incurs substantial memory overhead. A desirable alternative is ompressive memory: read a context once, store it in a compact state, and answer many queries from that state. We study this in a context removal setting, where the model must generate an answer without access to the original context at inference time. We introduce GradMem, which writes context into memory via per-sample test-time optimization. Given a context, GradMem performs a few steps of gradient descent on a small set of prefix memory tokens while keeping model weights frozen. GradMem explicitly optimizes a model-level self-supervised context reconstruction loss, resulting in a loss-driven write operation with iterative error correction, unlike forward-only methods. On associative key--value retrieval, GradMem outperforms forward-only memory writers with the same memory size, and additional gradient steps scale capacity much more effectively than repeated forward writes. We further show that GradMem transfers beyond synthetic benchmarks: with pretrained language models, it attains competitive results on natural language tasks including bAbI and SQuAD variants, relying only on information encoded in memory.
Abstract (translated)
URL
https://arxiv.org/abs/2603.13875
PDF
