Gradient_Descent
-
On Task Vectors and Gradients
2025-08-22 04:16:42Luca Zhou, Daniele Solombrino, Donato Crisostomi, Maria Sofia Bucarelli, Giuseppe Alessio D'Inverno, Fabrizio Silvestri, Emanuele Rodol\`aAbstract
Task arithmetic has emerged as a simple yet powerful technique for model merging, enabling the combination of multiple finetuned models into one. Despite its empirical success, a clear theoretical explanation of why and when it works is lacking. This paper provides a rigorous theoretical foundation for task arithmetic by establishing a connection between task vectors and gradients of the task losses. We show that under standard gradient descent, a task vector generated from one epoch of finetuning is exactly equivalent to the negative gradient of the loss, scaled by the learning rate. For the practical multi-epoch setting, we prove that this equivalence holds approximately, with a second-order error term that we explicitly bound for feed-forward networks. Our empirical analysis across seven vision benchmarks corroborates our theory, demonstrating that the first-epoch gradient dominates the finetuning trajectory in both norm and direction. A key implication is that merging models finetuned for only a single epoch often yields performance comparable to merging fully converged models. These findings reframe task arithmetic as a form of approximate multitask learning, providing a clear rationale for its effectiveness and highlighting the critical role of early training dynamics in model merging.
Abstract (translated)
任务算术作为一种简单而强大的模型合并技术已经浮现，它能够将多个微调后的模型合成为一个。尽管在实践中取得了成功，但关于该方法为何以及何时有效的确切理论解释却一直缺失。本文为任务算术提供了一个严格的理论基础，通过建立任务向量与任务损失梯度之间的联系来实现这一点。我们展示了，在标准的梯度下降过程中，从一个微调周期生成的任务向量等同于负梯度乘以学习率。对于实际中的多轮次设置，我们证明这种等价性近似成立，并且对前馈网络中的二阶误差项进行了明确界定。 我们的实证分析跨越了七个视觉基准测试集，验证了这一理论，表明第一个微调周期的梯度在范数和方向上主导着后续的微调过程。一个关键结论是，合并仅经过单个周期微调的模型通常可以取得与完全收敛后的模型相当的性能。这些发现将任务算术重新定义为一种近似的多任务学习形式，并清晰地解释了其有效性，同时也强调了早期训练动态在模型合并中的关键作用。
URL
https://arxiv.org/abs/2508.16082
PDF
-
Deep Equilibrium Convolutional Sparse Coding for Hyperspectral Image Denoising
2025-08-21 13:35:11Jin Ye, Jingran Wang, Fengchao Xiong, Jingzhou Chen, Yuntao QianAbstract
Hyperspectral images (HSIs) play a crucial role in remote sensing but are often degraded by complex noise patterns. Ensuring the physical property of the denoised HSIs is vital for robust HSI denoising, giving the rise of deep unfolding-based methods. However, these methods map the optimization of a physical model to a learnable network with a predefined depth, which lacks convergence guarantees. In contrast, Deep Equilibrium (DEQ) models treat the hidden layers of deep networks as the solution to a fixed-point problem and models them as infinite-depth networks, naturally consistent with the optimization. Under the framework of DEQ, we propose a Deep Equilibrium Convolutional Sparse Coding (DECSC) framework that unifies local spatial-spectral correlations, nonlocal spatial self-similarities, and global spatial consistency for robust HSI denoising. Within the convolutional sparse coding (CSC) framework, we enforce shared 2D convolutional sparse representation to ensure global spatial consistency across bands, while unshared 3D convolutional sparse representation captures local spatial-spectral details. To further exploit nonlocal self-similarities, a transformer block is embedded after the 2D CSC. Additionally, a detail enhancement module is integrated with the 3D CSC to promote image detail preservation. We formulate the proximal gradient descent of the CSC model as a fixed-point problem and transform the iterative updates into a learnable network architecture within the framework of DEQ. Experimental results demonstrate that our DECSC method achieves superior denoising performance compared to state-of-the-art methods.
Abstract (translated)
高光谱图像（HSI）在遥感领域中扮演着重要角色，但通常会受到复杂噪声模式的影响。确保去噪后的HSI保持其物理特性对于稳健的HSI去噪至关重要，这促使了基于深度展开的方法的发展。然而，这些方法将一个物理模型的优化映射到具有预定义深度的学习网络上，缺乏收敛保证。相比之下，Deep Equilibrium (DEQ) 模型将深层网络中的隐藏层视为固定点问题的解，并将其建模为无限深度的网络，自然地与优化保持一致。 在DEQ框架下，我们提出了一种名为Deep Equilibrium Convolutional Sparse Coding（DECSC）的框架，该框架统一了局部空间-光谱相关性、非局部的空间自相似性和全局的空间一致性，以实现稳健的HSI去噪。在卷积稀疏编码(CSC)框架内，我们在所有频带之间强制执行共享2D卷积稀疏表示，以确保跨频带的全球空间一致性，同时利用非共享3D卷积稀疏表示来捕捉局部空间-光谱细节。为了进一步挖掘非本地自相似性，在2D CSC之后嵌入了一个变压器块。此外，还集成了一个细节增强模块，与3D CSC结合使用以促进图像细节保留。 我们把CSC模型的近似梯度下降公式化为固定点问题，并将其迭代更新转化为DEQ框架内的可学习网络架构。实验结果表明，我们的DECSC方法在去噪性能上优于现有最先进的方法。
URL
https://arxiv.org/abs/2508.15553
PDF
-
Hybrid Least Squares/Gradient Descent Methods for DeepONets
2025-08-21 09:34:06Jun Choi, Chang-Ock Lee, Minam MoonAbstract
We propose an efficient hybrid least squares/gradient descent method to accelerate DeepONet training. Since the output of DeepONet can be viewed as linear with respect to the last layer parameters of the branch network, these parameters can be optimized using a least squares (LS) solve, and the remaining hidden layer parameters are updated by means of gradient descent form. However, building the LS system for all possible combinations of branch and trunk inputs yields a prohibitively large linear problem that is infeasible to solve directly. To address this issue, our method decomposes the large LS system into two smaller, more manageable subproblems $\unicode{x2014}$ one for the branch network and one for the trunk network $\unicode{x2014}$ and solves them separately. This method is generalized to a broader type of $L^2$ loss with a regularization term for the last layer parameters, including the case of unsupervised learning with physics-informed loss.
Abstract (translated)
我们提出了一种高效的混合最小二乘法/梯度下降方法，以加速DeepONet的训练过程。由于DeepONet的输出可以被视为与分支网络最后一层参数呈线性关系，因此这些参数可以通过最小二乘（LS）求解进行优化，而其余隐藏层参数则通过梯度下降方式进行更新。然而，为所有可能的分支和主干输入组合构建LS系统会产生一个规模过大且难以直接求解的线性问题。为了应对这一挑战，我们的方法将大型LS系统分解为两个较小、更易管理的子问题——一个是针对分支网络的问题，另一个是针对主干网络的问题，并分别解决这两个子问题。该方法还被推广到一种更为广泛的$L^2$损失类型，包括最后一层参数的正则化项，在无监督学习中涉及物理信息损失的情况也适用。
URL
https://arxiv.org/abs/2508.15394
PDF
-
Twin-Boot: Uncertainty-Aware Optimization via Online Two-Sample Bootstrapping
2025-08-20 19:20:38Carlos Stein BritoAbstract
Standard gradient descent methods yield point estimates with no measure of confidence. This limitation is acute in overparameterized and low-data regimes, where models have many parameters relative to available data and can easily overfit. Bootstrapping is a classical statistical framework for uncertainty estimation based on resampling, but naively applying it to deep learning is impractical: it requires training many replicas, produces post-hoc estimates that cannot guide learning, and implicitly assumes comparable optima across runs - an assumption that fails in non-convex landscapes. We introduce Twin-Bootstrap Gradient Descent (Twin-Boot), a resampling-based training procedure that integrates uncertainty estimation into optimization. Two identical models are trained in parallel on independent bootstrap samples, and a periodic mean-reset keeps both trajectories in the same basin so that their divergence reflects local (within-basin) uncertainty. During training, we use this estimate to sample weights in an adaptive, data-driven way, providing regularization that favors flatter solutions. In deep neural networks and complex high-dimensional inverse problems, the approach improves calibration and generalization and yields interpretable uncertainty maps.
Abstract (translated)
标准梯度下降方法只能提供点估计，而没有置信度的衡量。这种限制在过度参数化和低数据量的情况下尤为明显，此时模型相对于可用的数据拥有大量参数，并且很容易过拟合。自助法（Bootstrapping）是一种基于重采样的经典统计框架，用于不确定性估计，但直接将其应用于深度学习是不切实际的：它需要训练许多复制模型，产生事后估计无法指导学习过程，而且隐含假设各次运行之间的最优解具有可比性——这一假设在非凸景观中往往不成立。我们引入了双自助梯度下降法（Twin-Bootstrap Gradient Descent, Twin-Boot），这是一种基于重采样的训练程序，它将不确定性估计融入到优化过程中。两个相同的模型并行训练，在独立的自助样本上进行，并通过周期性的均值重置来保持两者在同一个局部最优区域内，从而使它们之间的差异反映该区域内的局部（局域）不确定性。在训练过程中，我们利用这种估计以一种自适应和数据驱动的方式采样权重，提供了一种倾向于更平坦解的正则化方法。这种方法在深度神经网络以及复杂的高维逆问题中改善了校准性和泛化能力，并提供了可解释的不确定性图谱。
URL
https://arxiv.org/abs/2508.15019
PDF
-
Fisher-Orthogonal Projection Methods for Natural Gradient Descent with Large Batches
Abstract
Modern GPUs are equipped with large amounts of high-bandwidth memory, enabling them to support mini-batch sizes of up to tens of thousands of training samples. However, most existing optimizers struggle to perform effectively at such a large batch size. As batch size increases, gradient noise decreases due to averaging over many samples, limiting the ability of first-order methods to escape sharp or suboptimal minima and reach the global minimum. Meanwhile, second-order methods like the natural gradient with Kronecker-Factored Approximate Curvature (KFAC) often require excessively high damping to remain stable at large batch sizes. This high damping effectively washes out the curvature information that gives these methods their advantage, reducing their performance to that of simple gradient descent. In this paper, we introduce Fisher-Orthogonal Projection (FOP), a novel technique that restores the effectiveness of the second-order method at very large batch sizes, enabling scalable training with improved generalization and faster convergence. FOP constructs a variance-aware update direction by leveraging gradients from two sub-batches, enhancing the average gradient with a component of the gradient difference that is orthogonal to the average under the Fisher-metric.
Abstract (translated)
现代GPU配备了大量高带宽内存，能够支持多达数万个训练样本的小批量处理。然而，大多数现有的优化器在如此大的批量大小下难以有效运行。随着批处理量的增加，由于大量样本平均效应，梯度噪声减少，这限制了一阶方法（如随机梯度下降）从尖锐或次优极小值中逃脱并达到全局最小值的能力。同时，像自然梯度结合Kronecker-Factored Approximate Curvature (KFAC)这样的二阶方法，在处理大规模批量时往往需要过高的阻尼以保持稳定。这种高阻尼会抹去这些方法所依赖的曲率信息优势，使它们的表现与简单的梯度下降相差无几。 在本文中，我们介绍了Fisher-Orthogonal Projection（FOP），这是一种新颖的技术，能够在非常大的批处理大小下恢复二阶方法的有效性，从而实现具有改进泛化能力和更快收敛速度的大规模训练。FOP通过利用两个子批量的梯度来构造一个方差感知更新方向，并在Fisher度量下的平均梯度中增加了一个正交于平均值的梯度差异分量。
URL
https://arxiv.org/abs/2508.13898
PDF
-
Widening the Network Mitigates the Impact of Data Heterogeneity on FedAvg
Abstract
Federated learning (FL) enables decentralized clients to train a model collaboratively without sharing local data. A key distinction between FL and centralized learning is that clients' data are non-independent and identically distributed, which poses significant challenges in training a global model that generalizes well across heterogeneous local data distributions. In this paper, we analyze the convergence of overparameterized FedAvg with gradient descent (GD). We prove that the impact of data heterogeneity diminishes as the width of neural networks increases, ultimately vanishing when the width approaches infinity. In the infinite-width regime, we further prove that both the global and local models in FedAvg behave as linear models, and that FedAvg achieves the same generalization performance as centralized learning with the same number of GD iterations. Extensive experiments validate our theoretical findings across various network architectures, loss functions, and optimization methods.
Abstract (translated)
联邦学习（FL）允许分散的客户端在不共享本地数据的情况下协作训练模型。联邦学习与集中式学习之间的一个关键区别在于，客户端的数据是非独立且异构分布的，这给跨不同局部数据分布泛化的全局模型训练带来了重大挑战。 本文分析了使用梯度下降法（GD）进行超参数化FedAvg的收敛性。我们证明随着神经网络宽度的增加，数据异质性的负面影响逐渐减少，并在宽度趋于无穷大时完全消失。在网络宽度无限的情况下，我们进一步证明了FedAvg中的全局模型和局部模型都表现得像线性模型一样，并且当迭代次数相同时，FedAvg达到了与集中式学习相同的泛化性能。 广泛的实验验证了我们的理论发现，在各种网络架构、损失函数以及优化方法下均适用。
URL
https://arxiv.org/abs/2508.12576
PDF
-
A Fluid Antenna Enabled Physical Layer Key Generation for Next-G Wireless Networks
2025-08-16 18:39:19Jiacheng Guo, Ning Gao, Yiping Zuo, Hao Xu, Shi Jin, Kai Kit WongAbstract
As a promising physical layer security technique, physical layer key generation (PLKG) enables legitimate users to obtain secret keys from wireless channel without security infrastructures. However, in harsh propagation environments, the channel characteristic becomes unsatisfactory, the key generation rate (KGR) is significantly deteriorated. In this paper, we propose a novel fluid antenna (FA) enabled PLKG system to address this challenge. Specifically, we first derive the closed-form expression of the KGR for FA array, and then jointly optimize the precoding matrix and the antenna positions via a particle swarm optimization (PSO) algorithm. Next, to further reduce the computational complexity of the optimization procedure, we develop an alternating optimization (AO) algorithm, which combines the projected gradient descent (PGD) and the PSO. Simulation results demonstrate that by exploiting the additional spatial degree of freedom (DoF), our FA enabled PLKG system is superior to the benchmarks, such as the conventional fixed-position antenna (FPA) array and the reconfigurable intelligent surface (RIS). It is worth highlighting that compared to the conventional uniform planar antenna (UPA), the FA enabled PLKG achieves a 35.42\% KGR performance improvement under PSO algorithm and a 67.73\% KGR performance improvement under AO algorithm, respectively.
Abstract (translated)
作为一种有前景的物理层安全技术，物理层密钥生成（PLKG）使合法用户能够从无线信道中获取秘密密钥，而不依赖于任何安全基础设施。然而，在恶劣传播环境中，信道特性变得不佳，导致密钥生成率（KGR）显著下降。为此，本文提出了一种新颖的流体天线（FA）增强型PLKG系统来应对这一挑战。 具体而言，我们首先推导了FA阵列的闭式KGR表达式，并通过粒子群优化算法联合优化预编码矩阵和天线位置。接下来，为了进一步降低优化过程中的计算复杂度，我们开发了一种交替优化（AO）算法，该算法结合了投影梯度下降法（PGD）和粒子群优化算法。 仿真结果表明，在利用额外的空间自由度后，我们的FA增强型PLKG系统在性能上优于基准方案，如传统的固定位置天线（FPA）阵列以及可重构智能表面（RIS）。尤其值得注意的是，与传统均匀平面阵列（UPA）相比，基于FA的PLKG在使用PSO算法时，KGR性能提升了35.42%，而在采用AO算法时，则达到了67.73%的提升。
URL
https://arxiv.org/abs/2509.00018
PDF
-
Implicit Hypergraph Neural Network
2025-08-16 16:58:59Akash Choudhuri, Yongjian Zhong, Bijaya AdhikariAbstract
Hypergraphs offer a generalized framework for capturing high-order relationships between entities and have been widely applied in various domains, including healthcare, social networks, and bioinformatics. Hypergraph neural networks, which rely on message-passing between nodes over hyperedges to learn latent representations, have emerged as the method of choice for predictive tasks in many of these domains. These approaches typically perform only a small number of message-passing rounds to learn the representations, which they then utilize for predictions. The small number of message-passing rounds comes at a cost, as the representations only capture local information and forego long-range high-order dependencies. However, as we demonstrate, blindly increasing the message-passing rounds to capture long-range dependency also degrades the performance of hyper-graph neural networks. Recent works have demonstrated that implicit graph neural networks capture long-range dependencies in standard graphs while maintaining performance. Despite their popularity, prior work has not studied long-range dependency issues on hypergraph neural networks. Here, we first demonstrate that existing hypergraph neural networks lose predictive power when aggregating more information to capture long-range dependency. We then propose Implicit Hypergraph Neural Network (IHNN), a novel framework that jointly learns fixed-point representations for both nodes and hyperedges in an end-to-end manner to alleviate this issue. Leveraging implicit differentiation, we introduce a tractable projected gradient descent approach to train the model efficiently. Extensive experiments on real-world hypergraphs for node classification demonstrate that IHNN outperforms the closest prior works in most settings, establishing a new state-of-the-art in hypergraph learning.
Abstract (translated)
超图提供了一种捕捉实体之间高阶关系的通用框架，并在医疗保健、社交网络和生物信息学等各个领域得到了广泛应用。基于超边进行节点间消息传递以学习潜在表示的超图神经网络，已成为这些领域许多预测任务中的首选方法。这些方法通常只执行少量的消息传递轮次来学习表示，然后利用这些表示来进行预测。这种少量的消息传递轮次会导致表示仅捕获局部信息，并忽略长程高阶依赖关系。然而，我们证明盲目增加消息传递轮数以捕捉长程依赖同样会降低超图神经网络的性能。 近期工作表明，隐式图神经网络能够在标准图中捕获长距离依赖性的同时保持良好的性能表现，但之前的工作没有研究过超图神经网络中的远程依赖问题。在这里，我们首先证明现有的超图神经网络在聚合更多信息以捕捉长程依赖时会失去预测能力。然后，我们提出了一种新的框架——隐式超图神经网络（IHNN），该框架通过端到端的方式共同学习节点和超边的固定点表示，从而缓解这一问题。利用隐式微分，我们引入了一种可操作的投影梯度下降方法来高效训练模型。 在真实世界的超图上进行节点分类任务的广泛实验表明，在大多数情况下，IHNN超越了最近的相关工作，建立了新的最先进的技术成果。
URL
https://arxiv.org/abs/2508.14101
PDF
-
Provable In-Context Vector Arithmetic via Retrieving Task Concepts
2025-08-13 13:54:44Dake Bu, Wei Huang, Andi Han, Atsushi Nitanda, Qingfu Zhang, Hau-San Wong, Taiji SuzukiAbstract
In-context learning (ICL) has garnered significant attention for its ability to grasp functions/tasks from demonstrations. Recent studies suggest the presence of a latent task/function vector in LLMs during ICL. Merullo et al. (2024) showed that LLMs leverage this vector alongside the residual stream for Word2Vec-like vector arithmetic, solving factual-recall ICL tasks. Additionally, recent work empirically highlighted the key role of Question-Answer data in enhancing factual-recall capabilities. Despite these insights, a theoretical explanation remains elusive. To move one step forward, we propose a theoretical framework building on empirically grounded hierarchical concept modeling. We develop an optimization theory, showing how nonlinear residual transformers trained via gradient descent on cross-entropy loss perform factual-recall ICL tasks via vector arithmetic. We prove 0-1 loss convergence and show the strong generalization, including robustness to concept recombination and distribution shifts. These results elucidate the advantages of transformers over static embedding predecessors. Empirical simulations corroborate our theoretical insights.
Abstract (translated)
在上下文学习（ICL）中，模型从演示中掌握功能/任务的能力引起了广泛关注。近期研究表明，在进行ICL时，大型语言模型（LLM）内部存在一个潜在的任务/函数向量。Merullo等人(2024)发现，LLMs利用这一向量与残差流结合来进行类似Word2Vec的向量运算，从而完成事实回忆型ICL任务。此外，近期的工作通过实证研究强调了问答数据在提升事实回忆能力方面的重要作用。尽管有了这些见解，但理论解释仍然缺失。 为了向前推进一步，我们提出了一种基于经验基础分层概念建模的理论框架。我们开发了一个优化理论，展示了非线性残差变压器如何通过对交叉熵损失进行梯度下降训练来通过向量运算执行事实回忆型ICL任务。我们证明了0-1损失收敛，并表明其具有强大的泛化能力，包括对概念重组和分布变化的鲁棒性。这些结果阐明了变压器相对于静态嵌入前体的优势。 实证模拟验证了我们的理论见解。
URL
https://arxiv.org/abs/2508.09820
PDF
-
Physics-guided Deep Unfolding Network for Enhanced Kronecker Compressive sensing
2025-08-13 06:21:58Gang Qu, Ping Wang, Siming Zheng, Xin YuanAbstract
Deep networks have achieved remarkable success in image compressed sensing (CS) task, namely reconstructing a high-fidelity image from its compressed measurement. However, existing works are deficient inincoherent compressed measurement at sensing phase and implicit measurement representations at reconstruction phase, limiting the overall performance. In this work, we answer two questions: 1) how to improve the measurement incoherence for decreasing the ill-posedness; 2) how to learn informative representations from measurements. To this end, we propose a novel asymmetric Kronecker CS (AKCS) model and theoretically present its better incoherence than previous Kronecker CS with minimal complexity increase. Moreover, we reveal that the unfolding networks' superiority over non-unfolding ones result from sufficient gradient descents, called explicit measurement representations. We propose a measurement-aware cross attention (MACA) mechanism to learn implicit measurement representations. We integrate AKCS and MACA into widely-used unfolding architecture to get a measurement-enhanced unfolding network (MEUNet). Extensive experiences demonstrate that our MEUNet achieves state-of-the-art performance in reconstruction accuracy and inference speed.
Abstract (translated)
深度网络在图像压缩感知（CS）任务中取得了显著的成功，即从其压缩测量值重构出高质量的图像。然而，现有的研究工作在传感阶段缺乏不相干的压缩测量，在重建阶段也未能明确表示测量结果，这限制了整体性能的表现。本文旨在回答两个问题：1）如何改进测量的不相关性以减少不适定性；2）如何从测量数据中学习出具有信息量的表示。 为此，我们提出了一个新颖的非对称克罗内克压缩感知（AKCS）模型，并从理论上证明了其比之前的克罗内克压缩感知方法在复杂度最小增加的情况下具备更好的不相关性。此外，我们揭示了解折叠网络优于非解折叠网络的原因在于充分的梯度下降过程，即显式的测量表示。 为了学习隐含的测量表示，我们提出了一个基于测量的认知交叉注意力（MACA）机制。我们将AKCS和MACA整合到广泛使用的解折叠架构中，得到了一种增强测量信息的解折叠网络（MEUNet）。大量的实验经验表明，我们的MEUNet在重建精度和推理速度方面达到了最先进的性能水平。
URL
https://arxiv.org/abs/2508.09528
PDF
-
GreenTEA: Gradient Descent with Topic-modeling and Evolutionary Auto-prompting
2025-08-12 06:48:30Zheng Dong, Luming Shang, Gabriela OlintoAbstract
High-quality prompts are crucial for Large Language Models (LLMs) to achieve exceptional performance. However, manually crafting effective prompts is labor-intensive and demands significant domain expertise, limiting its scalability. Existing automatic prompt optimization methods either extensively explore new prompt candidates, incurring high computational costs due to inefficient searches within a large solution space, or overly exploit feedback on existing prompts, risking suboptimal optimization because of the complex prompt landscape. To address these challenges, we introduce GreenTEA, an agentic LLM workflow for automatic prompt optimization that balances candidate exploration and knowledge exploitation. It leverages a collaborative team of agents to iteratively refine prompts based on feedback from error samples. An analyzing agent identifies common error patterns resulting from the current prompt via topic modeling, and a generation agent revises the prompt to directly address these key deficiencies. This refinement process is guided by a genetic algorithm framework, which simulates natural selection by evolving candidate prompts through operations such as crossover and mutation to progressively optimize model performance. Extensive numerical experiments conducted on public benchmark datasets suggest the superior performance of GreenTEA against human-engineered prompts and existing state-of-the-arts for automatic prompt optimization, covering logical and quantitative reasoning, commonsense, and ethical decision-making.
Abstract (translated)
高质量的提示对于大型语言模型（LLM）实现卓越性能至关重要。然而，手动设计有效的提示既费时又需要深厚的专业知识，这限制了其可扩展性。现有的自动提示优化方法要么广泛探索新的提示候选方案，在庞大的解决方案空间中进行低效搜索导致计算成本高昂；要么过度利用现有提示的反馈，由于复杂的提示景观而面临次优优化的风险。为了解决这些挑战，我们引入了一种名为GreenTEA的代理LLM工作流程，用于自动提示优化，它在候选探索和知识利用之间取得了平衡。 GreenTEA通过一个协作的智能体团队根据错误样本的反馈迭代地改进提示。其中，分析智能体通过主题建模识别当前提示产生的常见错误模式，而生成智能体则修订提示以直接解决这些关键缺陷。该优化过程由遗传算法框架引导，模拟自然选择，在交叉和变异等操作下进化候选提示，逐步优化模型性能。 在公共基准数据集上进行的大量数值实验表明，GreenTEA相对于人为设计的提示以及现有的自动提示优化方法中的最新技术具有更优越的表现，涵盖了逻辑和定量推理、常识及伦理决策等领域。
URL
https://arxiv.org/abs/2508.16603
PDF
-
Fast weight programming and linear transformers: from machine learning to neurobiology
2025-08-11 19:50:03Kazuki Irie, Samuel J. GershmanAbstract
Recent advances in artificial neural networks for machine learning, and language modeling in particular, have established a family of recurrent neural network (RNN) architectures that, unlike conventional RNNs with vector-form hidden states, use two-dimensional (2D) matrix-form hidden states. Such 2D-state RNNs, known as Fast Weight Programmers (FWPs), can be interpreted as a neural network whose synaptic weights (called fast weights) dynamically change over time as a function of input observations, and serve as short-term memory storage; corresponding synaptic weight modifications are controlled or programmed by another network (the programmer) whose parameters are trained (e.g., by gradient descent). In this Primer, we review the technical foundations of FWPs, their computational characteristics, and their connections to transformers and state space models. We also discuss connections between FWPs and models of synaptic plasticity in the brain, suggesting a convergence of natural and artificial intelligence.
Abstract (translated)
最近在机器学习领域，特别是在语言模型方面的进展，确立了一类循环神经网络（RNN）架构。与传统的使用向量形式隐状态的RNN不同，这类新架构采用二维矩阵形式的隐藏状态。这种具有2D状态的RNN被称为快速权重编程器（FWPs），可以被解释为一个其突触权重（称为快权重）会随输入观察结果动态变化的神经网络，并且这些权重作为短期记忆存储；相应的突触权重调整由另一个网络（即编程器）控制，该网络的参数通过训练获得（例如，通过梯度下降法）。在本文中，我们回顾了FWPs的技术基础、计算特性以及它们与变压器和状态空间模型之间的联系。此外，我们还讨论了FWPs与大脑突触可塑性模型之间的关联，这表明自然智能和人工智能正在趋于融合。
URL
https://arxiv.org/abs/2508.08435
PDF
-
Multi-head Transformers Provably Learn Symbolic Multi-step Reasoning via Gradient Descent
2025-08-11 17:40:47Tong Yang, Yu Huang, Yingbin Liang, Yuejie ChiAbstract
Transformers have demonstrated remarkable capabilities in multi-step reasoning tasks. However, understandings of the underlying mechanisms by which they acquire these abilities through training remain limited, particularly from a theoretical standpoint. This work investigates how transformers learn to solve symbolic multi-step reasoning problems through chain-of-thought processes, focusing on path-finding in trees. We analyze two intertwined tasks: a backward reasoning task, where the model outputs a path from a goal node to the root, and a more complex forward reasoning task, where the model implements two-stage reasoning by first identifying the goal-to-root path and then reversing it to produce the root-to-goal path. Our theoretical analysis, grounded in the dynamics of gradient descent, shows that trained one-layer transformers can provably solve both tasks with generalization guarantees to unseen trees. In particular, our multi-phase training dynamics for forward reasoning elucidate how different attention heads learn to specialize and coordinate autonomously to solve the two subtasks in a single autoregressive path. These results provide a mechanistic explanation of how trained transformers can implement sequential algorithmic procedures. Moreover, they offer insights into the emergence of reasoning abilities, suggesting that when tasks are structured to take intermediate chain-of-thought steps, even shallow multi-head transformers can effectively solve problems that would otherwise require deeper architectures.
Abstract (translated)
Transformer模型在多步推理任务中表现出色，但它们通过训练获得这些能力的内在机制，尤其是在理论层面上的理解仍然有限。本研究探讨了Transformer是如何通过链式思维过程学习解决符号化多步推理问题的，特别是关注于树形结构中的路径寻找。我们分析了两个相互交织的任务：一个是反向推理任务，在此任务中模型输出从目标节点到根节点的路径；另一个是更复杂的正向推理任务，即模型首先识别出从目标节点到根节点的路径，然后反转这条路径以生成从根节点到目标节点的路径。基于梯度下降动力学的理论分析表明，经过训练的一层Transformer可以证明解决这两个任务，并且对未见过的树形结构具有泛化保证。 特别地，我们为正向推理开发出一个多阶段训练动态过程，阐明了不同的注意力头如何自主学习专业化并协作以在单一自回归路径中解决两个子任务。这些结果提供了关于经过训练的Transformer如何实现顺序算法程序的具体机制解释。此外，它们还为推理能力的出现提供了见解：当任务结构被设计成需要中间链式思维步骤时，即使是浅层多头Transformer也能有效解决问题，而这些问题在其他情况下可能需要更深层的架构来解决。
URL
https://arxiv.org/abs/2508.08222
PDF
-
SGD Convergence under Stepsize Shrinkage in Low-Precision Training
2025-08-10 02:25:48Vincent-Daniel YunAbstract
Low-precision training has become essential for reducing the computational and memory costs of large-scale deep learning. However, quantization of gradients introduces both magnitude shrinkage and additive noise, which can alter the convergence behavior of stochastic gradient descent (SGD). In this work, we study the convergence of SGD under a gradient shrinkage model, where each stochastic gradient is scaled by a factor $q_k \in (0,1]$ and perturbed by zero-mean quantization noise. We show that this shrinkage is equivalent to replacing the nominal stepsize $\mu_k$ with an effective stepsize $\mu_k q_k$, which slows convergence when $q_{\min} < 1$. Under standard smoothness and bounded-variance assumptions, we prove that low-precision SGD still converges, but at a reduced rate determined by $q_{\min}$, and with an increased asymptotic error floor due to quantization noise. We theoretically analyze how reduced numerical precision slows down training by modeling it as gradient shrinkage in the standard SGD convergence framework.
Abstract (translated)
低精度训练已成为减少大规模深度学习计算和内存成本的关键方法。然而，梯度量化会引入幅度衰减以及加性噪声，这些都可能改变随机梯度下降（SGD）的收敛行为。在这项工作中，我们研究了在一种梯度缩放模型下的SGD收敛性，在这种模型中，每个随机梯度被一个因子$q_k \in (0,1]$ 缩小，并且受到零均值量化噪声的影响。 我们的研究表明，该缩放等同于将名义学习率$\mu_k$替换为有效学习率$\mu_k q_k$。当最小的缩放因子$q_{\min} < 1$时，这会导致收敛速度变慢。在标准平滑性和有界方差假设下，我们证明了低精度SGD仍然能够收敛，但以一个由$q_{\min}$决定的速度减缓，并且由于量化噪声导致最终误差增加。 从理论上讲，我们通过将数值精度降低建模为标准SGD收敛框架中的梯度缩放来分析其如何减慢训练速度。
URL
https://arxiv.org/abs/2508.07142
PDF
-
FIVA: Federated Inverse Variance Averaging for Universal CT Segmentation with Uncertainty Estimation
2025-08-08 11:34:01Asim Ukaye, Numan Saeed, Karthik NandakumarAbstract
Different CT segmentation datasets are typically obtained from different scanners under different capture settings and often provide segmentation labels for a limited and often disjoint set of organs. Using these heterogeneous data effectively while preserving patient privacy can be challenging. This work presents a novel federated learning approach to achieve universal segmentation across diverse abdominal CT datasets by utilizing model uncertainty for aggregation and predictive uncertainty for inference. Our approach leverages the inherent noise in stochastic mini-batch gradient descent to estimate a distribution over the model weights to provide an on-the-go uncertainty over the model parameters at the client level. The parameters are then aggregated at the server using the additional uncertainty information using a Bayesian-inspired inverse-variance aggregation scheme. Furthermore, the proposed method quantifies prediction uncertainty by propagating the uncertainty from the model weights, providing confidence measures essential for clinical decision-making. In line with recent work shown, predictive uncertainty is utilized in the inference stage to improve predictive performance. Experimental evaluations demonstrate the effectiveness of this approach in improving both the quality of federated aggregation and uncertainty-weighted inference compared to previously established baselines. The code for this work is made available at: this https URL
Abstract (translated)
不同的CT分割数据集通常来自不同类型的扫描仪和不同的采集设置，并且往往只为有限的、不连续的一组器官提供分割标签。有效地使用这些异质性数据并同时保护患者隐私是一项挑战。这项工作提出了一种新颖的联邦学习方法，通过利用模型不确定性和预测不确定性来实现跨多样腹部CT数据集的通用分割。我们的方法利用随机小批量梯度下降中的固有噪声来估计模型权重上的分布，从而在客户端级别实时提供关于模型参数的不确定性。然后，在服务器端使用基于贝叶斯的思想和逆方差聚合方案来整合这些参数，并结合额外的不确定性信息进行汇总。 此外，所提出的方法通过从模型权重中传播不确定性来量化预测中的不确定性和置信度量，这对临床决策至关重要。正如最近的工作所示，我们在推理阶段利用了预测的不确定性以提高预测性能。实验评估表明，与先前建立的基础方法相比，该方法在改善联邦聚合质量和基于不确定性的加权推断方面是有效的。 此工作的代码可在以下网址获取：this https URL
URL
https://arxiv.org/abs/2508.09196
PDF
-
Neural Field Representations of Mobile Computational Photography
Abstract
Over the past two decades, mobile imaging has experienced a profound transformation, with cell phones rapidly eclipsing all other forms of digital photography in popularity. Today's cell phones are equipped with a diverse range of imaging technologies - laser depth ranging, multi-focal camera arrays, and split-pixel sensors - alongside non-visual sensors such as gyroscopes, accelerometers, and magnetometers. This, combined with on-board integrated chips for image and signal processing, makes the cell phone a versatile pocket-sized computational imaging platform. Parallel to this, we have seen in recent years how neural fields - small neural networks trained to map continuous spatial input coordinates to output signals - enable the reconstruction of complex scenes without explicit data representations such as pixel arrays or point clouds. In this thesis, I demonstrate how carefully designed neural field models can compactly represent complex geometry and lighting effects. Enabling applications such as depth estimation, layer separation, and image stitching directly from collected in-the-wild mobile photography data. These methods outperform state-of-the-art approaches without relying on complex pre-processing steps, labeled ground truth data, or machine learning priors. Instead, they leverage well-constructed, self-regularized models that tackle challenging inverse problems through stochastic gradient descent, fitting directly to raw measurements from a smartphone.
Abstract (translated)
在过去二十年里，移动成像经历了深刻的变革，手机迅速超越了所有其他形式的数字摄影，在流行度上占据主导地位。今天的手机配备了各种成像技术——包括激光深度测距、多焦点相机阵列和分隔像素传感器——以及非视觉传感器，如陀螺仪、加速度计和磁力计。结合内置的图像和信号处理集成电路芯片，使手机成为一种多功能的手持式计算成像平台。 与此同时，在近年来我们见证了神经场（小型经过训练以将连续空间输入坐标映射到输出信号的小型神经网络）的发展，它们能够无需显式的数据表示（如像素阵列或点云）来重建复杂场景。在我的论文中，我展示了精心设计的神经场模型如何紧凑地代表复杂的几何结构和光照效果，并支持从收集的实际移动摄影数据直接进行深度估计、图层分离和图像拼接等应用。 这些方法在不依赖复杂的预处理步骤、标记的真实数据或机器学习先验的情况下超越了现有最佳的方法。相反，它们利用精心构造的、自我调节的模型通过随机梯度下降直接拟合智能手机采集的原始测量值来解决具有挑战性的逆向问题。
URL
https://arxiv.org/abs/2508.05907
PDF
-
Comparative Analysis of Novel NIRMAL Optimizer Against Adam and SGD with Momentum
2025-08-06 10:30:22Nirmal Gaud, Surej Mouli, Preeti Katiyar, Vaduguru Venkata RamyaAbstract
This study proposes NIRMAL (Novel Integrated Robust Multi-Adaptation Learning), a novel optimization algorithm that combines multiple strategies inspired by the movements of the chess piece. These strategies include gradient descent, momentum, stochastic perturbations, adaptive learning rates, and non-linear transformations. We carefully evaluated NIRMAL against two widely used and successful optimizers, Adam and SGD with Momentum, on four benchmark image classification datasets: MNIST, FashionMNIST, CIFAR-10, and CIFAR-100. The custom convolutional neural network (CNN) architecture is applied on each dataset. The experimental results show that NIRMAL achieves competitive performance, particularly on the more challenging CIFAR-100 dataset, where it achieved a test accuracy of 45.32\%and a weighted F1-score of 0.4328. This performance surpasses Adam (41.79\% accuracy, 0.3964 F1-score) and closely matches SGD with Momentum (46.97\% accuracy, 0.4531 F1-score). Also, NIRMAL exhibits robust convergence and strong generalization capabilities, especially on complex datasets, as evidenced by stable training results in loss and accuracy curves. These findings underscore NIRMAL's significant ability as a versatile and effective optimizer for various deep learning tasks.
Abstract (translated)
这项研究提出了NIRMAL（新颖的综合鲁棒多适应学习），这是一种新的优化算法，结合了多种策略，这些策略受到了国际象棋棋子移动方式的启发。这些策略包括梯度下降、动量、随机扰动、自适应学习率和非线性变换。我们仔细地将NIRMAL与两种广泛使用且成功的优化器——Adam和带动量的SGD，在四个基准图像分类数据集上进行了比较：MNIST、FashionMNIST、CIFAR-10 和 CIFAR-100。针对每个数据集，应用了自定义的卷积神经网络（CNN）架构。实验结果表明，NIRMAL实现了具有竞争力的表现，特别是在更具挑战性的CIFAR-100 数据集中，它的测试准确率为45.32%，加权F1分数为0.4328。这一性能超过了Adam（准确率 41.79%、F1 分数 0.3964）并且接近于带动量的SGD（准确率 46.97%、F1 分数 0.4531）。此外，NIRMAL展示了鲁棒收敛和强大的泛化能力，尤其是在复杂数据集上，这一点通过损失和精度曲线中的稳定训练结果得到了证实。这些发现强调了 NIRMAL 在各种深度学习任务中作为一种灵活且有效的优化器的重要性。
URL
https://arxiv.org/abs/2508.04293
PDF
-
When Good Sounds Go Adversarial: Jailbreaking Audio-Language Models with Benign Inputs
2025-08-05 12:14:01Bodam Kim, Hiskias Dingeto, Taeyoun Kwon, Dasol Choi, DongGeon Lee, Haon Park, JaeHoon Lee, Jongho ShinAbstract
As large language models become increasingly integrated into daily life, audio has emerged as a key interface for human-AI interaction. However, this convenience also introduces new vulnerabilities, making audio a potential attack surface for adversaries. Our research introduces WhisperInject, a two-stage adversarial audio attack framework that can manipulate state-of-the-art audio language models to generate harmful content. Our method uses imperceptible perturbations in audio inputs that remain benign to human listeners. The first stage uses a novel reward-based optimization method, Reinforcement Learning with Projected Gradient Descent (RL-PGD), to guide the target model to circumvent its own safety protocols and generate harmful native responses. This native harmful response then serves as the target for Stage 2, Payload Injection, where we use Projected Gradient Descent (PGD) to optimize subtle perturbations that are embedded into benign audio carriers, such as weather queries or greeting messages. Validated under the rigorous StrongREJECT, LlamaGuard, as well as Human Evaluation safety evaluation framework, our experiments demonstrate a success rate exceeding 86% across Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and Phi-4-Multimodal. Our work demonstrates a new class of practical, audio-native threats, moving beyond theoretical exploits to reveal a feasible and covert method for manipulating AI behavior.
Abstract (translated)
随着大型语言模型在日常生活中越来越普及，音频已成为人机交互的关键接口。然而，这种便利性也带来了新的安全漏洞，使音频成为潜在的攻击面。我们的研究介绍了WhisperInject，这是一种两阶段的对抗性音频攻击框架，可以操纵最先进的音频语言模型生成有害内容。我们使用了微小的音频输入扰动，这些扰动对人类听者来说是无害且不易察觉的。 第一阶段采用了一种基于奖励的优化方法——强化学习与投影梯度下降（RL-PGD），引导目标模型绕过自身的安全协议并生成有害响应。这种有害响应随后成为第二阶段“载荷注入”的攻击目标，在此阶段中，我们使用投影梯度下降（PGD）来优化嵌入到无害音频载体中的细微扰动，例如天气查询或问候信息。 在严格的StrongREJECT、LlamaGuard以及人类评估安全框架下验证后，我们的实验结果显示，WhisperInject在Qwen2.5-Omni-3B、Qwen2.5-Omni-7B和Phi-4-Multimodal模型上的成功率达到超过86%。这项工作揭示了一类新的实用且原生的音频威胁，并从理论性攻击方法转向了实际可行且隐蔽地操纵AI行为的方法。
URL
https://arxiv.org/abs/2508.03365
PDF
-
Adversarial Attacks on Reinforcement Learning-based Medical Questionnaire Systems: Input-level Perturbation Strategies and Medical Constraint Validation
2025-08-05 11:10:49Peizhuo LiuAbstract
RL-based medical questionnaire systems have shown great potential in medical scenarios. However, their safety and robustness remain unresolved. This study performs a comprehensive evaluation on adversarial attack methods to identify and analyze their potential vulnerabilities. We formulate the diagnosis process as a Markov Decision Process (MDP), where the state is the patient responses and unasked questions, and the action is either to ask a question or to make a diagnosis. We implemented six prevailing major attack methods, including the Fast Gradient Signed Method (FGSM), Projected Gradient Descent (PGD), Carlini & Wagner Attack (C&W) attack, Basic Iterative Method (BIM), DeepFool, and AutoAttack, with seven epsilon values each. To ensure the generated adversarial examples remain clinically plausible, we developed a comprehensive medical validation framework consisting of 247 medical constraints, including physiological bounds, symptom correlations, and conditional medical constraints. We achieved a 97.6% success rate in generating clinically plausible adversarial samples. We performed our experiment on the National Health Interview Survey (NHIS) dataset (this https URL), which consists of 182,630 samples, to predict the participant's 4-year mortality rate. We evaluated our attacks on the AdaptiveFS framework proposed in arXiv:2004.00994. Our results show that adversarial attacks could significantly impact the diagnostic accuracy, with attack success rates ranging from 33.08% (FGSM) to 64.70% (AutoAttack). Our work has demonstrated that even under strict medical constraints on the input, such RL-based medical questionnaire systems still show significant vulnerabilities.
Abstract (translated)
基于强化学习（RL）的医疗问卷系统在医学场景中展现了巨大的潜力，但其安全性和鲁棒性问题尚未解决。本研究对对抗攻击方法进行了全面评估，以识别和分析潜在的安全漏洞。我们把诊断过程描述为一个马尔可夫决策过程（MDP），其中状态是患者的回答以及未问的问题，动作则是提问或做出诊断。 我们在六种主要的攻击方法上进行实施：快速梯度符号法（FGSM）、投影梯度下降（PGD）、Carlini & Wagner 攻击（C&W）攻击、基本迭代方法（BIM）、DeepFool 和 AutoAttack。每种方法都使用了七个不同的ε值。 为了确保生成的对抗样本在临床上合理，我们建立了一个包含247项医学限制条件的全面验证框架，包括生理界限、症状相关性和条件性医疗限制。我们实现了97.6%的成功率，使得生成的对抗样本保持临床合理性。 我们的实验使用了全国健康访谈调查（NHIS）数据集进行，该数据集包含了182,630个样本，并预测参与者的四年死亡率。我们在提出的AdaptiveFS框架上评估了我们的攻击方法，发现这些攻击可以显著影响诊断准确性，成功率从FGSM的33.08%到AutoAttack的64.70%不等。 这项研究表明，即使在输入数据上有严格的医学限制条件下，基于强化学习的医疗问卷系统仍然存在明显的安全漏洞。
URL
https://arxiv.org/abs/2508.05677
PDF
-
How Diffusion Prior Landscapes Shape the Posterior in Blind Deconvolution
2025-08-04 21:53:12Minh-Hai Nguyen, Edouard Pauwels, Pierre WeissAbstract
The Maximum A Posteriori (MAP) estimation is a widely used framework in blind deconvolution to recover sharp images from blurred observations. The estimated image and blur filter are defined as the maximizer of the posterior distribution. However, when paired with sparsity-promoting image priors, MAP estimation has been shown to favors blurry solutions, limiting its effectiveness. In this paper, we revisit this result using diffusion-based priors, a class of models that capture realistic image distributions. Through an empirical examination of the prior's likelihood landscape, we uncover two key properties: first, blurry images tend to have higher likelihoods; second, the landscape contains numerous local minimizers that correspond to natural images. Building on these insights, we provide a theoretical analysis of the blind deblurring posterior. This reveals that the MAP estimator tends to produce sharp filters (close to the Dirac delta function) and blurry solutions. However local minimizers of the posterior, which can be obtained with gradient descent, correspond to realistic, natural images, effectively solving the blind deconvolution problem. Our findings suggest that overcoming MAP's limitations requires good local initialization to local minima in the posterior landscape. We validate our analysis with numerical experiments, demonstrating the practical implications of our insights for designing improved priors and optimization techniques.
Abstract (translated)
最大后验估计（MAP）是盲去卷积中广泛使用的一种框架，用于从模糊观测中恢复清晰图像。在该框架下，估计的图像和模糊滤波器被定义为后验分布的最大值。然而，当与促进图像稀疏性的先验条件结合时，MAP估计会倾向于产生模糊解，从而限制了其有效性。 本文重新审视了这一结果，并使用基于扩散的先验模型进行分析。这是一种捕捉现实图像分布的模型类型。通过实证检查先验似然景观，我们发现了两个关键特性：首先，模糊图像往往具有更高的可能性；其次，该景观包含大量局部极小值点，这些极小值对应于自然图像。 基于这些见解，我们对盲去卷积后验进行了理论分析。这揭示了MAP估计器倾向于产生锐利的滤波器（接近狄拉克δ函数）和模糊解。然而，通过梯度下降等方法可以获取到后验中的局部极小值点，这些局部极小值对应于真实的自然图像，从而有效解决了盲去卷积问题。 我们的研究结果表明，克服MAP局限性的关键是找到一个好的局部初始化以到达后验景观中的局部最小值点。我们通过数值实验验证了我们的分析，并展示了我们的见解对设计改进的先验条件和优化技术的实际意义。
URL
https://arxiv.org/abs/2508.02923
PDF
