Abstract
Existing adaptation techniques typically require architectural modifications or added parameters, leading to high computational costs and complexity. We introduce Attention Projection Layer Adaptation (APLA), a simple approach to adapt vision transformers (ViTs) without altering the architecture or adding parameters. Through a systematic analysis, we find that the layer immediately after the attention mechanism is crucial for adaptation. By updating only this projection layer, or even just a random subset of this layer's weights, APLA achieves state-of-the-art performance while reducing GPU memory usage by up to 52.63% and training time by up to 43.0%, with no extra cost at inference. Across 46 datasets covering a variety of tasks including scene classification, medical imaging, satellite imaging, and fine-grained classification, APLA consistently outperforms 17 other leading adaptation methods, including full fine-tuning, on classification, segmentation, and detection tasks. The code is available at this https URL.
Abstract (translated)
现有的适应技术通常需要进行架构修改或添加参数，这导致了高昂的计算成本和复杂性。我们引入了一种简单的方法——注意力投影层适配（Attention Projection Layer Adaptation, APLA），这种方法可以对视觉变换器（Vision Transformers, ViTs）进行调整而无需更改其架构或增加额外的参数。通过系统的分析，我们发现紧随注意机制之后的那一层对于适应至关重要。仅更新这一层中的投影层，甚至只是其中权重的一个随机子集，APLA就能达到最先进的性能，并且在不增加推理成本的情况下，最多可以减少52.63%的GPU内存使用和43.0%的训练时间。在涵盖场景分类、医学成像、卫星成像以及细粒度分类等任务的46个数据集中，APLA在分类、分割和检测任务上始终优于包括全量微调在内的17种其他领先适应方法。代码可在[此处](https://example.com/code)获取。
URL
https://arxiv.org/abs/2503.11335
