Abstract
Spatial functional organization is a hallmark of biological brains: neurons are arranged topographically according to their response properties, at multiple scales. In contrast, representations within most machine learning models lack spatial biases, instead manifesting as disorganized vector spaces that are difficult to visualize and interpret. Here, we propose a novel form of self-attention that turns Transformers into "Topoformers" with topographic organization. We introduce spatial querying - where keys and queries are arranged on 2D grids, and local pools of queries are associated with a given key - and spatial reweighting, where we convert the standard fully connected layer of self-attention into a locally connected layer. We first demonstrate the feasibility of our approach by training a 1-layer Topoformer on a sentiment classification task. Training with spatial querying encourages topographic organization in the queries and keys, and spatial reweighting separately encourages topographic organization in the values and self-attention outputs. We then apply the Topoformer motifs at scale, training a BERT architecture with a masked language modeling objective. We find that the topographic variant performs on par with a non-topographic control model on NLP benchmarks, yet produces interpretable topographic organization as evaluated via eight linguistic test suites. Finally, analyzing an fMRI dataset of human brain responses to a large set of naturalistic sentences, we demonstrate alignment between low-dimensional topographic variability in the Topoformer model and human brain language network. Scaling up Topoformers further holds promise for greater interpretability in NLP research, and for more accurate models of the organization of linguistic information in the human brain.
Abstract (translated)
空间功能组织是生物大脑的一个显著特征：根据神经元的响应特性，它们在多个尺度上按拓扑结构排列。相比之下，大多数机器学习模型中的表示缺乏空间偏差，反而表现为难以可视化和解释的无序向量空间。在这里，我们提出了一种新颖的自我注意形式，将Transformer转化为具有拓扑组织的"Topoformer"。我们引入了空间查询——其中键和查询在二维网格上排列，并且特定键与局部查询池相关联——以及空间重新加权，即将自注意力的标准全连接层转换为局部连接层。首先，我们通过在一个情感分类任务上训练一个单层的Topoformer来证明我们的方法的可行性。使用空间查询进行训练鼓励在查询和键中形成拓扑组织，并且空间重新加权单独地促进了值和自我注意输出中的拓扑组织。然后，我们在大规模上应用了Topoformer模式，在BERT架构的基础上以掩码语言建模为目标进行了训练。我们发现，拓扑变体在NLP基准测试中与非拓扑对照模型的表现相当，但通过八套语言测试评估时产生了可解释的拓扑结构。最后，通过对大量自然句子的人脑功能磁共振成像（fMRI）数据集进行分析，我们展示了Topoformer模型中的低维拓扑变化与人类大脑语言网络之间的对齐关系。扩大Topoformers的规模有望进一步提高NLP研究的可解释性，并为更好地描述人脑中语言信息组织提供更准确的模型。
URL
https://arxiv.org/abs/2510.18745
