Abstract
State-of-the-art speech-to-text models typically employ Transformer-based encoders that model token dependencies via self-attention mechanisms. However, the quadratic complexity of self-attention in both memory and computation imposes significant constraints on scalability. In this work, we propose a novel token-mixing mechanism, the Polynomial Mixer (PoM), as a drop-in replacement for multi-head self-attention. PoM computes a polynomial representation of the input with linear complexity with respect to the input sequence length. We integrate PoM into a self-supervised speech representation learning framework based on BEST-RQ and evaluate its performance on downstream speech recognition tasks. Experimental results demonstrate that PoM achieves a competitive word error rate compared to full self-attention and other linear-complexity alternatives, offering an improved trade-off between performance and efficiency in time and memory.
Abstract (translated)
最先进的语音转文本模型通常采用基于Transformer的编码器，通过自注意力机制来建模标记依赖关系。然而，自注意力在内存和计算上的二次复杂度对可扩展性带来了显著限制。在这项工作中，我们提出了一种新颖的标记混合机制——多项式混合器（Polynomial Mixer, PoM），作为多头自注意力的即插即用替代方案。PoM以线性复杂度计算输入的多项式表示，相对于输入序列长度而言。我们将PoM整合进一个基于BEST-RQ的自监督语音表征学习框架中，并对其在下游语音识别任务上的性能进行评估。实验结果表明，PoM在词错误率上与完整的自注意力和其他线性复杂度替代方案相当，但在时间和内存效率方面提供了更好的性能-效率平衡。
URL
https://arxiv.org/abs/2603.00683
