Speech
-
Select or Rewrite: Oracle-Guided Fine-Tuning for Generative ASR Post-Editing from N-Best Hypotheses
2026-10-10 16:48:27Shiran Liu, Gaohui Xu, Sergio Burdisso, Thibault Ba\~neras-Roux, Shashi Kumar, Petr MotlicekAbstract
Large language models can post-edit ASR by using linguistic context to compare N-best hypotheses and rewrite the transcript when needed, often improving WER without retraining the recognizer. However, prompt-only correction is brittle: it may over-edit when top-1 is correct and hallucinate when the reference is absent, and often needs very large models. We propose oracle-guided fine-tuning that turns N-best lists into supervision via three curated pools: corrections (reference in the list but not top-1) to teach selection, regularizers (top-1 correct) to discourage unnecessary rewrites, and hard negatives (reference absent) to suppress hallucination. Fine-tuning on this data provides more effective supervision than random subset fine-tuning for learning a hybrid select-or-rewrite policy, with the clearest gains from around 3B parameters on the proprietary dataset but less conclusive results on CommonVoice, a divergence suggesting that public read-speech text may overlap with LLM pretraining and mask genuine post-editing gains.
Abstract (translated)
URL
https://arxiv.org/abs/2610.15056
PDF
-
SemiCodec: Bridging Speech Tokenization and High-fidelity Reconstruction with Semi-discrete Representations
2026-10-10 11:52:24Kecan Mao, Nan Xu, Zhaoyang Meng, Zhengyao Ma, Yingming Gao, Yi Zhao, Ya LiAbstract
Speech representation learning requires a discrete modeling interface while preserving high waveform fidelity. Existing approaches rarely achieve both: discrete tokens ease modeling but lose fine-grained acoustic information, while continuous latents preserve fidelity but require specialized generative objectives. We introduce SemiCodec, a novel Semi-discrete neural speech Codec with synchronized discrete tokens and continuous latents. Direct joint optimization may create a reconstruction shortcut through the continuous pathway, leaving the discrete tokens insufficiently grounded. To address this issue, we propose a Preserve-then-Ground training paradigm that first establishes a high-fidelity synthesis space and then grounds SSL-guided discrete tokens within it. Experiments show that SemiCodec outperforms fully discrete codecs in reconstruction quality and intelligibility while remaining competitive with continuous representations in acoustic fidelity and speaker preservation. Audio samples are available at this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2610.14834
PDF
-
Cross-Lingual TTS for Indian Languages: A Benchmark and Inference-Time Duration Estimation Strategies
2026-10-10 11:41:56Adarsh Arigala, Arjun Gangwar, Srinivasan UmeshAbstract
Recent alignment-free Text-to-Speech (TTS) models achieve high synthesis quality by removing explicit alignment modules and simplifying training. However, they often struggle in cross-lingual scenarios due to differences in character durations across languages, leading to duration mismatch and degraded speech quality. Existing approaches typically address this issue using explicit duration predictors, increasing model complexity. Systematic evaluation of TTS performance for Indian languages in cross-lingual settings remains under-explored due to the lack of standardized evaluation benchmarks. In this work, we construct a cross-lingual evaluation benchmark covering English and nine Indian languages and propose a language-aware inference-time duration estimation strategy for alignment-free TTS that incorporates language-specific duration-per character statistics without retraining or architectural modifications. We further develop heuristic formulations that refine duration estimation based on language-dependent duration variations. Experiments on bilingual and multilingual A-DMA models demonstrate improved speech naturalness, intelligibility, and speaker similarity while reducing duration mismatch.
Abstract (translated)
URL
https://arxiv.org/abs/2610.14826
PDF
-
Training a Compact, Strong, and Realistic Text-to-Speech Model
2026-10-10 09:59:46Runyuan Cai, Yu Lin, Yiming Wang, Xiaodong ZengAbstract
Text-to-speech (TTS) systems are conventionally optimized for intelligibility and speaker similarity. However, neither objective captures the prosodic detail that distinguishes human performances from synthetic speech, leaving a realism gap that larger models and higher-quality data can narrow but not close. In the absence of an objective that directly targets this gap, realism remains a persistent challenge in TTS. We present Audio8-TTS, a compact 0.6B-parameter model for zero-shot voice cloning that achieves state-of-the-art performance in intelligibility and speaker similarity, with particularly strong results in perceptual naturalness. To align the training signal with human perception rather than signal-level fidelity, we construct a quality ladder from widely used open-source TTS systems and train a realism critic to assess generated speech against matched human recordings of the same sentences. An identity-erasure objective discourages the critic from relying on generator fingerprints. We train the model in three stages: multilingual supervised fine-tuning; group relative policy optimization (GRPO) with word error rate (WER) and speaker similarity (SIM) rewards; and GRPO guided by the realism critic. The final stage improves perceived prosodic naturalness and speaker similarity while preserving WER performance.
Abstract (translated)
URL
https://arxiv.org/abs/2610.14776
PDF
-
AlignAudio: A Unified Dual-Alignment Framework for Noise-Robust Speech-Conditioned Generation
2026-10-10 08:15:57Yixuan Li, Xuenan Xu, Zeyu Xie, Mengyue WuAbstract
Speech-conditioned generation maps speech signals to target audio outputs. Prior end-to-end speech-conditioned generative models achieve high fidelity in clean acoustic conditions, but their performance degrades substantially under environmental noise. While prior noise-robust approaches have achieved significant success in feature-level purification, maintaining consistent generation under clean and noisy conditioning remains a key challenge. To address this, we propose AlignAudio, a noise-robust generative framework based on dual alignment: (i) aligning the representations of clean and noisy speech to preserve semantic cues, and (ii) enforcing generation-level consistency within a flow matching paradigm to improve conditioning fidelity. Experiments on speech-to-audio (STA) and voice conversion (VC) tasks demonstrate that AlignAudio maintains performance comparable to baselines on clean speech while significantly outperforming them across various noisy conditions. Furthermore, the proposed dual-alignment strategy is applicable to both single- and multi-stream speech conditioning and can be integrated into different flow matching-based generation frameworks and post-training settings. Audio demonstrations and qualitative comparisons: \href{this https URL}{\textcolor{cyan}{\textit{this https URL}}}.
Abstract (translated)
URL
https://arxiv.org/abs/2610.14697
PDF
-
VoTSpeech: Decoupling Voice Design from Speech Generation in Instruction-Following TTS
2026-10-10 08:13:00Wenbing Yang, Qihang Lu, Zihan Sun, Peilei Jia, Yingming Gao, Ya Li, Jun GaoAbstract
Instruction-following text-to-speech (TTS) typically leaves voice design implicit, as conventional speaker embeddings encode reference audio rather than text-driven targets. We propose VoTSpeech, which decouples voice design from speech generation via an explicit intermediate voice latent. Built on a shared language model backbone, VoTSpeech first predicts a continuous voice latent from text instructions using a flow-based module. This latent serves as a shared condition for both the language model and the Acoustic DiT, guiding autoregressive sequence modeling and acoustic latent generation, respectively, before an audio VAE decoder converts the acoustic latents into waveforms. To capture speaker identity and complementary acoustic nuances, the voice latent is jointly supervised by speaker embeddings and temporally pooled audio VAE features. With task-specific adaptation, VoTSpeech performs strongly on InstructTTSEval-ZH, demonstrating competitive capabilities against strong open-source models while maintaining high naturalness. These results confirm that explicit voice latent design and its conditioning interface effectively empower text-driven voice design for controllable speech synthesis. Code, model, and audio samples are available at this https URL .
Abstract (translated)
URL
https://arxiv.org/abs/2610.14694
PDF
-
RASSG: Repair-Aware Semantic Survival Gating for Streaming Tool Execution
Abstract
Streaming speech assistants can launch tools before users finish speaking, overlapping execution with incoming speech to reduce user waiting time. However, subsequent self-corrections can invalidate apparently stable arguments, causing stale execution whose irreversible effects persist even after the final call is corrected. We propose Repair-Aware Semantic Survival Gating (RASSG) to jointly control tool launch and cancellation during streaming speech. RASSG launches a tool when three consecutive speech recognition results arrive with identical nonempty arguments, then combines calibrated argument invalidity risk with execution costs to decide cancellation at the last feasible opportunity before effects become irreversible. Evaluation on ReCAST, FDB-v3, and Audio2Tool uses a controlled execution clock with fixed tool runtimes and cancellation parameters. On ReCAST's prospective-repair subset, RASSG reduces the stale execution rate from LocalAgreement-2's 50.00% to 3.15%, with a mean delay of 3.676s across successful trajectories. The stale reduction also holds against every early-execution baseline on both public speech datasets. Ablations with identical initial launches support active cancellation, showing the value of controlling ongoing calls before their effects become irreversible. Code and data are available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2610.14659
PDF
-
AegisVox: Factorized Privacy and Clinical Preservation for Long-Form Speech
2026-10-10 06:53:23Praphul Singh, Akshat Agarwal, Ratul Ghosh, Aashna Kanuga, Devashish Khatwani, Anurag Dwarakanath, Sri Gadde, Krishnaram KenthapadiAbstract
Clinical speech reveals identity through both spoken identifiers and voice, yet its pauses, delivery, and speaker roles can also carry useful clinical evidence. Protecting that speech therefore requires more than preserving its words. We present AegisVox, a factorized framework that assigns consistent surrogate voices and edits identifier-bearing turns using document-consistent replacements. Across 390 public simulated or non-patient conversations totaling 63.6 hours, a matched comparison of two speech renderers separates replacement realization from retained identifiers, speaker linkage, and clinical preservation. Text-to-speech rendering realizes 535/601 planned replacements versus 453/601 for the codec-based speech editor. However, similar aggregate clinical preservation conceals more model-labeled contradictions, and conversation-wide speaker scores conceal greater edited-speech linkage under a score-direction sensitivity analysis. Fresh transcription of original audio provides a reference for interpreting the clinical measurement chain; an independent judge tests sensitivity to model-based assessment. Better replacement realization can therefore coexist with worse clinical or speaker-linkage measurements. AegisVox supports use-specific evaluation of those tradeoffs, not a guarantee of identifier removal or clinical safety.
Abstract (translated)
URL
https://arxiv.org/abs/2610.14643
PDF
-
Window Size Matters: Benchmarking Voice Activity Detectors in Noise
2026-10-10 05:51:27Alok Kumar, Mohit Srivastava, Anupam PurwarAbstract
Voice activity detection (VAD) sits in front of almost every streaming speech recognizer, yet the choice of detector is often made by feel rather than by measurement. We report a frame-level offline benchmark of three detectors on the test-clean split of LibriVAD-small: the neural TEN VAD, the neural Silero VAD and a simple RMS energy baseline. The evaluation covers 2,214 noisy utterances (about 7.1 hours) built from LibriSpeech, nine noise families and six SNR levels from -5 to 20 dB. At a 16 ms hop, TEN VAD reaches a ROC AUC of 0.917, an average precision of 0.968 and an F1 of 0.913. A sweep over 20, 30, 40 and 50 ms windows shows TEN to be almost insensitive to frame length (AUC 0.916 to 0.919), while RMS stays near 0.75 to 0.77. Silero matches TEN only near its native 32 ms input size and drops to 0.63 at 20 ms. We hypothesize that this is likely caused by our fixed-frame protocol rather than by the model itself. Zero padding, sub-chunk averaging and unreset model state are the likely factors, but we have not ablated them. In a per-condition breakdown of TEN, babble noise at low SNR stays close to chance. We also show that F1 at a fixed 0.5 threshold is a weak metric on this data, because a trivial all-speech detector already scores 0.854. All settings needed to reproduce the results are given in the paper.
Abstract (translated)
URL
https://arxiv.org/abs/2610.14607
PDF
-
SpikCTU: A Compact Spiking Network with Adaptive Recurrent Computation for Speech Command Recognition
2026-10-10 05:32:07Li Jin, Jiang Wenbin, Chen Tao, Wen FeiAbstract
Spiking Neural Networks offer event-driven, energy-efficient computation for speech command recognition, yet most existing methods follow fixed computation paths, allocating redundant computation to easy samples while providing limited insight into how evidence evolves during inference. To address these challenges, we introduce SpikCTU, a compact spiking network with adaptive recurrent computation for speech command recognition. Specifically, a hybrid spiking embedding module preserves continuous input information alongside spike features to construct a fixed temporal memory. A shared Continuous Thinking Unit (CTU) repeatedly reads this memory using keys derived from spikes and continuous values, and updates a binary recurrent state with lightweight stage adapters. Furthermore, combining a fixed base prediction with accumulated corrections at each step enables learned aggregation over all steps or confidence-based early exit using only the predictions available so far. Experiments on three speech command recognition benchmarks show that SpikCTU achieves competitive accuracy with only 0.16--0.17M parameters, and the early exit strategy can substantially further reduce computational cost without compromising accuracy. In addition, trajectory and attention analyses reveal that recurrent steps enable error correction and progressively refine predictions. Code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2610.14594
PDF
-
AphaVoice: Text-to-Speech Model for Aphasia Patient Simulation
2026-10-10 02:22:51Al Varrel Putra Kusuma, Fernando Bello, Joshua BrownAbstract
While language therapy is the primary treatment for post-stroke aphasia, limited clinical resources often restrict patient access to proper care. Addressing this gap requires reliable training simulations for medical professionals. However, current AI efforts focus almost exclusively on speech processing, leaving speech generation largely unexplored. This study introduces AphaVoice, a novel text-to-speech (TTS) framework explicitly engineered to synthesize highly realistic aphasic speech. Evaluated against state-of-the-art TTS systems, AphaVoice demonstrates a superior ability to capture the clinical feature of dysfluency, outperforming existing models in temporal metrics while consistently maintaining speaker identity. By leveraging latent space shifting, the model also supports multi-speaker generation without sacrificing phonetic realism. Ultimately, AphaVoice provides a scalable, high-fidelity tool for medical training and clinical simulation related to aphasia.
Abstract (translated)
URL
https://arxiv.org/abs/2610.14479
PDF
-
RUBATO: Decoupled Clocks Allow Full-Duplex Models to Think as Fast as They Can
2026-10-09 20:21:42Yi-Jen Shih, Puyuan Peng, Haohe Liu, David Harwath, Abdelrahman MohamedAbstract
Recent Speech Full-Duplex (FDX) models are capable of engaging in naturalistic conversation by listening and speaking at the same time. Building on this foundation, recent architectures introduce internal thinking streams to enhance their intelligence. To guarantee these streams process successfully in real time, existing methods enforce a fixed decoding rate, which assumes a hardware profile capable of a specific latency. We argue this static approach is severely suboptimal. Because internal thinking is intrinsically asynchronous, an ideal FDX model should sustain real-time conversational flow while allowing each stream to decode at its own hardware-adaptive rate. Thinking streams must process as fast as the hardware allows to deliver accurate answers faster. To solve this, we propose RUBATO, a novel multistream positional encoding that assigns independent clocks to entirely decouple the streams. This structural isolation empowers the model to zero-shot interpolate across any inference decoding speed. It drastically simplifies dataset construction by requiring training on only two extreme decoding speeds while smoothly adapting to any intermediate rate during deployment. RUBATO demonstrates consistent robustness across five spoken reasoning tasks. On Spoken GSM8K, a baseline model collapses to 1.7% accuracy at interpolated speeds. Conversely, RUBATO-Full sustains 48.7% to 51.1% accuracy across all rates, leveraging faster hardware to achieve a 6.5x shorter time-to-answer with a constant time-to-first-response. This advantage holds on BIG-Bench Audio and amplifies significantly during complex full-duplex interruptions. Specifically, when a user interjects while the model is actively speaking and reasoning, RUBATO-Full achieves 40.0% to 45.1% accuracy, whereas the baseline fails completely.
Abstract (translated)
URL
https://arxiv.org/abs/2610.14299
PDF
-
Speech event detector: Timestamped detection of non-verbal vocalizations and background sounds for rich speech transcription
2026-10-09 18:49:55Ayush Kumar Bar, Promila GhoshAbstract
Automatic speech recognition produces words but drops the non-verbal content of a recording: the breaths, coughs, sighs and laughter of the speaker, and the horns, music or barking around them. We describe Speech Event Detector, a 5.8-million-parameter convolutional-Transformer model that outputs probabilities every 40 ms for 54 event classes (20 speaker vocalizations and 34 background sounds), together with post-processing that converts these probabilities into timestamped events and a procedure for merging the events into a Whisper transcript. The model is trained on material drawn from seven public datasets whose labels come in incompatible forms: single detected regions, free-text tags with timestamps, temporally strong ontology labels, symbols embedded in transcripts, isolated recordings and environmental recordings. We unify them using rule-based label mapping, recover timestamps for transcript-embedded labels through CTC forced alignment, locating 6,360 of 8,620 sounds, and combine about 168 hours of real annotated training recordings with synthetic mixtures and augmentation using real background noise. Per-class thresholds for two operating profiles are tuned on real held-out recordings, and performance is estimated using two-fold cross-validation. On whole held-out recordings from four datasets, considering the 48 classes with at least ten held-out events, the sensitive profile detects an average of 49 percent of a class's events, covering 67 percent of all 9,367 events, at a macro precision of 0.30. The balanced profile achieves a macro recall of 0.37 at a precision of 0.36, with macro event F1 scores of 0.32 and 0.33, respectively. The detector runs about 39 times faster than real time on eight CPU threads.
Abstract (translated)
URL
https://arxiv.org/abs/2610.14232
PDF
-
OpenPA: Forced-alignment-free Multi-granular Pronunciation Assessment with Mixed-transcript Adaptation
2026-10-09 16:35:25Fu-An Chao, Bi-Cheng Yan, Berlin ChenAbstract
Recent advances in automatic pronunciation assessment (APA) have driven a shift from scripted to free-form assessment. However, existing methods often rely on multi-stage pipelines with forced alignment and separate feature extractors, limiting their practicality across response settings. In light of this, we propose OpenPA, a novel APA framework that leverages Whisper's cross-attention mechanism to jointly predict word- and utterance-level pronunciation scores without forced alignment. Through mixed-transcript adaptation and adaptive substitution weighting, OpenPA accommodates heterogeneous transcripts across closed- and open-response settings within a single model. Trained solely on speechocean762, OpenPA achieves the strongest overall performance among evaluated systems under in-domain closed and simulated open-response conditions, while maintaining its performance zero-shot on MultiPA. An additional cross-task evaluation on oral-cancer speech suggests its potential applicability beyond non-native speech.
Abstract (translated)
URL
https://arxiv.org/abs/2610.13826
PDF
-
Error Rate is Not Enough: Costs of Augmentation in Low-Resource ASR
2026-10-09 16:19:43Katsumi Ibaraki, David ChiangAbstract
Data augmentation is standard practice for low-resource automatic speech recognition (ASR), but its benefits are almost always reported as aggregate error rates, leaving open why and when a method helps. On four low-resource languages and English at five data sizes, we cross two augmentations that modify existing audio (speed perturbation and SpecAugment) with two that modify text transcriptions and synthesize audio from it. We analyze errors at the phoneme level and find that they help for complementary reasons. SpecAugment has the most reliable aggregate improvements, but increases tone and palatalization errors, whereas text-based methods introduce consonant insertions; in combination, each reduces the errors the other introduces. This complementarity is predictable. Tone and palatalization are each signaled by a single acoustic property and occur least frequently in training (lowest training exposure), and text-based methods that flatten word frequency raise their frequency by 37--38%, reversing those errors while further improving aggregate error. The improvement from combining the methods exceeds the sum of their individual improvements; the superadditivity vanishes by 100 hours, limiting the complementarity to the low-resource regime. Where specific error types matter, we recommend selecting augmentation by training exposure and acoustic properties, both measurable before any training.
Abstract (translated)
URL
https://arxiv.org/abs/2610.13807
PDF
-
OpenSVBench: Diagnosing Speaker Verification Generalization Across Deployment Scenarios
2026-10-09 15:46:43Qituan Shangguan, Bosen Xu, Shuai WangAbstract
Speaker verification (SV) systems now achieve very low error rates on standard public protocols, but those results do not fully describe deployment generalization. VoxCeleb-style evaluations remain useful anchors for English in-the-wild media speech. Scenario-specific evaluations and challenges are also valuable. However, neither setting directly shows whether the same system remains reliable across diverse deployment scenarios. We introduce OpenSVBench, a public benchmark for cross-scenario SV evaluation. Its fixed public core contains 26 trial sets organized into 11 scenario groups, with 9.15M verification trials, frozen target/non-target keys, and metadata for scenario-wise reporting. Around this core, the online leaderboard acts as a diagnostic evaluation interface rather than only a ranked list. It accepts score packages from new systems, evaluates them with frozen keys and common scoring, and reports trial-set breakdowns, scenario-group metrics, macro EER/minDCF, model-wise profiles, and scenario-wise comparisons. We evaluate 18 public checkpoints spanning x-vector, ECAPA-TDNN, CAMPPlus, ResNet and ERes variants, ReDimNet2, SAM-ResNet, and pretrained Transformer speaker recipes. The results show clear cross-scenario gaps: systems with similar macro EER can have different scenario profiles, strong performance in one scenario does not guarantee strong performance in another, and same-scenario trial sets can still expose source and protocol gaps.
Abstract (translated)
URL
https://arxiv.org/abs/2610.13760
PDF
-
MixedTruth: Component-Level Detection of Synthetic Speech and Synthetic Background in Mixed Audio
2026-10-09 14:35:37Xindong Zheng, Jiashu Zhu, Ke Li, Dan LinAbstract
Modern generative audio can fabricate not only voices but also the acoustic environments that provide evidence of when and where speech occurred. This capability undermines a longstanding forensic assumption that background sound provides reliable evidence of a recording's provenance. Yet existing detectors produce only a single real-or-synthetic verdict, leaving analysts unable to determine whether the voice, the environment, or both were manipulated. In this paper, we propose a component-level forensic detector, namely TruMix, which classifies recordings into eight categories spanning the presence and authenticity of speech and background. TruMix has a unique multi-path architecture that jointly analyzes speech-oriented and background-oriented spectral evidence, contextual information from the mixed waveform, and low-level synthesis artifacts. This design avoids the artifact propagation inherent in conventional separate-then-detect pipelines while enabling generalization across independent speech and environmental-sound generators. We also curated MixedTruth, a 4,917-hour benchmark covering all eight categories under diverse mixing conditions. Extensive experiments show that TruMix achieves 98.51% AUC for component-level attribution, significantly outperforms ten state-of-the-art synthetic-speech detectors in conventional binary classification, and exhibits strong robustness across four attack tiers.
Abstract (translated)
URL
https://arxiv.org/abs/2610.14141
PDF
-
Enhancing Speech Recognition through Text-to-Speech and Voice Conversion Augmentation
2026-10-09 14:33:36Yunus Emre Ozkose, Ali HaznedarogluAbstract
Given the challenges associated with obtaining large volumes of real-world audio, the advancement of Automatic Speech Recognition (ASR) systems increasingly depends on synthetic data. This study focuses on the call center-banking domain, providing a targeted analysis of ASR models in this context. We evaluate real and synthetic datasets, using Word Error Rate (WER) and Character Error Rate (CER) as metrics, and employ a speech quality model to assess the impact on ASR performance. Our research compares Text-to-Speech (TTS) and advanced voice conversion methods, including KNNVC, Seed-VC, and Vec2Wav2, revealing significant improvements in speech quality and ASR accuracy, particularly with Seed-VC. Additionally, our domain-specific experimentation provides insight into the unique challenges and opportunities that arise when applying ASR technologies to industry-relevant settings. This highlights the potential of voice conversion technologies to enhance ASR systems, guiding future research in diverse linguistic scenarios, and paving the way for the broader application of ASR innovations across various fields.
Abstract (translated)
URL
https://arxiv.org/abs/2610.13662
PDF
-
Vocal Intensity Conversion with Continuous SPL Control via Adversarial Training
2026-10-09 13:55:07Quentin Le Tellier, Albert Rilliard, Olivier Perrotin, Marc EvrardAbstract
Vocal intensity conversion renders a recorded utterance at a different vocal intensity, rather than simply scaling its amplitude, while preserving its content and speaker identity. Existing systems control intensity through a reference utterance or a few discrete categories. Instead, we control the conversion with any target sound pressure level (SPL) on a continuous dB scale. A lightweight converter, trained as a conditional cycle-consistent GAN without parallel data, transforms the features of a frozen speech encoder, from which a pretrained vocoder resynthesizes the speech. We compare three representations, log-mel spectrograms, WavLM, and SpeechTokenizer, and introduce an objective protocol that measures the achieved intensity change against the target one on a calibrated corpus. With WavLM, the converter achieves a measured 83% of the target change in intensity and also shifts the fundamental frequency. A perceptual evaluation confirms that perceived vocal intensity increases with the target level on amplitude-normalized stimuli.
Abstract (translated)
URL
https://arxiv.org/abs/2610.13608
PDF
-
AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs
2026-10-09 13:49:22Shreya Sajid, Anna Vettoruzzo, Joaquin VanschorenAbstract
Large Audio Language Models (LALMs) achieve strong performance on speech-text tasks, but standard audio encoders produce hundreds of tokens for even short audio clips. Because self-attention scales quadratically in sequence length, this inflates inference-time compute substantially. Current training-free token compression methods prune or merge tokens based solely on coarse attention or feature similarity. However, treating audio tokens as generic, isotropic features smooths over spectral transitions, destroying fine-grained phonetic transitions essential for decoding speech. In contrast, speech signals exhibit highly structured temporal patterns where information density peaks at acoustic transitions. We propose AudioTETRIS, a training-free, two-stage framework that uses the audio encoder's representation geometry as an acoustic prior for token importance before applying attention-based compression. Instead of discarding encoder structure after the encoding stage, AudioTETRIS propagates this prior directly into the LLM decoder to compute a per-token fused score. This score drives progressive top-$k$ token selection across decoder layers without requiring additional training or auxiliary models. Evaluated on Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3, AudioTETRIS retains near-uncompressed performance on several speech tasks using only 25% of audio tokens, matches or surpasses full-token accuracy at moderate compression rates, and consistently outperforms existing training-free baselines.
Abstract (translated)
URL
https://arxiv.org/abs/2610.13600
PDF
