Language_Model
-
Memory-Aware Reward Modeling: Learning Human Preferences for LLM Memory Use
2026-10-22 13:43:33Yijiang River Dong, Zexue He, Cheng Yu, Tiancheng Hu, Michael J Ryan, Jan Batzner, Zheng Hui, Shaolong Wu, Orestis Papakyriakopoulos, Alex Pentland, Nigel Collier, Jiaxin PeiAbstract
Memory enables language model assistants to maintain continuity across sessions and generate personalized responses, but knowing more about a user does not necessarily produce responses they prefer. Existing evaluations assess memory capabilities without directly establishing whether memory use benefits the user. To study this gap, we collect approximately 1,000 pairwise judgments from 87 participants using their own ChatGPT queries and personal memories. We find that memory-conditioned responses are not consistently preferred over responses generated without memory and in some cases, users strongly prefer the latter. Automatic evaluators also show limited agreement with users and favor memory-conditioned responses substantially more often than users do. Motivated by these gaps, we introduce a memory-aware reward model that learns from user judgments, achieving 72.8\% preference prediction accuracy and outperforming the existing baselines. Our findings motivate evaluation and optimization objectives that align memory use with the preferences of the people whose information is being used.
Abstract (translated)
URL
https://arxiv.org/abs/2610.31272
PDF
-
Is the Judge Its Twin? Post-training makes a model's writing more predictable but barely moves its taste, as a judge, toward predictable writing
Abstract
Language models now routinely grade other language models. If post-training makes a model's writing more predictable, it may also teach the same model, as a judge, to reward predictable writing. We test this directly. We follow two open 7-billion-parameter model families, OLMo-2 and Zephyr, from their base models through supervised fine-tuning (SFT) and direct preference optimization (DPO), and measure every stage as a writer of short stories and as a judge of story pairs. As writers, the models drift as feared. Each family's trained model finds its stages' stories more familiar in the order base, SFT, DPO, in all ten prompts, and a model from the other family finds the trained stories 7.0 to 9.2 percent (OLMo-2) and 24 to 25 percent (Zephyr) less surprising per token. As judges, they barely move toward predictable writing. Asked which story is better, a question every judge passes a competence check on, no trained judge's preference for the more predictable story grows by as much as one point of pick probability at its estimate. A one-sided 95% upper bound on that growth is about 5 points on the widest pairs. Training instead makes judges more swayed by story length and by display order, and it breaks "more creative" as a question. Trained judges no longer reliably prefer a story to its own words in scrambled order. For these two families, the practical hazard in a small trained judge of creative writing lies in these measurable habits.
Abstract (translated)
URL
https://arxiv.org/abs/2609.32196
PDF
-
A Systematic Comparison of Search Space Complexity and Efficiency for Time-Synchronous End-to-End ASR Models
2026-10-22 13:35:50Simon Berger, Larissa Kleppel, Ralf Schl\"uterAbstract
Search is a central component of Automatic Speech Recognition (ASR) systems and has been deeply investigated for classical ASR modeling based on Hidden Markov Models (HMMs). However, in work on current sequence-to-sequence ASR architectures it is often simplified to a small fixed-size beam search without external Language Model (LM). In this work, we present a comparative analysis of search space complexity and efficiency for Connectionist Temporal Classification (CTC) and neural transducer models under different pruning strategies and LM configurations. We show that these end-to-end (E2E) models work with a small search space most of the time but exhibit large temporary peaks in hypothesis count during periods of uncertainty. Score-based pruning dynamically adapts the search to model uncertainty, leading to a significantly lower Real-Time Factor (RTF). Our results highlight the limitations of search with low fixed-size beam pruning and provide practical guidelines for efficient search in E2E ASR.
Abstract (translated)
URL
https://arxiv.org/abs/2610.31259
PDF
-
Query-Conditioned Liquid Temporal Grounding for Long-Form Audio
2026-10-22 13:32:12Yuelan Cheng, Yonggang Zhu, Yongqiang Chen, Peiwei Chang, Yingyi Chen, Wenwu WangAbstract
Long-form temporal audio grounding (TAG) is challenging because target events often occupy only a small portion of a recording, while queries may impose multiple semantic constraints over a large temporal search space. We propose a query-conditioned liquid temporal grounding framework that injects query semantics through FiLM before a closed-form continuous-time (CfC) network models long-range temporal dependencies. Multiple query-relevant candidates are generated, from which a frozen large audio-language model selects the most compatible one for lightweight local boundary refinement. The pretrained backbone remains frozen, with only lightweight grounding modules optimized. Experiments on the English TAG subset of LAT-Bench show consistent improvements across MOSS-Audio and Qwen3-Omni, while ablations confirm the complementary roles of FiLM and CfC.
Abstract (translated)
URL
https://arxiv.org/abs/2610.31252
PDF
-
A Visual Grounding Benchmark for Retinal OCT with Multimodal Large Language Models
2026-10-22 12:54:19Ronald Fecso, Jos\'e Morano, Botond Fazekas, Paul Widmann-Sedlnitzky, Dorota Himel, Laetita Hinterhuber, Virginia Mares, Andreas Pollreisz, Hrvoje Bogunovi\'cAbstract
Multimodal large language models (MLLMs) can answer clinical questions about medical images, but trustworthy use requires visual grounding: models should localize the pathology they report rather than rely on spurious shortcuts. Whether MLLMs can visually ground clinical findings in retinal optical coherence tomography (OCT), the most widely used imaging modality in ophthalmology, remains largely unexplored. To address this gap, we introduce RetinaGroundBench, the first benchmark for biomarker visual grounding in retinal OCT. It comprises 815 B-scans exhaustively annotated by ophthalmologists with 4,537 bounding boxes covering 13 biomarkers across 8 diseases. Because MLLMs often fragment a single lesion into multiple partial predictions, we propose AggIoU, a metric that evaluates whether the union of overlapping predicted boxes covers the annotated region. Across 27 open-source MLLMs (0.8-38B parameters), we find that visual grounding remains largely unsolved: the best general-purpose model achieves a macro F1 of just 23.7%, with consistently low recall, despite reaching up to 73% balanced accuracy on biomarker presence questions. To narrow this gap, we developed RetGrounder, an MLLM fine-tuned using an OCT-specific strategy integrating grounding and textual information. Despite its relatively small size (8B), RetGrounder improves grounding F1 to 33.9%, a 43% relative improvement. Our benchmark provides a reference point for measuring progress in the field of OCT diagnosis with MLLMs.
Abstract (translated)
URL
https://arxiv.org/abs/2610.31205
PDF
-
Predicting Music Valence and Arousal via Textual and Multimodal In-Context Learning in Large Language Models
2026-10-22 12:49:14Chi-En Dai, Chih-Pin Tan, Yi-Hsuan YangAbstract
While arousal can be reliably predicted from musical audio, emotional valence remains difficult to model due to its subjective and context-dependent nature. In this work, we investigate the potential of in-context learning (ICL) with large language models to advance continuous valence and arousal prediction. Rather than predicting emotion scores directly, our framework leverages contextual demonstrations from an auxiliary dataset to refine the initial predictions produced by a small, task-specific base model. We systematically evaluate a text-based ICL variant using lyrics and generated audio captions, alongside a multimodal variant that incorporates raw audio signals. Using the entire MER60 dataset strictly as a held-out test set, our experiments demonstrate substantial performance gains, particularly for valence. Text-based ICL improves the valence $R^2$ from 0.110 to 0.427 with 80 demonstration shots, while the multimodal variant achieves 0.346 with only 10 shots. These findings suggest that training-free ICL offers an effective inference-time strategy for current language models to predict continuous music emotion.
Abstract (translated)
URL
https://arxiv.org/abs/2610.31202
PDF
-
Investigating Multi-Cultural Value Selection Biases in LLMs
2026-10-22 12:46:22Siddhesh Pawar, Arnav Arora, Dustin Wright, Dirk Hovy, Isabelle AugensteinAbstract
Large language models (LLMs) serve users whose cultural identities are multi-layered, yet cultural evaluations commonly reduce identity to one label. In this work, we study LLMs under simultaneous identity cues, with a country persona in the system prompt and a second identity signaled in the user turn, either by naming a country (explicit cue) or through the query language (implicit cue). We test World Values Survey (WVS) multiple-choice probing and open-ended sociopolitical generation. In multiple-choice probing, language cues shift answers about three times more than explicit cues but away from the associated country's distribution, and explicit cues do not consistently move answers toward the named country. Across 1.6 million open-ended responses, naming a country closes 30-44% of the gap to the model's own responses for that country, and explicit cues produce more differentiated value changes than language cues. Our results suggest that country and language cues should be evaluated as distinct signals rather than interchangeable proxies for culture.
Abstract (translated)
URL
https://arxiv.org/abs/2610.31199
PDF
-
Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?
2026-10-22 12:41:36Aneerban Saha, Vijay Kumar, Sameer Madhav D., Priyam Srivastava, Dhruv BansalAbstract
Some questions about an image need commonsense or factual knowledge that its pixels do not contain. Large Language Models (LLMs) can supply this knowledge to Visual Question Answering (VQA) systems, but the depth at which their guidance should enter a vision-language model remains unclear. We prompt the 70-billion-parameter Llama~3.3 model to describe, for each question, the visual concepts to focus on, the distractors to ignore and the external facts needed. We inject this guidance into a frozen Vision Transformer (ViT) paired with an instruction-tuned Text-to-Text Transfer Transformer (T5) in two ways: decision-level late fusion and attention-matrix injection. On the Outside Knowledge VQA (OK-VQA) benchmark, and contrary to the common view that deeper is better, the shallow route wins: late fusion reaches 39.4% exact match, +10.0 points above an unguided baseline, while attention injection reaches only 31.8% under data-scarce fine-tuning. An LLM paraphrasing pipeline also enlarges the training set more than tenfold. The results suggest that integration depth should be matched to the data regime.
Abstract (translated)
URL
https://arxiv.org/abs/2610.17079
PDF
-
Prompt Sensitivity in Tool-Augmented Model Comparisons
2026-10-22 12:07:01Mateusz Modrzejewski, Tomasz RadzikowskiAbstract
Language models are compared on question-answering benchmarks that pose each question in one wording, and a reworded question can change which of two models scores higher. We ask whether giving the models tools makes such comparisons more stable. An accuracy gain cannot answer this, because tools change several steps between the question and the scored answer at once. We propose a matched protocol that separates those steps: the answer readout, reasoning, the decision to call, the tool's answer, and the report supplied without a call. We apply it to six open models in five domains under five wordings. We find that tools raise accuracy where a program answers the question and widen the wording sensitivity of many pairwise comparisons. The widening follows how often a model calls, and call frequency follows the wording for some models and not others. It also depends on the interface and the instructions: a native tool-calling interface and a second tool description leave no detected widening in molecules, and sensitivity persists where tools return intermediate results. These findings show that deterministic reports do not guarantee stable comparisons. We recommend reporting call frequency and comparison stability under several wordings beside accuracy.
Abstract (translated)
URL
https://arxiv.org/abs/2610.31148
PDF
-
RankAlign: Vision-Language Representation Alignment via Rank Transformation
2026-10-22 12:03:19Enzhe Zhao, Marco Fiorucci, Lamberto BallanAbstract
While the Platonic Representation Hypothesis posits that vision and language models converge toward compatible latent geometries, recovering class-level correspondences between unpaired modalities without cross-modal supervision remains challenging. Existing relational matching methods compare within-modality similarity kernels directly, although the same semantic structure may be represented through different numerical parameterizations across modalities. We propose RankAlign, a rank-based normalization strategy for class-level unpaired vision-language matching. RankAlign transforms each within-modality similarity kernel into an ordinal relational representation before permutation matching, while leaving the correspondence space and matching solver unchanged. The resulting representation is invariant to strictly increasing reparameterizations of similarity values while preserving the ordinal structure used for matching. We provide exact-recovery and stability guarantees that characterize correspondence recovery under both exact and partial ordinal agreement. Experiments across diverse frozen vision and language encoders and multiple datasets demonstrate improved and more stable class-level correspondence recovery, including under label corruption and domain shift.
Abstract (translated)
URL
https://arxiv.org/abs/2610.31145
PDF
-
Delta N-gram: Forward-Only Continual Writing into a Language Model's Native N-gram Memory
Abstract
Our previous analysis found that a model's native hashed N-gram table is where continual updates disturb the base least, but its writes needed backpropagation and changed the pretrained table. We present Delta N-gram, a forward-only write path into that table. The pretrained table T^0 is never modified. Evidence is written into a sparse patch \Delta on the rows it addresses: a 4.1M-parameter writer emits a target and a gate per (position, hash head) event, and a block-averaged delta rule keeps each row a convex combination of its old value and the gated targets. The writer is trained offline on raw text with the pretraining objective; at use time writing takes forward passes only, and clearing \Delta restores the original model. A 0.6B prototype trained to answer questions keeps 16/16 facts after a second write; online gradient writes keep 2--4/16. We study two writers: a plain writer that encodes evidence with the T^0-only model, and a delta writer that reads its current memory while encoding. On Qwen3.8-Flash-Next, the plain writer's memory is document-specific: on 64 held-out news events it lowers continuation NLL by 0.068 nats/token against empty memory and by 0.024 against another document's memory, and 0.063 and 0.013 after 16 intervening articles. The delta writer's memory is more document-specific and keeps more of it under distraction (70% vs 54%). Over 35,000 writes on our previous seven-task chain both writers stay stable and the delta writer drifts less (layer-2 math drift 176% vs 188%); later accuracy changes by -0.26/-0.52pp (-7.29pp for gradient updates) and small-k_i drift is 17.9/17.5% (39.2%). Acquisition is negative, and general-input drift reaches 32.8% (20.75%): format rows read by every example grow to 4--12 times their pretrained norm. Zeroing eight such rows removes over 98% of layer-2 drift on math inputs, together with their NLL gain.
Abstract (translated)
URL
https://arxiv.org/abs/2610.31125
PDF
-
LAMB: A Modern Contextual Language Model for Latin
Abstract
This article introduces Latin Modern BERT (LAMB), an encoder for the Latin language based on the ModernBERT architecture, pretrained in two phases on more than 74B Latin tokens using a novel morphologically informed masking regimen. We also present two corpora for the Latin language. Acervus contains 24B BERT WordPiece tokens, including over 760K samples from heavily damaged OCR transcriptions, repaired by a ByT5 model trained for Latin text repair. The second dataset, Corpus Trinum, contains a mixture of over 100M strong and weak Latin language semantic triplets. Corpus Trinum is the first Latin semantic triplet dataset, opening the door for Latin language models to be adapted to general sentence semantic similarity tasks. We construct an experimental grid using five tasks and seventeen subtasks, applying a variety of finetuning configurations and subword aggregation techniques for 15 pretrained encoders, 6 LAMB training checkpoints, and an ablation model designed to quantify the impact of different masking regimens. Using this benchmark, we rank Latin encoders and find that LAMB outperforms all other evaluated models on 16 of 17 subtasks, coming within 0.8 percentage points of state of the art on the seventeenth. We provide additional analyses relating Latin encoder performance to tokenizer behavior, syntactic structure, finetuning configuration, pretraining state, subword aggregation, and variation in source texts and domains. In these, we identify unique acquisition dynamics for Latin open class words during terminal decay phases, and establish that for some task categories, finetuning configuration selection can have a greater effect on downstream scores than additional pretraining tokens.
Abstract (translated)
URL
https://arxiv.org/abs/2610.31723
PDF
-
MQAScore: Fine-Grained Text-Music Alignment Evaluation via Large Audio Language Model
2026-10-22 10:56:25Wei-Jaw Lee, Chun-Ping Wang, Xuanjun Chen, Yi-Hsuan YangAbstract
Text-to-music alignment is commonly evaluated by the global similarity between text and audio embeddings, which collapses all described attributes into a single score and obscures which ones are missing. We instead introduce MQAScore, which verifies each atomic musical concept along five clip-level dimensions (genre, instrument, mood/theme, vocal, and usage) by the probability that a large audio language model (LALM) answers affirmatively to a binary question. To assess evaluator reliability, we construct verification labels from presence/absence annotations and LLM-guided concept substitutions, and propose the Sensitivity-Invariance Ratio, which measures how selectively an evaluator responds to meaning-changing substitutions. Experiments show that concept-level scoring improves both question-answering and similarity-based evaluators, and MQAScore can be a more reliable and interpretable fine-grained complement to global similarity for evaluating text-to-music alignment.
Abstract (translated)
URL
https://arxiv.org/abs/2610.31083
PDF
-
Autoregressive Multimodal Anomaly Detection with Large Language Models for Microservice Systems
2026-10-22 09:59:34Yixiang Tang, Fagui Liu, Guoxiang Zhong, Xi Yao, C. L. Philip ChenAbstract
Microservice systems have become a dominant architecture for large-scale cloud applications, enabling rapid deployment and elastic scaling. However, ensuring the reliability and availability of microservice systems critically relies on effective anomaly detection, as failures and performance anomalies can quickly propagate across services and severely impact end-to-end system behavior. Existing multimodal anomaly detection approaches for microservice systems often struggle with two fundamental limitations: (i) the neglect of modality-specific characteristics in favor of cross-modality fusion, and (ii) the "identity-shortcut" problem that affects the feature reconstruction process. To address these issues, we propose LLMMAD, an autoregressive multimodal anomaly detection framework for microservice systems based on Large Language Models (LLMs). Initially, to reconcile cross-modality fusion with modality-specific preservation, LLMMAD embeds and recombines multimodal data into a unified sequence for LLM processing to model inter-modality relationships, while employing modality-wise reconstruction heads to retain modality-specific information. Subsequently, to eliminate the identity-shortcut, we implement a prediction-driven autoregressive objective, under which the model predicts future states rather than copies current inputs. Finally, we conduct extensive experiments on three real-world datasets. The results demonstrate that LLMMAD consistently outperforms state-of-the-art methods, achieving F1-score improvements of 3.1%, 1.4%, and 0.9%, respectively, thereby providing a more reliable solution for microservice monitoring.
Abstract (translated)
URL
https://arxiv.org/abs/2610.31722
PDF
-
Padding to k: A Minimum Count in the Schema Makes Open-Weight Models Pad Their Extraction Lists
Abstract
Extraction pipelines ask language models for JSON that follows a schema, and the schema can set a minimum number of list entries. We ask what models do when the document holds fewer items than that minimum. Under a preregistered plan, we test eight open-weight models on 558 real documents from nine annotated datasets, with six more models on hosted services as a reference. Three findings stand out. First, models treat the count as a target: a minimum shown in the schema, but not enforced, raises the share of lists longer than the true count from 34.5% to 55.4%. Second, the padding is hard to catch, because most added entries are real text taken from the document; about half of them pass a check that each entry occurs in the source. Third, common fixes help only partly: a line saying fewer items are fine removes the effect on average but misses more true items, a count in prose ("up to k") also raises over-counting, an "N/A" option under an enforced minimum still leaves half the lists padded, and re-asking with the validation error makes padding far more common. We release the code, answer lists and run records at this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2610.30997
PDF
-
A Reality Check of Inference-Time Context Augmentation Methods
2026-10-22 09:31:18Haeun Yu, Geert Heyman, Pepa Atanasova, Isabelle AugensteinAbstract
Large language models (LLMs) often fail to utilise the context given in the input, resulting in hallucinations or a fallback to parametric knowledge. Context augmentation methods aim to improve model performance by modifying the input, from inserting filler tokens that add computational time to adding relevant content such as elaboration of a context during inference. However, these methods have been evaluated in isolation and only by their performance gains, leaving it unclear how they compare and whether they reliably improve performance. In this paper, we present an evaluation framework spanning five methods that differ in how informative the inserted content is and apply them at the sentence level across seven models spanning multiple sizes on three datasets. We find that accuracy gains are not always genuine improvements but undirected perturbations: the methods correct some predictions while breaking others, particularly when the inserted text allows only for more computational time. Our analysis reveals that methods that bring no additional content primarily change model predictions near the decision boundary. Our findings highlight the need to evaluate the reliability of the accuracy gains before adopting them in practice.
Abstract (translated)
URL
https://arxiv.org/abs/2610.30983
PDF
-
Diagnostic Loss: Asymmetric Supervision for Radiology Report Generation
2026-10-22 09:26:17Ignacy Kolton, Wenxuan Li, Zongwei Zhou, Jaros{\l}aw B. \'Cwik{\l}a, Arkadiusz Sitek, Pedro R. A. S. Bassi, Szymon P{\l}otkaAbstract
Automated radiology report generation requires models to do more than produce fluent clinical narratives: they must explicitly state supported diagnoses while avoiding unsupported ones. However, existing medical vision-language models (MedVLMs) are typically trained with token-level maximum likelihood objectives, which prioritize text agreement over diagnostically critical terms. As a result, models can miss key diagnoses and introduce wrong ones. We identify this as an objective misalignment between conventional language modeling and the asymmetric diagnostic requirements of radiology reporting. To encode these requirements, we introduce Diagnostic Loss (DLoss), a differentiable, architecture-agnostic objective that operates directly on decoder token probabilities. DLoss separately aggregates probability mass over vocabularies associated with diagnoses supported by the reference report and with unsupported alternatives. Using asymmetric pooling across token positions in the report, it encourages supported diagnoses to receive sufficient probability in at least one position while suppressing spurious probability spikes for unsupported diagnoses across all positions during training. The loss can be applied to existing 3D MedVLMs without modifying their vision backbones, language decoders, or inference procedures, and introduces no additional inference-time computation. We train and evaluate DLoss on 50,000 abdominal CT scans across several 3D MedVLM architectures on both in-distribution and out-of-distribution report-generation benchmarks. Across these settings, DLoss improves diagnostic faithfulness while maintaining competitive performance on conventional language-generation metrics, showing the value of asymmetric sequence-level supervision for radiology report generation. The code for DLoss is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2610.30980
PDF
-
LAST-V2X: Language-Driven Adaptive Scenario Synthesis and Testing for Communication-Induced Failures in V2X Safety Applications
2026-10-22 09:24:29Keshu Wu, Hao Zhang, Rui Gan, Pei Li, Jiwan Jiang, Yang ZhouAbstract
Vehicle-to-everything (V2X) communication enables connected vehicles to make safety-critical decisions, but delayed, missing, or incorrect information can compromise driving safety. Systematically testing such failures is challenging because traffic interactions, communication conditions, and application responses must be jointly configured to produce meaningful scenarios. This paper presents LAST-V2X, a language-driven, adaptive framework for automated scenario synthesis and safety testing of V2X applications. LAST-V2X leverages large language models (LLMs) to translate natural-language test descriptions into executable scenarios grounded in roadway geometry, V2X communication semantics, and vehicle behavior. The framework integrates physical traffic simulation with spatially aware communication modeling and message-level fault injection to evaluate how information degradation affects vehicle decisions. To establish whether an observed safety failure is caused by communication rather than the underlying traffic conditions, LAST-V2X employs matched executions under degraded and ground-truth information. Furthermore, an execution-guided refinement mechanism combines parameter search and LLM self-reflection to identify ineffective test configurations, improve scenario effectiveness, and produce minimal reproducible failure cases. Experiments across diverse CARLA environments and V2X safety applications demonstrate the framework's ability to generate reliable scenarios, distinguish communication-induced failures from unrelated safety violations, and substantially improve failure discovery through adaptive refinement. These results establish LAST-V2X as a systematic approach to language-driven, causally verified, and adaptive safety testing for connected vehicle systems.
Abstract (translated)
URL
https://arxiv.org/abs/2610.30979
PDF
-
Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters
Abstract
Many language-model pipelines end in a closed-set decision, such as selecting a tool. Prompting a large language model for it is slow and order-sensitive. We ask how small an encoder for such decisions can be and whether it should read the options jointly or separately. In a three-seed study at 32 million parameters, per-option scoring beats joint scoring on a held-out sample of a 37-benchmark decision suite by 5.8 points of chance-corrected skill at matched optimizer steps and by 2.8 at approximately matched training compute; two external benchmarks agree, though not significantly on their unseen tasks. Crossing reader with initialisation shows that the advantage requires a reranker initialisation, whose query-candidate pair input matches per-option reading; from the raw encoder the two readers are indistinguishable. Per-option scoring needs 5 to 13 times more encoder passes: at most twice the median latency, but more than four times the mean. Of six released joint encoders (17 million to 1 billion parameters), the 68- and 150-million-parameter models exceed a 435-million-parameter DeBERTa-v3 zero-shot classifier by 4.9 and 5.9 accuracy points on 209 overlap-audited System1-Arena tasks and are at least on par with it on the 145 tasks unseen in training. The smallest answers a median request in 14 milliseconds on 16 processor threads, against 728 for the classifier. Reordering options changes 12 to 23% of the joint models' answers (per-option scoring: none). Up to 68 million parameters, they lead their size range on a public leaderboard's private, mainly in-distribution tests.
Abstract (translated)
URL
https://arxiv.org/abs/2610.30970
PDF
-
Once a Tree, Always a Tree? Cross-lingual Transfer of Hierarchical Generalization in Language Models
2026-10-22 09:07:44Shin Yamaguchi, Masaya Taniguchi, Tatsuki Kuribayashi, Taro Watanabe, Yoshinobu KanoAbstract
Language models (LMs) often exhibit successful cross-lingual transfer through continued (pre)training, still their underlying process remains unclear. One possibility is that LMs can leverage the evidence toward language-universal principles, namely, hierarchical structure, from a source language (L1) to a new language (L2), while many other non-linguistic explanations, e.g., improved initialization of position embeddings, can be potentially possible. In this study, to quantify the cross-linguistic transferability of linguistic evidence in cross-lingual LM training, we introduce a new experimental design, namely, a cross-lingual poverty-of-stimulus (CPOS) setting that ablates how LMs' hierarchical generalization in L2 can be led by structural evidence provided in L1. We contrast counterfactual L1 and L2 learning scenarios where the L2 corpus is ambiguous between linear and hierarchical grammar rules, while the amount of positive evidence for the hierarchical rule is controlled on the L1 side. Our experimental results show that LMs can partially, yet not perfectly, transfer L1 structural evidence to L2, yielding only a small but detectable hierarchical bias, while the strength of transfer depends on language pairs. We further ablate the conditions for successful transfer, motivated by recent reports on the substantial impact of vocabulary overlap. We indeed find that vocabulary overlap is crucial even for the cross-lingual transfer of hierarchical syntactic generalization. This suggests that this also explains the language-pair-dependent tendencies in the main experiments and suggests that the transfer of linguistic evidence in continued new-language learning is only conditionally achievable.
Abstract (translated)
URL
https://arxiv.org/abs/2610.30960
PDF
