The Latest Papers about AI
-
The Punjabi Geet Toxicity Dataset : Fine-Grained Multimodal Annotations for Toxic Content in Punjabi Songs
Abstract
Resources for toxicity detection in Punjabi remain limited, particularly for musical content that combines linguistic and multimodal information. We present the Punjabi Geet Toxicity Dataset (PGT-FG), a segment-level resource developed to support research on toxicity detection in Punjabi musical content. The dataset was constructed from publicly available Punjabi songs collected from YouTube. An initial collection of 620 songs resulted in 465 successfully downloaded songs and 3,402 preliminary segments from 450 songs. After quality control and annotation, the final dataset contains 2,396 unique segments from 392 songs. Four native Punjabi annotators produced 2,870 annotation records, including a shared subset of 158 segments annotated by all four. Each annotation includes binary toxicity, eight fine-grained toxicity categories, severity, sentiment, mode, and an uncertainty flag. Inter-annotator agreement was measured with Fleiss' kappa on the shared subset, and majority-based resolution produced the final segment-level labels. Of the 2,396 segments, 705 (29.42%) were labelled toxic and 1,691 (70.58%) non-toxic. Violence was the most frequent category, occurring in 539 toxic segments (76.45%), followed by gang references (227, 32.20%) and drug references (133, 18.87%). The overall toxicity label achieved substantial agreement (kappa = 0.6521). PGT-FG provides a fine-grained resource for Punjabi toxicity detection and multimodal content analysis. The resolved dataset, with one final record per segment, is publicly available on Zenodo (this https URL). Original lyrics, lyric excerpts, and copyrighted audio or video content are not redistributed.
Abstract (translated)
URL
https://arxiv.org/abs/2610.32513
PDF
-
Fast and Useful: Adapting Distilled Generators for Visual Learning
Abstract
Distilled text-to-image models generate images quickly, but classifiers trained on those images can perform substantially worse than classifiers trained on images from their slower base models. We adapt released distilled generators to produce better training data while retaining fast generation. We fit a low-rank update to the student's first prediction so that it reaches an intermediate state produced by its unguided base, then disable the update and let the unchanged student finish generation. The base supplies training targets and is absent at inference. Guided targets and shorter jumps give classifiers lower accuracy in our ablations, showing why the state used for supervision matters. Across three generator families, the adapted images train better classifiers than images from the released students. With class-name templates at full ImageNet-1k scale, ResNet-50 top-1 rises from 8.4% to 34.2% for FLUX.2-klein and from 12.7% to 32.1% for SD-Turbo, using two and three denoiser evaluations per image, respectively. Fixed adapters also improve results on additional classes, fine-grained datasets, and other learner architectures and training recipes. Template-based adaptation remains useful against richer-prompt alternatives, while the benefit of combining adaptation with LLM generation prompts depends on the generator. Adapting a single generation step makes fast generators substantially more useful for training visual models.
Abstract (translated)
URL
https://arxiv.org/abs/2610.32512
PDF
-
Augmentation Strategy for DINO in Glaucoma Classification
2026-10-23 11:33:28Abstract
Self-supervised learning (SSL) offers a promising approach to medical image representation learning, yet augmentation strategies of SSL frameworks validated on natural images do not reliably transfer to specialized medical imaging domains. We analyze DINO, a knowledge-distillation based SSL framework, and empirically demonstrate that its default augmentation strategy produces unreliable representations for retinal fundus images, degrading downstream glaucoma classification performance. Guided by the principle that effective cropped views should share minimal redundant information while retaining clinically relevant signal, we redesign DINO's augmentation strategy. Our approach biases the learned representations toward diagnostically critical features rather than low-level image statistics. We evaluate our method for glaucoma classification by training a linear classifier on the frozen representations, achieving strong generalization across seven benchmark datasets. URL for code and weights: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2610.32511
PDF
-
Distortion-Aware Spatiotemporal Learning for Real-Time Wide-Angle Video Rectification
Abstract
While wide-angle lenses provide an expanded field of view (FoV), they exhibit severe nonlinear distortion near image boundaries, unnaturally stretching people in these regions. Existing methods mostly focus on facial regions, require high computational costs, and lack temporal modeling, resulting in incomplete full-body rectification, impractical real-time deployment, and noticeable inter-frame shakes. To tackle these issues, we propose \textbf{FastVC}, an efficient full-body video rectification framework driven by distortion-aware spatiotemporal learning. Grounded in lens physics, FastVC introduces a distortion-aware thin-plate spline (DA-TPS) module that nonlinearly remaps control points into a peripheral-dense layout, enabling more accurate geometric fitting without additional computational cost. We further develop a dual-level spatiotemporal learning scheme for causal and stable video rectification. At the model level, neighborhood spatiotemporal attention (NST) aggregates local context and selectively fuses historical features with low overhead. At the optimization level, sparse spatiotemporal supervision applies spatial alignment only to anchor frames, decoupling frame-wise accuracy from temporal stability. Besides, we contribute the first benchmark dataset for wide-angle full-body video correction. Extensive experiments show that FastVC achieves state-of-the-art rectification fidelity and visual stability at 57.0 FPS.
Abstract (translated)
URL
https://arxiv.org/abs/2610.32500
PDF
-
MusicTwins: Training Audio-Language Models to Listen like Their Persona
Abstract
People differ in how they perceive music, what associations they make with it, what they like about it, and how they respond to it. Computational models intended to simulate people in environments that include music should account for these differences. Although LLMs have been shown to reproduce aspects of human behavior, whether audio-language models (ALMs) can reproduce a specific person's responses to textual and musical stimuli remains underexplored. We introduce MusicTwins, a framework that combines persona summaries, personal memory, and post-training to generate person-specific responses. We construct persona representations from human writings, capturing background, views, personality, writing style, and a musical profile, and post-train Audio Flamingo Next through supervised fine-tuning and reinforcement learning. Evaluation on ten held-out personas shows notable improvements for Audio Flamingo Next in persona fidelity, including personality, writing style, and responses to music, supporting MusicTwins as a foundation for training audio-language models to listen like their persona.
Abstract (translated)
URL
https://arxiv.org/abs/2610.32495
PDF
-
K-CLUE: A Segmentation-to-Clustering Framework for Ki67 Cell Typing
Abstract
Ki67 is a key marker of tumor proliferation in breast cancer, but its quantification from immunohistochemistry images normally depends on supervised classifiers trained per dataset. We examine the role of frozen pretrained visual representations in annotation-free Ki67 cell typing. To this end, we use K-CLUE, a framework that segments individual nuclei, represents each nucleus with a pretrained encoder, reduces the resulting embeddings with principal component analysis, and separates Ki67-positive from Ki67-negative nuclei using k-means clustering, without task-specific training or manual annotations. Clustering the raw components is highly sensitive to the choice of encoder: across seven encoders, the Ki67-positive F1-score on matched MIST nuclei varies by 44 points, although Ki67 status remains linearly decodable from every representation, with supervised probe AUCs above 0.98. Analysis of the embedding spaces shows that clustering succeeds when DAB-related variation occupies the leading directions of variance. Whitening the retained principal components removes this dependence, reducing the spread across encoders to 1.00 F1-score point. Handcrafted color representations are also competitive, confirming that Ki67 status is strongly reflected in chromatic information, but they require an explicit choice of color space. In end-to-end evaluation, K-CLUE reaches F1-scores of 85.00% and 81.91% for Ki67-positive and Ki67-negative nuclei on MIST, and 79.59% and 81.16% on BCData, outperforming DeepLIIF and IHCScoreGAN on both datasets. These results show that pretrained representations can support annotation-free Ki67 typing when their variance structure is accounted for.
Abstract (translated)
URL
https://arxiv.org/abs/2610.32493
PDF
-
Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping
2026-10-23 11:07:45Abstract
Large language models serve applications with diverse inference budgets, but training and maintaining separate models for each budget incurs substantial costs. Recent Looped Transformers enable flexible inference computation through parameter reuse, but maintaining strong performance across different inference budgets remains an open challenge. We introduce Accordion, a mixture-of-experts-based model that supports five inference budgets through fixed-prefix layer looping. Each budget follows a pre-defined execution path, while the minimum budget retains the unlooped model. Joint pretraining samples one budget per micro-batch and optimizes its next-token prediction loss without additional cross-budget consistency objectives. We further preserve these inference budgets through supervised fine-tuning and reinforcement learning, where parameter-group merging and budget-matched multi-teacher on-policy distillation integrate three domain experts into one model. After 3.6T tokens of pretraining, increasing inference depth improves Accordion-6B-A0.5B's five-category macro-average across 37 tasks by 3.01 percentage points, with the middle budget capturing 89% of this gain using half as many additional layer executions. The final model surpasses each domain expert on its corresponding task-and-budget average, while achieving strong instruction-following performance across all budgets and retaining depth-dependent gains in mathematical reasoning and code generation.
Abstract (translated)
URL
https://arxiv.org/abs/2610.32492
PDF
-
L0-Norm-Based CycleGAN for Unpaired Brain CT-to-MR Synthesis
Abstract
Computed tomography (CT) and magnetic resonance (MR) imaging provide complementary information for the brain, but acquiring both modalities is not always possible. The cycle-consistent adversarial network (CycleGAN) enables unpaired CT-to-MR synthesis, but because the standard L1-norm cycle?consistency penalty grows linearly with the residual magnitude, the network tends to focus on large residuals. This paper introduces an L0-norm-based cycle?consistency term, in which the L0-norm is approximated by a smooth and bounded function. This approximation not only increases sensitivity to small residuals but also effectively suppresses the influence of large residuals. The proposed model and an otherwise identical L1-norm baseline were trained on 102 images and evaluated on 9 test images. The proposed model achieves a lower mean squared error (0.02746 versus 0.03464, a 20.7% reduction), a higher mean peak signal-to-noise ratio (15.80 versus 14.77 dB), and a higher mean structural similarity index (0.5689 versus 0.5407). It also has a lower maximum MSE and higher minimum PSNR and SSIM than the baseline. These results indicate that modifying the cycle-consistency penalty can improve reconstruction accuracy on this dataset.
Abstract (translated)
URL
https://arxiv.org/abs/2610.32491
PDF
-
Efficient Zero-Shot Video Anomaly Detection via Signal-Aware Adaptive Fusion
Abstract
Deploying Vision-Language Models for video anomaly detection confronts a fundamental constraint. Their semantic reasoning demands computational budgets incompatible with the scale of real-world video streams. We present Signal-Aware Adaptive Fusion (SAAF), a framework that achieves efficient zero-shot anomaly detection through three key innovations. First, we introduce a dual-signal adaptive sampler operating in semantic embedding space: a delta signal captures sudden events via frame-to-frame cosine distance, while a drift signal detects static and gradual anomalies through deviation from rolling temporal context. Their OR-gated combination with adaptive EMA-based thresholding provides comprehensive anomaly coverage without training data. Second, we propose a threshold-gated cascaded architecture where lightweight frame-level scoring precedes expensive video-level temporal analysis, invoking it only when confidence warrants additional context. Third, we design a signal-aware weighted fusion mechanism that dynamically adapts score combination based on which signal triggered frame selection. Across four benchmarks, SAAF delivers competitive accuracy at substantially lower cost. Among training-free zero-shot methods, SAAF sets a new state of the art on XD-Violence (92.07%), MSAD (86.70%), and ShanghaiTech (80.54%) AUC-ROC, with XD-Violence and MSAD also surpassing supervised baselines. SAAF sustains 44.95, 48.61, and 41.06 FPS on XD-Violence, UCF-Crime, and MSAD, a 2.43$\times$, 2.63$\times$, and 2.22$\times$ throughput improvement over the prior best zero-shot method. Code: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2610.32486
PDF
-
PCFBench: How Far Can Large Vision-Language Models Go in Physics-aware Photonic Inverse Design?
2026-10-23 11:02:02Abstract
Photonic crystal fibers (PCFs) underpin critical applications in telecommunications, sensing, and high-power laser delivery, yet their design remains a manual, expert-intensive process that couples geometric intuition with deep waveguide physics. Vision-language models (VLMs) offer a promising path toward automating PCF design, but can they truly reason about photonic physics from microscopy images, or do they merely recognize visual patterns? We introduce PCFBench, the first comprehensive benchmark for evaluating VLMs on the perception, reasoning, and decision steps of the photonic inverse-design loop. PCFBench spans 26 tasks organized into six categories of increasing cognitive demand (Geometry Perception, Physics Understanding, Text Generation, Multi-modal Reasoning, Inverse-Design Decision, and Code Generation), built from over 220K FDTD simulation samples across 18 structurally diverse fiber families. We evaluate both individual VLMs on isolated tasks and multi-agent systems (MAS) on the full sequential pipeline from perception to design. Our results reveal a striking perception-design gap: current models can identify fiber geometry with moderate success, yet consistently fail to translate that visual understanding into physics-aware design actions. Moreover, cascading agents through the pipeline exposes significant error propagation, where upstream perception mistakes compound into downstream design failures. These findings suggest that neither model scaling nor naive agent chaining suffices, and that closing the loop demands explicit physical grounding. PCFBench provides the community with a rigorous, multi-dimensional testbed to measure progress toward AI-assisted photonic engineering. The project is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2610.32485
PDF
-
When Agents Fail: Model or Harness--Who's to Blame?
Abstract
LLM agents combine models with execution harnesses, so agentic benchmark performance reflects their joint behavior. We argue that harnesses should therefore be evaluated through their interactions with underlying models. We develop a framework combining five complementary measures: (i) Valid Completion Rate (VCR), how reliably agents finish normally and deliver evaluable results, irrespective of task correctness; (ii) Model-Harness Fit, how well a harness supports task achievement through valid completion for a given model; (iii) Cross-Model Fit, a harness's overall suitability across evaluated models; (iv) Interaction Residuals, pairing-specific execution strengths and weaknesses hidden by component-level summaries; and (v) Token Load, the token demands of agent execution, including failed attempts. We evaluate four general-purpose harnesses (OpenCode, Codex CLI, Pi CLI, and DeepAgents) with four core models, plus two fixed native model-harness systems, on six benchmarks covering context management, tool use, long-context planning, multimodal interaction, and artifact handoff. Including native references and stress conditions, 111 evaluation settings yield 52,168 trajectories. Harness rankings vary across models and benchmarks; task performance and valid completion can favor different pairings, and similar task scores can accompany distinct execution profiles. OpenCode ranks first in cross-model performance, while DeepSeek V4 Flash Preview maintains consistently high Mean VCR across harnesses on the shared four-benchmark panel. These findings support evaluating model-harness interactions, training transferable harness-use capabilities in models, and harness designs adapted to the model and task.
Abstract (translated)
URL
https://arxiv.org/abs/2610.32483
PDF
-
CASE: Confusion-Aware Skill Evolution for Frozen Multimodal Emotion Models
2026-10-23 10:50:03Abstract
Multimodal large language models (MLLMs) have substantially advanced Multimodal Emotion Recognition (MER) by enabling fine-grained emotion understanding through reasoning over multimodal evidence. However, they still exhibit systematic confusion between semantically adjacent emotions with highly overlapping multimodal cues. Existing approaches typically address these errors through global parameter adaptation, making confusion correction costly and inflexible. We instead observe that emotion confusion is highly localized and therefore better addressed through reusable pair-specific expertise. Building on this insight, we propose Confusion-Aware Skill Evolution (CASE), the first agent-skill framework for MER. CASE maintains a sparse, self-evolving library of pair-specific EmotionSkills that externalize affective expertise from model parameters, enabling the learned expertise to be continuously refined and readily transferred across diverse frozen MLLM backbones. During inference, CASE dynamically retrieves the appropriate EmotionSkill to reinforce boundary-aware reasoning while keeping the MLLM backbone frozen. Extensive experiments on CA-MER, DFEW, and EMER demonstrate the effectiveness and transferability of CASE across both general-purpose and emotion-specialized MLLMs. Notably, CASE improves the average accuracy of Qwen3-VL-8B on the CA-MER benchmark from 46.80% to 59.13%, while achieving consistent gains across MLLMs.
Abstract (translated)
URL
https://arxiv.org/abs/2610.32481
PDF
-
PathJev: Decision-Native Vision-Language Modelling for Histopathology
2026-10-23 10:47:40Abstract
Vision-language models used for decisions in histopathology rarely return calibrated probabilities over the candidates a user cares about. Decision-native models write a question and a candidate set chosen at run time into the input and return a normalised distribution from a small decision head in one forward pass. We build PathJev-P, a decision-native model for histopathology: MedGemma-1.5-4B with LoRA adapters and a decision head, trained on 40,000 patch states from three public collections whose expert labels are turned into typed questions with organ-specific candidate sets and a "none of these" option. In a comparison over 3 seeds with an analysis plan fixed before testing, on test sets without documented overlap with the training data of the backbone, PathJev-P is more accurate than a generative model trained with the same backbone, data and selection rule, the zero-shot pathology model CONCH and a general medical decision model (balanced accuracy 0.541 against 0.492, 0.345 and 0.213); its advantage over the generative model comes from two organs absent from training. The calibration advantage depends on how the generative model's probabilities are read out. With the summed candidate log-likelihood used in previous work, PathJev-P has the lower calibration error (ECE 0.204 against 0.279); with a length-normalised read-out of the same generative model, examined post hoc, the two have similar mean calibration error (0.201) and the generative model is better calibrated on the unseen organs, where PathJev-P remains over-confident. A supervised linear probe on a pathology foundation model is better calibrated in every setting. In a qualitative whole-slide demonstration, PathJev-P labels unsupervised tissue clusters with abstention and separates carcinoma from other tissue while running the model on a fraction of the tiles.
Abstract (translated)
URL
https://arxiv.org/abs/2610.32478
PDF
-
CauProphet: Teaching Agents to Develop Causal Cognition like Humans
Abstract
Large language model (LLM) agents increasingly support long-horizon tasks by leveraging external memory and past experience. However, existing agents mainly retrieve records or summaries, lacking the ability to form and refine causal understanding of the world. Inspired by human causal cognition, we introduce CauProphet, an agent that develops and updates causal knowledge within its Causal Cognition. For each task, CauProphet combines relevant knowledge in Causal Cognition with current evidence to construct a Causal Belief, a structured belief capturing causal factors and the effects of possible actions. It uses this belief for decision-making and continuously updates both the belief and Causal Cognition as new observations emerge, enabling causal knowledge that persists beyond individual experiences. Evaluations demonstrate that CauProphet progressively acquires causal knowledge and transfers it to future decision-making. In financial research, CauProphet leverages prior discoveries to improve downstream analyses and outperforms all baselines in mean rank correlation with next-day returns, achieving a 25.4% improvement over a baseline agent. It also improves prediction accuracy from 21.3% to 28.6% in dynamic real-world settings. Further experiments validate its ability to perform reliable causal analysis and action selection across scientific and everyday tasks, demonstrating the potential of causal experience learning for next-generation LLM agents.
Abstract (translated)
URL
https://arxiv.org/abs/2610.32477
PDF
-
Beyond Post-Hoc Rewards: Privileged Reasoning Exploration for Multimodal Embeddings
Abstract
Advances in Multimodal Large Language Models (MLLMs) have driven substantial progress in multimodal embeddings. Recent methods exploit generative capabilities by augmenting embedded inputs with self-generated reasoning to produce more informative embeddings. However, conventional retrieval supervision for training embedding models specifies pairwise relevance relations without explicitly guiding the reasoning used for input augmentation. Such reasoning is neither directly observed nor uniquely determined by the input, allowing multiple plausible traces to introduce semantic cues of varying retrieval utility. Existing approaches largely optimize reasoning conditioned on the input alone, limiting the exploration of retrieval-relevant traces. We introduce PILOT, a framework that models reasoning as a latent variable and instantiates its prior and posterior distributions as distinct reasoning modes of a shared MLLM. The prior conditions only on the input, whereas the posterior uses additional training-time relational evidence to guide reasoning exploration. To translate the posterior's broader exploration into inference-time improvements, we derive an importance-weighted variational objective and instantiate it as a prior-posterior co-evolution framework that improves both modes under a shared inference-compatible reward while selectively transferring posterior trajectories to the prior according to their utility and prior reachability. On the 78-task MMEB-V2 benchmark, PILOT outperforms the primary baseline across all 12 task groups at both 2B and 7B parameters, improving overall scores by 3.2 and 3.3 points, respectively, with the largest gains in visual-document retrieval (+6.1 and +6.4 points). Our results show that relational supervision is most valuable not as a post-hoc reward but as a signal for shaping how reasoning is explored in training reasoning-enhanced embedding models.
Abstract (translated)
URL
https://arxiv.org/abs/2610.32474
PDF
-
From Perception to Pixels: Human-Derived Visually Distinct Colors for Improved Dominant Color Extraction
Abstract
Dominant color extraction is a fundamental task in computer vision, with applications in design, image retrieval, and content recommendation. However, conventional algorithms based on device-oriented color spaces (e.g., RGB) and fixed quantization often fail to capture how humans perceive color differences. In this study, we propose a human-perception-based framework for dominant color extraction, built on a novel Human Color Dictionary (HCD) derived from a perceptual experiment. Participants (n=110) were sequentially shown 120 colors and asked to identify those they perceived as unique. Based on their responses, we constructed a 120 by 120 co-occurrence matrix and applied a greedy selection algorithm to derive 64 perceptually distinct colors, forming the HCD. This dictionary was integrated into RGB color histogram and K-means color extraction algorithms by initializing centroids and mapping pixel values to their nearest perceptual color using the deltaE 00 distance. Evaluation on a dataset of 4,562 images paired with expert-designed palettes demonstrated that our method improves perceptual alignment, as measured by Earth Mover's Distance (EMD) and best-match distance. Our findings suggest no meaningful gender differences in the number of uniquely perceived colors, indicating that sex does not significantly influence color perception in this task. Another key finding is that highly saturated colors are perceived as the most distinguishable, whereas dark and deep colors tend to perceptually collapse and are less reliably differentiated. The results confirm that incorporating experimentally derived human color representations ensures the generated color clusters align more closely with human perception, thus improving the usability of color palette generation in practical applications.
Abstract (translated)
URL
https://arxiv.org/abs/2610.32473
PDF
-
Online Continual Learning in Edge-Cloud Applications
Abstract
Deployed machine learning models in real-world IoT and edge/cloud systems face a critical reliability bottleneck: the requirement for online adaptation to non-stationary data streams while operating without access to the full original training dataset. Performance is severely degraded when models encounter Out-of-Distribution (OOD) inputs or suffer from catastrophic forgetting. Addressing these challenges requires adaptation mechanisms capable of learning from evolving data streams while preserving previously acquired knowledge. This paper proposes an edge-cloud methodology for integrating Continual Learning (CL) with Out-of-Distribution (OOD) Detection and Concept Drift Detection in IoT applications. The methodology incorporates ODIN and ECDD as alternative mechanisms for identifying informative samples for annotation and triggering rehearsal-based model updates. Experimental results across automotive, healthcare, and agricultural datasets demonstrate the applicability of the proposed methodology and its potential to support model adaptation while mitigating catastrophic forgetting.
Abstract (translated)
URL
https://arxiv.org/abs/2610.32472
PDF
-
ALLMTS: Automated LLM-based Medical Triage System
Abstract
Telephone triage is a safety critical task: from a brief intake note, a nurse must decide whether a caller needs an ambulance, an emergency department, an appointment, or self care at home. Queried directly, large language models (LLMs) are unreliable: they over-escalate routine cases and cannot justify their recommendations against an auditable clinical standard. We present the Automated LLM-based Medical Triage System (ALLMTS), which grounds every disposition in the same commercially maintained telehealth guidelines nurses use. Rather than asking an LLM to triage, ALLMTS makes it execute the guideline: it selects the appropriate guideline and evaluates its triage assessment questions. The disposition follows deterministically from the first positive answer, using the same traversal rule a nurse follows. We evaluate ALLMTS on 566 synthetic, expert-validated scenarios spanning 14 dispositions and designed to test specific guideline questions. ALLMTS achieves 0.95 accuracy, compared with 0.47 for a raw-LLM baseline. On a curated 50-scenario subset released as an open benchmark, ALLMTS reaches 0.98 accuracy, on par with the 0.944 correct disposition rate that a five nurse expert panel recorded on those same scenarios. Critically, on that subset the baselines have over-triage rates of up to 0.50 (0.44 on the full dataset), whereas ALLMTS reduces the over-triage rate to at most 0.02 while maintaining low under-triage. These results support coupling LLM language understanding with executable clinical guidelines for triage from intake notes.
Abstract (translated)
URL
https://arxiv.org/abs/2610.32469
PDF
-
OPTARI: An open-source software framework for clinical translation of optoacoustic imaging
Abstract
Combined optoacoustic (OptA) and ultrasound imaging is a promising modality across a wide range of pathologies. Its use in clinical research, however, is limited by a lack of analysis software accessible to users without programming expertise. Existing tools are either script-based engineering frameworks or proprietary vendor applications. We present OPTARI, an open-source graphical software application to facilitate large-scale clinical optoacoustic-ultrasound (OPUS) studies. OPTARI builds on the napari image viewer and the PATATO toolkit for OptA data processing. It supports native vendor and open HDF5 data formats, aligns with the IPASC standard, and is distributed as a standalone executable. Features include region-of-interest annotation, reconstruction, chromophore unmixing, deep-learning-based tissue segmentation, and temporal and spectral analysis modules. OPTARI offers a combination of features for clinical study analysis that none of the compared tools provide. The automated tissue segmentation produced results comparable to expert labeling for muscle tissue, and batch processing reduced analysis time from minutes to under one second per scan. The tool is used in ongoing clinical OPUS studies. OPTARI connects engineering frameworks with medical research and provides an open-source basis for reproducible and standardized analysis in clinical OptA imaging studies.
Abstract (translated)
URL
https://arxiv.org/abs/2610.32468
PDF
-
Reading Between the Layers: Cross-Modal Alignment and Representational Hierarchy in Vision Transformers
2026-10-23 10:30:51Abstract
While large vision-language models excel at optical character recognition, it remains unclear how their underlying vision encoders process text and its meaning, and whether they support anything resembling reading comprehension. We systematically study transformer-based vision encoders, using cross-modal alignment between rendered text and visual concepts together with probing and retrieval-based analyses as a controlled window into these broader capabilities. Our layer-wise analyses span orthographic, grammatical, and semantic organization, and reveal that strong cross-modal mapping emerges only in the deepest layers, after unimodal object and text recognition have already developed, coinciding with a progression from surface-level to semantic representations. Although this hierarchy is strongest in contrastively trained models, larger self-supervised models and text-rich training data partially support similar behaviors without explicit language supervision. Together, our results suggest that increasing model and training-data scale, as well as text-rich training data, can partially compensate for the absence of language supervision.
Abstract (translated)
URL
https://arxiv.org/abs/2610.32463
PDF
