Abstract
A medical vision-language model may analyze a given CT scan and correctly detect both kidneys in view. Continuing this conversation, if a user provides a prescription with an opposite direction, medical VLMs often sycophantically choose to realign their diagnosis. In this work, we study regressive sycophancy under fabricated evidence, where a model abandons a correct answer when pushback comes with a fake document. We test four open VLMs on 400 VQA-RAD yes/no questions in 19,200 multi-turn conversations, with and without a fabricated prescription. Without a disclosure that documents are synthetic, the models abandon 80.1% of 6,488 correct answers. Notably, under one script each, the same prescription raises the flip rate of Qwen2.5-VL-7B from 45.9% to 89.6% and lowers that of Qwen3-VL-8B from 77.7% to 50.9%. A model rejected the prescription while still preserving the correct answer in 915 of 3,232 conversations with the prescription (28.3%), and 40% of these flipped in later turns. System-prompt sentences that disclose the documents as synthetic lower one model's flip rate from 77.8% to 47.3%. Thus, we conclude that awareness about a forgery does not prevent a model from adopting its answer.
Abstract (translated)
URL
https://arxiv.org/abs/2610.17989
