Abstract
Language models now routinely grade other language models. If post-training makes a model's writing more predictable, it may also teach the same model, as a judge, to reward predictable writing. We test this directly. We follow two open 7-billion-parameter model families, OLMo-2 and Zephyr, from their base models through supervised fine-tuning (SFT) and direct preference optimization (DPO), and measure every stage as a writer of short stories and as a judge of story pairs. As writers, the models drift as feared. Each family's trained model finds its stages' stories more familiar in the order base, SFT, DPO, in all ten prompts, and a model from the other family finds the trained stories 7.0 to 9.2 percent (OLMo-2) and 24 to 25 percent (Zephyr) less surprising per token. As judges, they barely move toward predictable writing. Asked which story is better, a question every judge passes a competence check on, no trained judge's preference for the more predictable story grows by as much as one point of pick probability at its estimate. A one-sided 95% upper bound on that growth is about 5 points on the widest pairs. Training instead makes judges more swayed by story length and by display order, and it breaks "more creative" as a question. Trained judges no longer reliably prefer a story to its own words in scrambled order. For these two families, the practical hazard in a small trained judge of creative writing lies in these measurable habits.
Abstract (translated)
URL
https://arxiv.org/abs/2609.32196
