Abstract
Inference-time alignment enables large language models (LLMs) to generate outputs aligned with end-user preferences without further training. Recent post-training methods achieve this by using small guidance models to modify token generation during inference. These methods typically optimize a reward function KL-regularized by the original LLM taken as the reference policy. A critical limitation, however, is their dependence on a pre-trained reward model, which requires fitting to human preference feedback--a potentially unstable process. In contrast, we introduce PITA, a novel framework that integrates preference feedback directly into the LLM's token generation, eliminating the need for a reward model. PITA learns a small preference-based guidance policy to modify token probabilities at inference time without LLM fine-tuning, reducing computational cost and bypassing the pre-trained reward model dependency. The problem is framed as identifying an underlying preference distribution, solved through stochastic search and iterative refinement of the preference-based guidance model. We evaluate PITA across diverse tasks, including mathematical reasoning and sentiment classification, demonstrating its effectiveness in aligning LLM outputs with user preferences.
Abstract (translated)
推理时间对齐使大型语言模型（LLM）能够在无需进一步训练的情况下生成与最终用户偏好一致的输出。最近的后期训练方法通过使用小规模引导模型来修改推断过程中的令牌生成，从而实现了这一点。这些方法通常优化一个由原始LLM作为参考策略进行KL正则化的奖励函数。然而，一个关键限制是它们依赖于预训练的奖励模型，该模型需要适应人类偏好的反馈——这是一个可能不稳定的过程。相比之下，我们引入了PITA这一新颖框架，它直接将偏好反馈整合到LLM的令牌生成过程中，从而消除了对奖励模型的需求。PITA学习一个小规模基于偏好的引导策略，在推理时间修改令牌概率而无需对LLM进行微调，这样可以减少计算成本并绕过预训练奖励模型的依赖性。该问题被构架为识别潜在偏好分布的问题，并通过随机搜索和迭代改进基于偏好的引导模型来解决。我们在包括数学推理和情感分类在内的各种任务上评估了PITA，证明了它在将LLM输出与用户偏好对齐方面的有效性。
URL
https://arxiv.org/abs/2507.20067
