评估评估者：调查 LLM 判官在个性化写作风格评估中的表现
Evaluating the Evaluators: Investigating LLM Judges for Personalized Writing Style Assessment
研究人员评估了大型语言模型（LLM）判官在个性化写作任务中的表现，发现判官质量与 MMLU 测量的通用 LLM 能力相关性较弱（Pearson’s ρ ≤ 0.2），且表现受写作任务和评估标准（如通用帮助性 vs 风格个性化）影响。LLM 评估器在成对比较中比独立评分响应时更一致且表现更佳，而“最强”LLM 并未最可靠地复现人类评估，可能因对人类标注者和较弱 LLM 忽略的细微细节过度敏感。
AuthorsNatalie Mackraz‡, Andrew Silva, Aini Putkonen, Brihi Joshi†**, Joelle Alcaidinho, Arnav Arora§**, Barry-John Theobald, Katherine Metcalf‡
This paper was accepted at the Can We Trust AI Evaluation? Workshop at NeurIPS 2026.
We evaluate how good large language model (LLM) judges are for personalized writing tasks. This is important when a user’s preferred writing style is known (e.g., “upbeat”) and an LLM judge is used to evaluate whether generated text adheres to this preference. We examine how the writing task, the combination of generator and judge LLM, the evaluation objective, and general commonsense and reasoning ability impact LLM judge performance. To this end, we generate text with four LLMs for three long-form writing tasks, collect human labels evaluating which of two responses is better, collect judge-LLM, pairwise-ranking annotations, and then compare the human and judge LLM annotations. We find that judge quality correlates weakly with general LLM ability measured using MMLU (Pearson’s ρ ≤ 0.2) and varies by both writing task and evaluation criteria (i.e., general helpfulness versus style personalization). We also find that LLM evaluators are more consistent and performant using paired comparisons rather than scoring responses independently. Finally, we find that the “strongest” LLMs do not most reliably reproduce the human evaluations, which we hypothesize relates to exaggerated sensitivity to subtle details that human annotators and weaker LLMs ignore.
- ‡ Equal contribution
- † University of Southern California
- § University of Copenhagen
- ** Work done while at Apple
Related readings and updates.
Large language models (LLMs) exhibit a wide range of human-like behaviors, from expressing thoughts and emotions, to engaging in relationship-building with users, to refusing requests and maintaining boundaries. Despite their prevalence, researchers and practitioners lack methods and empirical insights to make informed decisions about when and what types of human-like behaviors LLMs should exhibit. To fill this gap, we present a multi-dimensional…
Accommodating human preferences is essential for creating aligned LLM agents that deliver personalized and effective interactions. Recent work has shown the potential for LLMs acting as writing agents to infer a description of user preferences. Agent alignment then comes from conditioning on the inferred preference description. However, existing methods often produce generic preference descriptions that fail to capture the unique and…
来源: Apple Machine Learning Research · machinelearning.apple.com