HABIT-Bench 发布：评估智能体从纵向弱证据中习得习惯的能力
HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memory
HABIT-Bench 评估智能体从数百次会话的合成用户历史中习得习惯的能力，要求智能体整合零散证据并选择新请求的适当行动。使用 Qwen3-235B-A22B 和 GLM-5.3-Flash 评估八种记忆方法，最佳配置在四选任务中达到 33.61% 宏平均准确率，优于随机猜测的 25%。即使提供标注证据，仍存在大量错误，表明理解应如何指导决策比单纯回忆历史更重要。
Hi everyone! I'm one of the authors of HABIT-Bench 👋
Can an agent learn your habits without being explicitly told—and know when not to apply them?
Remembering a past choice is not enough. An agent needs to distinguish a recurring habit from a one-off exception, recognize when habits change, and avoid treating its own suggestions as the user's preferences.
We introduce HABIT-Bench to evaluate this ability across food, finance, software, and travel. It uses synthetic user histories spanning hundreds of sessions. The tasks require agents to piece together scattered, incomplete evidence and select an appropriate action for a new request—not simply recall something the user explicitly stated.
We evaluate eight memory methods with Qwen3-235B-A22B and GLM-5.3-Flash. The best memory-method configuration reaches 33.61% macro accuracy on four-choice tasks, compared with 25% for random guessing. Even when annotated evidence is provided directly, substantial errors remain: finding relevant history is only part of the challenge.
Remembering what happened is not the same as understanding what should guide the next decision.
💻 Code: https://github.com/LUMIA-Group/habit-bench
🤗 Dataset: https://huggingface.co/datasets/hollyming/HABIT-Bench
We'd love to hear your thoughts, discuss the findings, and see how other memory systems perform on HABIT-Bench!
来源: Hugging Face Daily Papers · huggingface.co