HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memory

Brief overview

HABIT-Bench evaluates an agent's ability to learn habits from synthetic user histories involving hundreds of interactions, requiring the agent to integrate scattered evidence and choose appropriate actions for new requests. Using Qwen3-235B-A22B and GLM-5.3-Flash, eight memory methods were evaluated, with the best configuration achieving a macro-average accuracy of 33.61% in a four-choice task, outperforming random guessing at 25%. Even with labeled evidence, there were still numerous errors, indicating that understanding how to guide decisions is more important than simply recalling history.

Article

Hi everyone! I'm one of the authors of HABIT-Bench 👋

Can an agent learn your habits without being explicitly told—and know when not to apply them?

Remembering a past choice is not enough. An agent needs to distinguish a recurring habit from a one-off exception, recognize when habits change, and avoid treating its own suggestions as the user's preferences.

We introduce HABIT-Bench to evaluate this ability across food, finance, software, and travel. It uses synthetic user histories spanning hundreds of sessions. The tasks require agents to piece together scattered, incomplete evidence and select an appropriate action for a new request—not simply recall something the user explicitly stated.

We evaluate eight memory methods with Qwen3-235B-A22B and GLM-5.3-Flash. The best memory-method configuration reaches 33.61% macro accuracy on four-choice tasks, compared with 25% for random guessing. Even when annotated evidence is provided directly, substantial errors remain: finding relevant history is only part of the challenge.

Remembering what happened is not the same as understanding what should guide the next decision.

💻 Code: https://github.com/LUMIA-Group/habit-bench
🤗 Dataset: https://huggingface.co/datasets/hollyming/HABIT-Bench

We'd love to hear your thoughts, discuss the findings, and see how other memory systems perform on HABIT-Bench!