Skip to content
Live
Trending storyPast story

HABIT-Bench release

1 reports1 outlets22 days agoUpdate

The story so far

AI overview

LUMIA-Group released HABIT-Bench, a benchmark for evaluating an agent's ability to learn habits from longitudinal weak evidence. The benchmark requires agents to integrate scattered evidence and choose appropriate actions for new requests. Qwen3-235B-A22B and GLM-5.3-Flash were used to evaluate eight memory methods, with the best configuration achieving a macro-average accuracy of 33.61% in a four-choice task, outperforming random guessing at 25%. Even with labeled evidence, there are still many errors, indicating that understanding how to guide decisions is more important than simply recalling history. HABIT-Bench includes code and datasets, available at https://github.com/LUMIA-Group/habit-bench and https://huggingface.co/datasets/hollyming/HABIT-Bench, respectively. The benchmark is used to evaluate an agent's habit learning ability in the fields of food, finance, software, and travel.

AI-written from the reports · Updated 22 days ago

Coverage timeline

Follow the coverage to see every side of the story.

28 Oct
  1. Hugging Face Daily Papers
    HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memory

    HABIT-Bench evaluates an agent's ability to learn habits from synthetic user histories involving hundreds of interactions, requiring the agent to integrate scattered evidence and choose appropriate actions for new requests. Using Qwen3-235B-A22B and GLM-5.3-Flash, eight memory methods were evaluated, with the best configuration achieving a macro-average accuracy of 33.61% in a four-choice task, outperforming random guessing at 25%. Even with labeled evidence, there were still numerous errors, indicating that understanding how to guide decisions is more important than simply recalling history.