StateBench tests language models for tracking state changes
The story so far
Hugging Face has launched StateBench, a benchmark for testing the performance of language models in tracking state changes in long documents. The benchmark includes four difficulty levels, ranging from simple immutable assignments to complex dependency assignments, and found that cyclic and hybrid models have stronger generalization capabilities on direct training tasks, but cutting-edge models like Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol still fail in complex state tracking with long inputs.
AI-written from the reports · Updated 15 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- Hugging Face BlogStateBench: Testing how language models track state changes over long documents
StateBench is a long-context state tracking benchmark that tests how language models maintain tracking changes in entity information within long documents. The benchmark includes four difficulty levels, ranging from simple immutable assignments to complex dependency assignments. It found that cyclic and hybrid models generalize better than Transformers on direct training tasks, but cutting-edge models like Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol still fail in complex state tracking with long inputs.