When you read a long document, you track changes naturally. If a company’s budget starts at one million dollars and later increases by two million, you know the current budget is three million. On November 3, 2026, the Allen Institute for AI (Ai2) released StateBench, a test designed to see if AI models can do the same thing.
It turns out that some of the most famous AI models struggle when facts change too many times.
Testing Four Levels of Memory
StateBench does not just ask models to find a hidden fact. It measures how well models track variables across four difficulty levels.
The easiest level is "r-trivial," where a value is set once and never changes. The next level, "aperiodic," overwrites the value, so the model only needs to find the most recent update. The third level, "soluble," forces the model to do math on the current state, like adding five to an existing number.
The hardest level is "general finite." Here, one variable copies the current value of another variable. If a model loses track of one number early on, the error cascades through the rest of the document.
How the Top Models Performed
Ai2 tested major models like Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and Luna, and the open-source Olmo 3. The test asked them to generate the final values purely from memory, without using outside coding tools to calculate the answers.
The results were striking. All these models did well on the simple tasks. However, as the documents grew longer and the tasks reached the hardest "general finite" level, their performance collapsed. On the longest tests, 13% accuracy was the absolute best they could manage, with some scoring even lower.
A Problem With the Foundation
To figure out why this happens, Ai2 did not just test giant models. They built small, 275-million-parameter test models from scratch to compare different AI architectures.
Most of today's big AI systems are built on an architecture called a transformer. Ai2 found that these small transformers completely failed the hardest tracking tasks, regardless of how they were trained.
However, Ai2 also tested "hybrid" models using a newer memory system called Gated DeltaNet. These hybrids learned the hardest rules and successfully applied them to documents much longer than anything they saw during training.
What This Means for the Future
This finding only proves the architecture difference at a small scale. We do not yet know for sure if hybrid designs will behave the same way when scaled up to the size of a frontier model.
Still, the results suggest a clear takeaway. If an AI keeps forgetting how facts connect in a long document, the solution might not be more training data. Instead, engineers might need to rethink the basic structure of how the model stores its memory.
