In March 2026, the AI safety research group METR looked at real pull requests that had passed SWE-bench Verified, one of the most cited coding benchmarks in AI. Their finding was blunt: roughly half of those "passing" fixes would not have actually been merged into the real codebase they came from. A repo maintainer looked at each one and made the call. The benchmark said the model solved the problem. The maintainer, about half the time, said no. That gap, between what a benchmark measures and what actually works, is the whole reason evaluating AI models has turned into its own small industry.
What a benchmark actually is
A benchmark is a fixed set of test questions or tasks, with a known correct answer, that every model gets scored against the same way. MMLU asks multiple-choice questions across 57 subjects, from law to physics to history. It tests general knowledge, not any one skill. GPQA-Diamond is harder. It uses graduate-level science questions, picked specifically because experts get them right but a quick search doesn't help much. AIME uses real competition math problems where the answer is a single number, so grading is simple. That's the appeal of a fixed benchmark. It's cheap to run and hard to argue with. Either the model's answer matches the known correct one, or it doesn't.
The problem is that fixed benchmarks wear out over time. Once every major model scores 90% or higher on something, the benchmark stops telling good models apart from great ones. And once a benchmark's questions have been public for a while, there's a real risk they leaked into some model's training data. That inflates scores in a way that has nothing to do with actual skill.
Coding and reasoning tests try to fix that, imperfectly
HumanEval and SWE-bench moved coding evaluation closer to real work. Instead of multiple choice, models write actual code, and that code either passes hidden tests or it doesn't. SWE-bench goes a step further: it pulls real GitHub issues and checks whether a model's patch actually fixes them. That sounds like a strong test of real engineering, and it is a big step up from toy problems. But METR's audit, the one mentioned above, found that around half of SWE-bench-passing fixes wouldn't survive a real code review. They passed the test suite. They just weren't changes a human maintainer would accept. The benchmark checks whether tests go green. It doesn't fully check whether the code is good.
Reasoning benchmarks run into a similar problem, one researchers call saturation. Once a benchmark gets easy enough for top models to ace every time, it stops telling models apart. So the field keeps building harder versions. MMLU gave way to MMLU-Pro. GPQA added a tougher "Diamond" tier. Some 2026 researchers have started calling this cycle exactly what it is: an ouroboros, a benchmark that keeps eating its own tail and needing to be replaced.
Human evaluation and arena-style testing
Fixed benchmarks miss things like tone, helpfulness, and how a model handles a vague or ambiguous request. To catch that, platforms like LMArena show people two anonymous model answers to the same question and ask them to vote for the better one. Do this enough times and you get an Elo-style ranking, like chess ratings, based on what people actually prefer instead of a known right answer.
This catches something benchmarks can't: whether people actually like using the model. But it has a real weakness too. People tend to reward answers that sound confident and well-formatted over ones that are correct but blunt. So an arena ranking can end up favoring a model that's persuasive over one that's simply right.
Real-world performance: the newest and most demanding test
In 2025, OpenAI introduced GDPval to close the exact gap this article opened with. Instead of exam-style questions, GDPval uses real work: legal briefs, engineering documents, customer support transcripts, all built from the actual work of industry professionals with an average of 14 years of experience, across 44 occupations. Instead of an automated right-or-wrong grade, GDPval mostly relies on blind human experts. They compare a model's output to a real professional's output without knowing which is which.
The 2026 results show something genuinely worth noting. Frontier model performance on GDPval has been improving in a roughly straight line over time, and the best current models are getting close to expert quality on many of these tasks. But "close to expert quality" and "reliably replacing an expert" are still two very different claims.
Hallucination rates, cost, and speed round out the picture
A model can be highly capable and still unreliable if it hallucinates, states something false with total confidence, often enough to matter. That's why hallucination rates usually get reported separately from raw capability scores instead of folded into one number.
Cost and speed matter just as much if you're actually building something. A model that scores two points higher on a reasoning benchmark, but costs five times as much per token or takes ten times longer to respond, might be the worse choice for a real product. That's why serious comparisons increasingly plot score against price and against response time, instead of ranking purely by capability.
So which model is actually better?
No single number answers that. The honest approach, the one researchers building evaluation tools use in 2026, is to look at several benchmarks that test different things, check scores against more than one leaderboard since methodology varies, and weigh all of that against real usage, cost, and speed for the specific task you actually care about.
A model that tops a coding leaderboard isn't automatically the best choice for legal writing. A model that wins on human preference in a chat arena isn't automatically the most accurate one. "Better" only means something once you say better at what. And increasingly, better according to which kind of test.