How to Read AI Benchmarks Without Being Fooled

What is a benchmark?

In computing, a benchmark is a test that runs programs or operations to measure how a system performs. Specifications alone are not enough to compare: a processor with a slower clock can perform as well as, or better than, a faster one on paper. That is why standard workloads are used.

In AI, a benchmark can be a set of knowledge questions, programming problems or software engineering tasks, with an automatic way to tell whether an answer is correct. The model answers and the percentage of correct responses is calculated. Other benchmarks use human pairwise evaluation: two anonymous answers, and a person votes for the one they prefer.

What do the best-known benchmarks measure?

These four appear again and again in provider presentations. The table summarizes what each one measures and what its authors reported when they published it.

The figures in the table are the ones from each publication date. They show the scale of the test; they are not for comparing today’s models. For that, check each benchmark’s current table, with its date and configuration. Arena, for example, started in April 2023 as Chatbot Arena, was renamed LMArena, and since January 28, 2026 is called Arena.

Why can scores mislead?

Goodhart’s law summarizes the problem. In the wording popularized by anthropologist Marilyn Strathern in 1997: when a measure becomes a target, it ceases to be a good measure. If labs optimize for a benchmark, the score rises without necessarily improving usefulness on other tasks. Hardware benchmarks already had this problem: vendors tend to tune their products for standard tests, and many benchmarks focus only on speed and leave out total cost.

There are concrete cases in AI:

• Tests that wear out. According to Latent Space’s analysis of OpenAI’s February 2026 announcement, OpenAI stopped reporting SWE-bench Verified. It found that frontier models could reproduce patches or problem statements for specific tasks, probably because those open repositories are in the training data. A review of 138 hard problems also found tests that rejected correct solutions or required features the problem statement never mentioned. OpenAI now recommends SWE-bench Pro. This is contamination: if the questions were in the training data, the score measures memory, not capability.
• Rankings with unequal advantages. The study The Leaderboard Illusion (April 2025) argues that private testing lets some providers try several variants and publish only the best one. It cites 27 private variants tested by Meta before the Llama 4 release and estimates that Google and OpenAI received 19.2% and 20.4% of Arena’s data. These are the authors’ claims, but they show why a public ranking is not a neutral yardstick.
• Different versions of the same model. Meta entered Llama 4 Maverick on LMArena with a version different from the one it released, and LMArena updated its policies in April 2025.
• Configuration. Temperature, number of attempts or enabled tools change the result. Two scores are comparable only if they were measured the same way.

What should you check before trusting a score?

A designer who takes part in choosing a model does not need to run any math, but does need to ask good questions. A programming score says nothing about the quality of a welcome message, and your users do not answer exam questions.

How do you evaluate with your own cases?

A practical process for the team:

1. Gather real cases from your product: as a starting point, 30 to 50, with inputs exactly as users send them.
2. Define what a good answer is with concrete criteria: correct, on-brand tone, the requested format, no invented data.
3. Run the candidates with the same prompt, the same temperature and the same context.
4. Grade blind: the evaluator should not know which model produced each answer.
5. Also measure latency and cost per answer, because the best answer is not always the most useful one. A small model that meets your criteria can be the best decision.
6. Repeat the test when the model or the prompt changes. If you are also deciding between an API and your own model, see build vs. buy.

Example

Illustrative example: a team wants to choose between two models for a customer support assistant. Model A’s provider publishes a higher score on a reasoning benchmark. The team builds 40 real customer queries and grades blind. Model B gets better marks on tone and on following the returns policy, and costs a third less per answer. The benchmark did not fail; it simply measured something else. The decision is made with the table of real cases and the cost.

Common mistakes

• Choosing a model only by its place in a general ranking.
• Comparing published scores that used different configurations or dates.
• Quoting a figure from a 2021 paper as if it described current models.
• Ignoring sample size and small differences.
• Evaluating without blinding, so the model’s brand influences the grades.
• Forgetting to repeat the evaluation when the provider updates the model.

Sources

• Measuring Massive Multitask Language Understanding (arXiv:2009.03300)
• Evaluating Large Language Models Trained on Code (arXiv:2107.03374)
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (arXiv:2310.06770)
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (arXiv:2403.04132)
• The Leaderboard Illusion (arXiv:2504.20879)
• Arena (AI platform), Wikipedia
• SWE-Bench is dead, Latent Space
• Goodhart’s law, Wikipedia
• Benchmark (computing), Wikipedia