# How to Read AI Benchmarks Without Being Fooled

> What MMLU, HumanEval, SWE-bench and Arena measure, why a score can mislead (contamination, tuning to the test) and how to compare models with your own cases.

*Tags: ai, evaluation, models, metrics*

---


> [!info] Quick Definition
> An **AI benchmark** is a standardized test, with defined tasks and criteria, used to compare models. Its score is a useful reference, but it only describes performance on what that test measures, under the conditions in which it was run.


## What is a benchmark?

In computing, a benchmark is a test that runs programs or operations to measure how a system performs. Specifications alone are not enough to compare: a processor with a slower clock can perform as well as, or better than, a faster one on paper. That is why standard workloads are used.

In AI, a benchmark can be a set of knowledge questions, programming problems or software engineering tasks, with an automatic way to tell whether an answer is correct. The model answers and the percentage of correct responses is calculated. Other benchmarks use human pairwise evaluation: two anonymous answers, and a person votes for the one they prefer.

## What do the best-known benchmarks measure?

These four appear again and again in provider presentations. The table summarizes what each one measures and what its authors reported when they published it.

| Benchmark | What it measures | Figure from the original publication |
|---|---|---|
| MMLU | Knowledge and understanding across 57 subjects: elementary mathematics, US history, computer science, law and more | Hendrycks et al. (2020): the largest GPT-3 model beat random chance by almost 20 percentage points |
| HumanEval | Functional correctness of programs generated from a description (docstring) | Chen et al. (2021): Codex solved 28.8% of the problems and GPT-3 0% |
| SWE-bench | Solving 2,294 real software engineering problems taken from issues and pull requests in 12 popular Python repositories | Jimenez et al. (2023): the best model, Claude 2, solved 1.96% |
| Arena (formerly Chatbot Arena) | Human preference in pairwise comparisons between anonymous models | Chiang et al. (2024): over 240,000 votes and good agreement with expert raters, according to the authors |

The figures in the table are the ones from each publication date. They show the scale of the test; they are not for comparing today's models. For that, check each benchmark's current table, with its date and configuration. Arena, for example, started in April 2023 as Chatbot Arena, was renamed LMArena, and since January 28, 2026 is called Arena.

## Why can scores mislead?

**Goodhart's law** summarizes the problem. In the wording popularized by anthropologist Marilyn Strathern in 1997: when a measure becomes a target, it ceases to be a good measure. If labs optimize for a benchmark, the score rises without necessarily improving usefulness on other tasks. Hardware benchmarks already had this problem: vendors tend to tune their products for standard tests, and many benchmarks focus only on speed and leave out total cost.

There are concrete cases in AI:

- **Tests that wear out.** According to Latent Space's analysis of OpenAI's February 2026 announcement, OpenAI stopped reporting SWE-bench Verified. It found that frontier models could reproduce patches or problem statements for specific tasks, probably because those open repositories are in the training data. A review of 138 hard problems also found tests that rejected correct solutions or required features the problem statement never mentioned. OpenAI now recommends SWE-bench Pro. This is **contamination**: if the questions were in the training data, the score measures memory, not capability.
- **Rankings with unequal advantages.** The study *The Leaderboard Illusion* (April 2025) argues that private testing lets some providers try several variants and publish only the best one. It cites 27 private variants tested by Meta before the Llama 4 release and estimates that Google and OpenAI received 19.2% and 20.4% of Arena's data. These are the authors' claims, but they show why a public ranking is not a neutral yardstick.
- **Different versions of the same model.** Meta entered Llama 4 Maverick on LMArena with a version different from the one it released, and LMArena updated its policies in April 2025.
- **Configuration.** Temperature, number of attempts or enabled tools change the result. Two scores are comparable only if they were measured the same way.

## What should you check before trusting a score?

A designer who takes part in choosing a model does not need to run any math, but does need to ask good questions. A programming score says nothing about the quality of a welcome message, and your users do not answer exam questions.

| Question | What to check | Warning sign |
|---|---|---|
| What task does it measure? | Test description | A task very different from yours |
| How many cases does it have? | Size of the set | Two-point gaps on few cases: could be noise |
| Who ran it? | Author or third parties | Only the provider, no published method |
| Under what conditions? | Configuration and format | Unpublished conditions |
| Could there be contamination? | Date of the test and source of the data | Questions public for years |
| Is it still current? | Whether the author or the labs still use it | Abandoned or saturated test |

## How do you evaluate with your own cases?

A practical process for the team:

1. **Gather real cases** from your product: as a starting point, 30 to 50, with inputs exactly as users send them.
2. **Define what a good answer is** with concrete criteria: correct, on-brand tone, the requested format, no invented data.
3. **Run the candidates with the same prompt**, the same temperature and the same context.
4. **Grade blind**: the evaluator should not know which model produced each answer.
5. **Also measure latency and cost** per answer, because the best answer is not always the most useful one. A [[small-vs-large-models|small model]] that meets your criteria can be the best decision.
6. **Repeat the test** when the model or the prompt changes. If you are also deciding between an API and your own model, see [[build-vs-buy-models|build vs. buy]].

## Example

Illustrative example: a team wants to choose between two models for a customer support assistant. Model A's provider publishes a higher score on a reasoning benchmark. The team builds 40 real customer queries and grades blind. Model B gets better marks on tone and on following the returns policy, and costs a third less per answer. The benchmark did not fail; it simply measured something else. The decision is made with the table of real cases and the cost.

## Common mistakes

- Choosing a model only by its place in a general ranking.
- Comparing published scores that used different configurations or dates.
- Quoting a figure from a 2021 paper as if it described current models.
- Ignoring sample size and small differences.
- Evaluating without blinding, so the model's brand influences the grades.
- Forgetting to repeat the evaluation when the provider updates the model.

## Sources

- [Measuring Massive Multitask Language Understanding (arXiv:2009.03300)](https://arxiv.org/abs/2009.03300)
- [Evaluating Large Language Models Trained on Code (arXiv:2107.03374)](https://arxiv.org/abs/2107.03374)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (arXiv:2310.06770)](https://arxiv.org/abs/2310.06770)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (arXiv:2403.04132)](https://arxiv.org/abs/2403.04132)
- [The Leaderboard Illusion (arXiv:2504.20879)](https://arxiv.org/abs/2504.20879)
- [Arena (AI platform), Wikipedia](https://en.wikipedia.org/wiki/Arena_(AI_platform))
- [SWE-Bench is dead, Latent Space](https://latent.space/p/swe-bench-dead)
- [Goodhart's law, Wikipedia](https://en.wikipedia.org/wiki/Goodhart%27s_law)
- [Benchmark (computing), Wikipedia](https://en.wikipedia.org/wiki/Benchmark_(computing))


---

Source: https://www.fernandoux.com/en/wiki/ai/ai-benchmarks/