Skip to content

What Common Benchmarks Test ​

A benchmark is a fixed set of exam questions. MMLU-Pro, GPQA, SWE-bench, and HLE listed at model releases each construct their questions in completely different ways. Once you understand what they test, you'll know which scores matter to you and which you can skip.

What You'll Be Able to Do After This ​

  • Explain what each of the four common benchmarks tests
  • Pick the score worth looking at based on your own tasks

Four Benchmarks in One Table ​

BenchmarkWhat It TestsQuestionsCharacteristics
MMLU-ProMulti-disciplinary knowledge and reasoningMultiple-choice questions across many tasksProposed by the University of Waterloo, harder and more robust than the original MMLU
GPQAHard problems in biology, physics, and chemistry448 multiple-choice questions written by domain expertsAnswers can't be found online
SWE-benchFixing issues in real code repositories2,294 problems from 12 popular Python repositoriesRequires modifying code; tests are run to judge correctness
HLEHard problems at the frontier of human knowledge2,500 questions across dozens of subjects, including image-based questionsAnswers are definitive but can't be searched for

MMLU-Pro: Knowledge Plus Reasoning ​

MMLU was once the most common knowledge benchmark, but popular models now exceed 90% accuracy on it, so it no longer separates them. MMLU-Pro is a harder, more robust version proposed by the University of Waterloo (arXiv 2406.01574).

The paper analyzed why models get questions wrong: reasoning errors account for 39%, missing specific knowledge for 35%, and calculation errors for 12%. So it's not just about memorization—reasoning carries a lot of weight.

Who should look at this: Your task is general Q&A or a knowledge-based assistant.

GPQA: Science Problems You Can't Look Up Online ​

GPQA has 448 multiple-choice questions in biology, physics, and chemistry, all written by domain experts. How hard is it:

  • PhDs (including students) in the relevant fields scored 65% accuracy, or 74% after excluding obvious mistakes they admitted to afterwards
  • High performers from other fields, spending over 30 minutes per question on average with unrestricted internet access, still only got 34%
  • The strongest baseline at the time of the paper's publication, GPT-4, was 39%

The authors call these questions "Google-proof": search doesn't help.

Who should look at this: Your task requires professional science reasoning.

Quiz timeIn GPQA, non-experts could freely search the web, yet only scored 34%. What does this tell you?

SWE-bench: Fixing Real Bugs ​

SWE-bench has 2,294 software engineering problems, drawn from real GitHub issues and the corresponding fix PRs of 12 popular Python repositories. The model must read the codebase, modify code, and resolve the issue. The paper was published at ICLR 2024, where the best model at the time, Claude 2, solved only 1.96%.

Its biggest difference from the first two: It's not multiple choice—it's real work. Correctness is judged by running the repository's own tests.

SWE-bench Verified on release sheets, which OpenAI no longer reports ​

What you see more often in model release sheets is SWE-bench Verified: a subset of 500 problems cleaned up from the original version through a collaboration between OpenAI and the SWE-bench team. It was once the "north star" for measuring coding ability, but frontier models generally report around 80% on it, after which the numbers barely move.

2026-02-23, Mia Glaese and Olivia Watkins, OpenAI researchers who worked on Verified, announced in a Latent Space interview: OpenAI will no longer report SWE-bench Verified and is pushing SWE-bench Pro instead. There are two reasons:

  1. Many of the remaining problems simply cannot be solved. OpenAI had more engineers re-review 138 problematic problems and found that over 60% were unsolvable: 49 had tests defined too narrowly, so even correct fixes would be judged wrong; 26 had tests defined too broadly, requiring functionality never mentioned in the problem statement.
  2. The problems have been "memorized." The problems come from open-source repositories, and those repositories were already commonly used to train models. OpenAI found that frontier models from various labs, given just the problem number, could reproduce the original fix patch verbatim.

So a difference of a point or two on Verified no longer tells you who is better at writing code. The swebench.com homepage still defaults to showing the Verified leaderboard by default—keep this context in mind when you see it.

Who should read this: You want to use models to write code, fix bugs, or build coding assistants. When looking at vendor release sheets, first check whether they list the original version, Verified, or Pro, and whether the evaluation setups are the same.

HLE: Problems Written for the Strongest Models ​

Humanity's Last Exam (HLE) has 2,500 questions covering dozens of subjects, including both multiple-choice and short-answer questions, with image-based questions included. Every question has a definitive answer, but it can't be obtained through quick searches.

Why it was created: on popular benchmarks, models already exceed 90%, so differences can't be measured. On HLE, the latest models' accuracy and calibration (judging how confident they are in their answers) are still very low.

Who should look at this: You want to know a model's ceiling on the hardest problems. For everyday tasks, you can skip this score.

  • I know what each of the four benchmarks tests

How to Use Benchmarks ​

Quiz timeYou want to pick a model as a code review assistant. Which benchmark matters most?

Four principles:

  1. Only look at benchmarks close to your task. A high score on hard science problems doesn't mean it writes good marketing copy
  2. Benchmarks saturate. When everyone is above 90%, a few points of difference doesn't mean much
  3. Benchmarks can be contaminated. Problems have leaked into training data, and models may have "memorized" the answers—SWE-bench Verified is an example
  4. Benchmarks are problems written by others. In the end you must test with your own problems—see the next stop
  • I understand why high scores on SWE-bench Verified should be taken with a grain of salt

Checkpoint ​

  • MMLU-Pro: knowledge and reasoning, with reasoning accounting for 39% of errors
  • GPQA: 448 expert-written science questions, PhDs 65%, non-experts with internet access only 34%
  • SWE-bench: 2,294 real issues, requiring code changes to pass tests; its 500-problem subset Verified has been deprecated by OpenAI (bad tests + contamination)
  • HLE: 2,500 frontier problems, model scores are still low

Common Misconceptions ​

"The more benchmarks and the higher the scores, the better the model." Benchmarks test very different things, so adding them up is meaningless. Pick the ones relevant to you.

"Benchmarks are public, so scores must be comparable." Different vendors may use different prompts, sampling counts, and tool-use permissions. When comparing, check whether the settings are the same.

Next Section ​

The most reliable benchmark is the one you write yourself: Build Your Own Evaluation Set.

gaodong.aiAboutPrivacyContact

Independent tutorials, not affiliated with any product mentioned. Product names and trademarks belong to their owners. Check official pages for prices and policies.