Skip to content

Is the top of the leaderboard really the best ​

Not necessarily. Ranking first means this model is most often chosen by Arena users on Arena's questions. Researchers have found that this result is mixed with several factors unrelated to "being better overall as a model." You can use the leaderboard, but apply a discount.

What you'll be able to do after this lesson ​

  • Name three known biases of leaderboards
  • Know which questions to ask when a new model suddenly tops the chart

Bias 1: Being able to pick the best score ​

The 2025 paper "The Leaderboard Illusion" (arXiv 2504.20879) found that Arena has undisclosed private testing. A few vendors can test multiple variants before public release, publish only the best one, and withdraw scores when needed.

An extreme example from the paper: before the release of Llama-4, Meta privately tested 27 variants.

Why does this inflate scores? An analogy: the same student takes a test 27 times and reports only the highest score. Every test has an element of luck, and the highest attempt is often above the student's true level.

Quiz timeA vendor privately submitted 10 nearly identical versions and only published the highest score. What happens to the published score?

  • I can explain why "picking the best score" inflates scores

Bias 2: Some models receive far more battle data ​

The same paper tallied whose models got drawn into battles more often: closed-source models were drawn at a higher rate and removed less often. It estimates that Google and OpenAI received 19.2% and 20.4% of all battle data respectively, while 83 open-weight models together received only about 29.7%.

Battle data is valuable: it records what users asked and what kinds of answers they preferred. Having more of it means a better understanding of this leaderboard's "taste."

Bias 3: You can train specifically for this leaderboard ​

The paper also found that training on Arena data clearly boosts scores: a small amount of extra data can bring up to a 112% relative gain on ArenaHard. The authors conclude that these factors lead to overfitting to Arena's specific distribution, not an improvement in overall model quality.

In other words, what the model may have learned is "pleasing Arena users," not getting better at your tasks.

  • I can name each of the three biases

One more: style ​

As covered in the previous lesson, longer and more elaborately formatted answers get chosen more often, which the Arena team itself confirmed—that's why they introduced style control. In rankings without style control enabled, this bonus is still there.

How to apply a discount when reading the leaderboard ​

What you seeQuestions to ask first
A new model tops the chart right after debutingHow many votes? How wide is the interval? Does it still rank first with style control enabled?
Two models have close scoresDo their intervals overlap?
A model ranks high on the leaderboard but feels mediocre to youDo your tasks resemble Arena's questions?

The most reliable approach is to test with your own tasks. That's the topic of the third stop.

Checkpoint ​

  • Privately testing multiple variants and keeping only the best score inflates scores
  • Battle data is unevenly distributed; big closed-source players get more of it
  • Training on Arena data can inflate Arena scores, but that doesn't mean overall improvement

Common misconceptions ​

"Since there are biases, leaderboards are useless." No. They are still the result of large-scale, real human blind testing, and are great for narrowing down candidates. Just don't take the gap between place 1 and place 3 at face value.

"If an open-source model ranks low, it must be bad." The numbers above show that open-weight models receive far less battle data, so they start from a different baseline.

Next section ​

Looking for an AI that writes code and runs tasks? There's also a dedicated leaderboard: How to read the Agent leaderboard for picking a workhorse AI.

gaodong.aiAboutPrivacyContact

Independent tutorials, not affiliated with any product mentioned. Product names and trademarks belong to their owners. Check official pages for prices and policies.