Skip to content

How to Read the LMArena Leaderboard ​

LMArena scores aren't exam results — they're relative strengths calculated from many people blindly voting on "which answer is better". So when reading the leaderboard, look at three things: the score, the score's margin of error, and whether this ranking has removed the bonus from "longer answers with flashy formatting".

What You'll Be Able to Do After This Lesson ​

  • Explain clearly where Arena scores come from
  • When two models have close scores, judge whether they're actually indistinguishable
  • Know why rankings change when "style control" is turned on or off

Where Scores Come From: Pairwise Blind Testing ​

Arena was originally called Chatbot Arena and is now called Arena (arena.ai). Here's how it works: you type a question, two anonymous models each give an answer, and you vote for the better one. Only after voting are the models revealed.

By the time the paper was published, the platform had accumulated over 240,000 votes. The authors verified that the crowdsourced questions were diverse enough and that voting results agreed closely with expert ratings.

With a large record of "who beat whom," a statistical model (the paper uses the Bradley-Terry model) converts it into one score per model. The score only has relative meaning: it says "this model is more often chosen than that one, for this audience and this kind of question."

Arena now has more than just a text leaderboard—there are also WebDev, Vision, Search, Document, text-to-image, image editing, text-to-video, and other arenas. If you want to pick a model for writing code, look at WebDev rather than the overall leaderboard; to pick an AI that can run commands on its own and get the job done, check the Agent leaderboard.

  • I can explain what Arena scores are based on

The Range Next to the Score Matters More ​

Every score is estimated from a limited number of votes, so it has error. Arena provides confidence intervals; in the official dataset, every model has rating (the score), rating_lower and rating_upper (the lower and upper bounds of the interval), plus vote_count (the vote count).

Judge it like this:

SituationHow to Read It
The two models' intervals don't overlapThe higher-ranked one is very likely genuinely more popular
The two models' intervals overlap a lotThey're indistinguishable; the ranking order may just be luck
A new model with few votesIts interval is usually wider, and its ranking will still change
Drag itMore votes, narrower intervalIllustrative: scores and interval widths are demo values
#1Model A
1450 ±10.0
#2Model B
1446 ±10.0

The two ranges overlap: you can't tell which is stronger

About 1000 votes. The scores differ by only 4 points,The intervals still overlap—"#1" and "#2" may just be luck.

Quiz timeModel A scores 1450, range 1440–1460; model B scores 1446, range 1436–1456. Which is stronger?

  • I look at the interval first, then the ranking

Style Control: Separating "Saying It Beautifully" from "Saying It Right" ​

When people vote, they're influenced by how an answer looks: longer answers with headings and lists are more likely to be chosen. The Arena team studied this specifically in a blog post dated 2024-8-28: after controlling for the effects of answer length and Markdown formatting, the rankings changed noticeably. In their own words: "Style indeed has a strong effect on models' performance in the leaderboard".

The purpose of style control is to separate the influence of style from content. With it on, what you see is closer to "is the content itself good"; with it off, what you see is "who do ordinary users actually prefer." Both are useful, depending on what you want:

  • If users will see answers directly, e.g. customer service or writing assistants: look at both leaderboards; the one with style control off is closer to real-world impressions
  • If you want the model to produce accurate content and you'll format it yourself: prioritize the style-controlled ranking

Quiz timeYou're building an internal tool where model output is parsed into tables by a program, so format doesn't matter. Which ranking should you trust more?

  • I know which kind of ranking I should look at

Checkpoint ​

  • Arena scores come from anonymous pairwise comparison votes, converted by a statistical model
  • Check whether confidence intervals overlap first, then the ranking order
  • To pick a specialized model, look at the corresponding arena
  • Style control changes rankings — choose based on your use case

Common Misconceptions ​

"A score 10 points higher means 10% better." The score is a scale of relative strength, not a percentage, and it can't be directly converted into "how much better on your task".

"There's a big difference between rank 1 and rank 2." When intervals overlap, rank 1 and rank 2 may be completely indistinguishable. Ranking forcibly cuts continuous scores into integers, which magnifies small gaps.

"If it's #1 on the overall leaderboard, it's also the best at writing code." The overall leaderboard mixes all kinds of questions. Look at the arena or category that matches your task.

Next Section ​

The top of the leaderboard changes hands often. How fast? The next section uses official historical data to calculate it: how long the #1 spot lasts.

gaodong.aiAboutPrivacyContact

Independent tutorials, not affiliated with any product mentioned. Product names and trademarks belong to their owners. Check official pages for prices and policies.