Skip to content

Choosing an AI that gets work done: how to read the Agent leaderboard ​

Xiao Zhou wanted an AI to help him modify code, run scripts, and organize a pile of files. He opened the text leaderboard and picked the number one. When chatting, it answered eloquently, but once it started doing actual work, he had to correct it again and again.

The problem was that he read the wrong leaderboard. The text leaderboard compares "which answer is more pleasing," while what he wanted was "which one can get the job done." Arena has a dedicated leaderboard for the latter: the Agent leaderboard. Take a guess first:

Quiz timeIn the 2026-11-18 edition, Gemini 4 Carbon (High), first on the text leaderboard, ranked where on the Agent overall leaderboard?

The Agent leaderboard doesn't rely on votes; instead, it looks at whether the work got done, how many corrections were needed, and whether commands errored out when real users had AI do work. On 2026-11-18 it was revamped: the overall leaderboard is now called the Agent Arena Index, six sub-indexes were added, and the scores switched to a new scale.

What you'll be able to do after learning ​

  • State the fundamental difference in scoring between the Agent leaderboard and the text leaderboard
  • Pick the right sub-index to look at based on your use case
  • Read the revamped anchored scores and know why the Alignment Index should be viewed separately

How it differs from the text leaderboard ​

You already know how the text leaderboard works: two anonymous models each answer once, and users vote. The Agent leaderboard (launched on 2026-06-04) is completely different:

  • No pairwise voting. Users do real work in Agent Mode on arena.ai/agent, such as writing software or doing financial analysis. Each session is randomly assigned a model.
  • Signals from the process are observed. Whether the user said "done," whether they praised or complained, how many corrections were made, and whether the commands run by the model errored out — all of this is recorded.
  • Because models are randomly assigned, the officials treat it as a randomized controlled trial, calling it "causal tracing," and compute the "net lift" each model brings.

So the scores of the two leaderboards cannot be compared side by side at all. Ranking first on the text leaderboard means "answers are most often chosen," while ranking first on the Agent leaderboard means "best overall performance when doing work."

  • I can state that the Agent leaderboard's scores don't come from voting

What the four capability signals each measure ​

The Agent Arena Index is derived from the average of four signals:

SignalHow it's measuredWhat a high score means
Task Completion (formerly Confirmed Success)At the end of each turn, a popup asks the user "Did this complete your task?"; the proportion of "yes" answers is computedThe user confirms the work got done
Praise vs ComplaintUsers' praise and complaints are labeled separately; a task counts as a success if praise outnumbers complaintsUsers are more satisfied
SteerabilityLooks at how many corrections the user has to make; if things go well after one correction it counts as "low," and needing another correction counts as "high"You don't have to keep pulling it back on track
Bash Reliability (formerly Bash Recovery)For each command call, errors caused by the model itself (such as mistyped parameters or bugs in its own scripts) count as failures; things like network outages don't countCommands rarely fail

One signal was retired in the revamp: Tool Hallucination (calling tools that don't exist at all), on the grounds that frontier models rarely make this mistake anymore.

  • I can state what a high Steerability score means

Picking a sub-index by use case ​

Beyond the overall leaderboard, the revamp added six sub-indexes. Find yours by use case:

Look it upWhat I want to do with AI, and which index to check

Click one to see.

New models often appear on the Agent leaderboard first. For example, GPT-6.2 Astra (Max) and GPT-6.2 Sol (Max) entered the Agent Arena Index on 11-18 and the Coding Index on 11-19, and as of 11-19 they still hadn't appeared on the text leaderboard.

  • I found the index that matches my use case

How to read the revamped scores ​

The old scores had a flaw: they measured "how much better than the average level of all models on the leaderboard." When a very strong new model arrived, the average rose and older models' scores would drop, even if they hadn't changed at all themselves.

The revamped score is a linear transformation of the old score: new score = 1000 × old score + Δ.

  • Multiplying by 1000 makes it readable: 1 point equals one thousandth of the standard deviation of that signal.
  • Δ is called the "anchor." Each time a leaderboard is released, Δ is recomputed so that stable models that appear on the leaderboard in both periods and have many sessions keep the same average score.
  • The starting point is set at 2026-7, when the models' average score was about 1000.
  • Δ is the same for all models, so the rankings are exactly the same as under the old algorithm; it's just that scores can now be compared across periods.

On the webpage, each index can be expanded to see the individual signals that make it up, and you can also switch to the raw signal scores, i.e., the success rate of each signal.

Quiz timeLast month a model scored 1050 on the Agent benchmark; this month a very strong new model arrived, but it didn't change at all. Under the revised algorithm, what happens to its score?

  • I know the Agent scores and the text leaderboard scores are not on the same scale

Hands-on: take a look at the latest period's data ​

Why: once you dig through the data yourself, you'll see that scores, intervals, and costs appear together.

Arena puts every period's leaderboard into the public dataset lmarena-ai/leaderboard-dataset on Hugging Face; the Agent overall leaderboard is in the agent subset. Just open the following address in your browser:

text
https://datasets-server.huggingface.co/rows?dataset=lmarena-ai/leaderboard-dataset&config=agent&split=latest&offset=0&length=8

What you should see: a piece of JSON, with one model per row in rows, carrying rank, score, score_ci_lower, score_ci_upper (interval), session_count (session count), and cost_per_task_p50 (median cost per task). The top five for the 2026-11-18 period:

RankModelScore in the dataset (interval)Session countMedian cost per task
1Claude Fable 5.2 (Max)0.274(0.245–0.304)12,3309.16
2Claude Opus 5.5 (Max)0.186(0.166–0.207)14,9205.89
3Gemini 4 Neon (High)0.184(0.159–0.209)5,6962.60
4Gemini 4 Carbon (High)0.161(0.154–0.168)64,1141.33
5Claude Sonnet 5.5 (Max)0.156(0.127–0.185)8,9123.40

Two things to note:

  1. The score in the dataset is still on the old scale (not multiplied by 1000 or with Δ added), so it differs from the numbers on the webpage, but the ranking is the same.
  2. The intervals for ranks 2 and 3 overlap almost completely, so they can't be told apart; the lower edge of rank 1's interval is also higher than the upper edge of rank 2's. Meanwhile, rank 1's median cost per task is about 7 times that of rank 4. Whether it's worth spending that extra money depends on how hard your work is.
  • I opened the data myself and found the top-ranked model's interval and cost

Alignment Index: why it should be viewed separately ​

Being capable doesn't mean following the rules. The Alignment Index is a separate set of indexes alongside the overall leaderboard (launched on 2026-10-08, with a maximum score of 100, higher is better), looking only at three kinds of failures:

SignalSituations that get flagged
Unauthorized ActionDoing things the user didn't ask for, or exceeding permissions, such as deleting or "cleaning up" the user's files without permission
False Attributionattributing a statement, decision, or approval to the user when the evidence the user provided does not support it
Deceptive Completionexplicitly saying "it's done" or "it's been checked" when the evidence at the time shows otherwise

At launch it compared 27 models and 90,000 real conversations, with OpenAI's models taking the top five spots. 2026-11-17 the judging method changed again: every conversation (including tool calls) is reviewed in full, LLM judges from multiple labs judge separately, and a majority must agree for it to count as a failure. The result was that measured failure rates rose across the board, and officials said the rankings stayed broadly the same.

If you are going to let AI touch real files or real accounts, this index is just as important as the capability index.

  • I can name three types of failures in the Alignment Index

Checkpoint ​

  • I can explain that the Agent leaderboard's scores come from real work processes, not pairwise voting
  • I can pick one of the sub-indices like Coding, Work, or Chat based on my use case
  • I know that the revised score is "1000 × old score + Δ", that rankings are unchanged, and that it cannot be compared with text leaderboard scores
  • I look at both the interval and the cost per task, not just the rank
  • I know that when letting AI touch real files, I also need to check the Alignment Index

Common misconceptions ​

"Number one on the text leaderboard means it's also the strongest at getting work done." The two leaderboards measure two different abilities. 2026-11-18 in that edition, Gemini 4 Carbon (High), first on the text leaderboard, ranked fourth on the overall Agent leaderboard.

"The scores changed, so the rankings changed too." The 11-18 revision only changed the scale, and officials explicitly stated it does not affect rankings. What actually changes rankings is new models entering the list and new data.

"A high capability score means it's safe to hand things over." The capability index says nothing about whether it will delete files without permission or falsely report completion—those are covered by the Alignment Index.

Next section ​

Beyond leaderboards, model releases also list a row of benchmark scores. What do they measure? What common benchmarks measure.

gaodong.aiAboutPrivacyContact

Independent tutorials, not affiliated with any product mentioned. Product names and trademarks belong to their owners. Check official pages for prices and policies.