Skip to content

How many points apart counts as a real difference ​

It depends on how many questions there are. With 100 questions and an accuracy around 80%, the score itself fluctuates by roughly plus or minus 8 percentage points. If A scores 82 and B scores 79, that 3-point gap is very likely just luck.

Anthropic's 2024 article "Adding Error Bars to Evals" (arXiv 2411.00640) addresses exactly this: an eval is really an experiment. Your questions are just a small handful sampled from "all possible questions"—swap in a different handful and the score changes.

What you'll be able to do after this lesson ​

  • Compute the margin of error for your own eval scores with one formula
  • Judge whether the gap between two models is real
  • Know how to get more reliable conclusions with fewer questions

An old industry bad habit ​

The article points out that the industry convention is "the highest score is the best," bolding the top score without testing whether the gap is significant. Chatbot Arena popularized confidence intervals, but error bars are conspicuously missing from traditional Q&A evals.

Don't make this mistake in your own evals.

Step 1: Compute the margin of error ​

For the average score p of right/wrong questions (each worth 0 or 1), the standard error is:

Standard error = √( p × (1 − p) ÷ n )
95% margin of error ≈ plus or minus 1.96 × standard error

n is the number of questions. At an accuracy of 80%:

Number of questionsStandard error95% margin of error
505.7 percentage pointsroughly ±11
1004.0 percentage pointsroughly ±8
4002.0 percentage pointsroughly ±4
10001.3 percentage pointsroughly ±2.5

You have to multiply the number of questions by 4 to shrink the margin of error by only half.

Drag itHow big a gap counts as a real gapCalculated in real time using the lesson's formula

Too close to call

差 4 percentage points;50 questions, the margin of error for the difference between the two models is about ±16.2。 The gap falls within the margin of error—with a different set of questions, the ranking could flip.

Try it: keep 80% and 76% unchanged, and just drag the number of questions from 50 to 1000.

Quiz timeYou tested two models with 50 questions: A scored 80% correct, B scored 76%. Can you say A is better?

  • I've computed the margin of error for my own eval

Step 2: Shrink the error with paired comparison ​

The formula above treats the two models as if each took its own test. But in your eval, both models answer the same set of questions. In that case you can compare question by question: for each question, compute "A's score minus B's score" once.

The article discusses paired analysis specifically: the difficulty of a question affects both models at the same time, and subtracting per question cancels out that part, so the error is usually smaller than computing them separately.

How to do it: add a column "A − B" to your eval table, where each question is 1, 0, or −1, and look at the mean of this column and its error.

  • My eval table has a per-question difference column

Step 3: Answer open-ended questions multiple times ​

The model gives a different answer each time—this is another source of fluctuation. Among the variance-reduction methods the article lists is repeated sampling of the same question followed by averaging.

For open-ended questions, have each model answer each question about 3 times and take the average score; it's more stable than answering once.

Step 4: Watch out for grouped questions ​

If your questions come in groups—for example, 5 questions based on the same article—the right/wrong outcomes of those 5 questions often move together. The article calls this "clustered" questions, and in that case the true error is larger than what the formula above gives.

A simple fix: treat each group as one big question, and estimate the error based on the number of groups rather than the number of questions.

Checkpoints ​

  • Margin of error ≈ ±1.96 × √(p(1−p)/n)
  • 100 questions, roughly ±8 percentage points—a few points of difference tells you nothing
  • Use paired comparison for the same set of questions for a smaller error
  • Use repeated sampling for open-ended questions, and count by groups for grouped questions

Common misconceptions ​

"My eval has few questions, but they're carefully hand-picked, so it's reliable." Careful selection makes the questions better matched to the task, but it doesn't reduce random fluctuation. Fewer questions means larger error.

"Even a 1-point difference on a leaderboard is meaningful." First check whether it provides a margin of error. Arena gives confidence intervals; many benchmark tables don't.

Next section ​

Once you've picked a model, how do you confirm the API you're calling is really it? Could the API quietly swap in a different model.

gaodong.aiAboutPrivacyContact

Independent tutorials, not affiliated with any product mentioned. Product names and trademarks belong to their owners. Check official pages for prices and policies.