Skip to content

Build Your Own Eval Set ​

Write questions based on the real things you actually need to do, have each candidate model answer them once with the same setup, and score against criteria defined in advance. This is closer to your actual experience than any leaderboard. After finishing this section, you will have a table: a few dozen questions, several models, one score per cell.

Before You Start ​

  • A concrete task, such as "classify customer emails" or "turn meeting notes into minutes"
  • Two to three candidate models, all of which you can call
  • A spreadsheet program

Step 1: Pick Questions from Real Work ​

Don't make up questions yourself. Go through your past work records and pull out inputs that actually occurred: customer emails, meeting notes, error messages.

When picking questions, cover three categories:

TypeSuggested ProportionWhy
Common onesThe majorityDetermines everyday experience
Hard, edge-case onesA portionSeparates the models
Ones that went wrong beforeA fewThe failures you care about most

Start with 30 to 50 questions. Lesson 3 explains why too few questions can't reveal differences.

You should end up with: a list of questions, each being a piece of real input.

  • I have picked at least 30 real questions

Step 2: Write Down "What Counts as Correct" in Advance ​

Write the criteria before looking at the answers. If you set criteria after seeing the answers, you will unconsciously favor the one you like.

Choose a scoring method based on the task type:

TaskScoring Method
Classification, extraction, tasks with reference answersCompare against the reference answer: 1 if correct, 0 if wrong
Writing codeRun tests; passing means correct
Writing, summarizing, open-ended answersList 3 to 5 checklist items, such as "no missed resolutions" or "no fabricated numbers"; score one point per item satisfied

Quiz timeYou had two models each write a summary, read them, felt A was better, and then set the scoring criteria accordingly. What's the problem?

  • Every question has pass criteria written in advance

Step 3: Run Every Model with the Same Setup ​

Every model must answer under the same conditions; otherwise you're not comparing the models:

  • Same prompt, same system prompt
  • Same temperature and other parameters
  • Put each question in a fresh conversation so earlier content doesn't influence later answers

For open-ended tasks, you can let each model answer the same question several times. A model's answers will vary each time, and answering multiple times shows how stable it is.

You should end up with: at least one answer per question per model.

  • All models used the same prompt and parameters

Step 4: Blind Scoring, Recorded in a Table ​

When scoring, hide the model names and look only at the answers. You can have a colleague shuffle the order for you.

The table looks like this:

Question #TypeModel AModel BModel C
1Common110
2Hard010
…
Average

Add two more columns you care about: how much each question cost and how long the wait was. The best model isn't necessarily the best value.

  • I have a question × model score table

Checkpoint ​

  • Questions come from real work and include common, hard, and previously failed ones
  • Pass criteria were written before looking at the answers
  • Every model used exactly the same prompt and parameters
  • Model names were hidden during scoring

Didn't Work? ​

The scores are all about the same. The questions are too easy or too few. Add more hard questions and increase the count until differences show; see How Many Points Apart Counts as a Real Difference.

Scoring open-ended questions by hand is exhausting. Score a dozen or so by hand first, then try letting AI help with scoring; the next section covers its pitfalls.

Next Section ​

Manually scoring open-ended questions is exhausting—can AI serve as the judge? Using AI to Score AI.

gaodong.aiAboutPrivacyContact

Independent tutorials, not affiliated with any product mentioned. Product names and trademarks belong to their owners. Check official pages for prices and policies.