Build Your Own Eval Set
Write questions based on the real things you actually need to do, have each candidate model answer them once with the same setup, and score against criteria defined in advance. This is closer to your actual experience than any leaderboard. After finishing this section, you will have a table: a few dozen questions, several models, one score per cell.
Before You Start
- A concrete task, such as "classify customer emails" or "turn meeting notes into minutes"
- Two to three candidate models, all of which you can call
- A spreadsheet program
Step 1: Pick Questions from Real Work
Don't make up questions yourself. Go through your past work records and pull out inputs that actually occurred: customer emails, meeting notes, error messages.
When picking questions, cover three categories:
| Type | Suggested Proportion | Why |
|---|---|---|
| Common ones | The majority | Determines everyday experience |
| Hard, edge-case ones | A portion | Separates the models |
| Ones that went wrong before | A few | The failures you care about most |
Start with 30 to 50 questions. Lesson 3 explains why too few questions can't reveal differences.
You should end up with: a list of questions, each being a piece of real input.
- I have picked at least 30 real questions
Step 2: Write Down "What Counts as Correct" in Advance
Write the criteria before looking at the answers. If you set criteria after seeing the answers, you will unconsciously favor the one you like.
Choose a scoring method based on the task type:
| Task | Scoring Method |
|---|---|
| Classification, extraction, tasks with reference answers | Compare against the reference answer: 1 if correct, 0 if wrong |
| Writing code | Run tests; passing means correct |
| Writing, summarizing, open-ended answers | List 3 to 5 checklist items, such as "no missed resolutions" or "no fabricated numbers"; score one point per item satisfied |
Quiz timeYou had two models each write a summary, read them, felt A was better, and then set the scoring criteria accordingly. What's the problem?
- Every question has pass criteria written in advance
Step 3: Run Every Model with the Same Setup
Every model must answer under the same conditions; otherwise you're not comparing the models:
- Same prompt, same system prompt
- Same temperature and other parameters
- Put each question in a fresh conversation so earlier content doesn't influence later answers
For open-ended tasks, you can let each model answer the same question several times. A model's answers will vary each time, and answering multiple times shows how stable it is.
You should end up with: at least one answer per question per model.
- All models used the same prompt and parameters
Step 4: Blind Scoring, Recorded in a Table
When scoring, hide the model names and look only at the answers. You can have a colleague shuffle the order for you.
The table looks like this:
| Question # | Type | Model A | Model B | Model C |
|---|---|---|---|---|
| 1 | Common | 1 | 1 | 0 |
| 2 | Hard | 0 | 1 | 0 |
| … | ||||
| Average |
Add two more columns you care about: how much each question cost and how long the wait was. The best model isn't necessarily the best value.
- I have a question × model score table
Checkpoint
- Questions come from real work and include common, hard, and previously failed ones
- Pass criteria were written before looking at the answers
- Every model used exactly the same prompt and parameters
- Model names were hidden during scoring
Didn't Work?
The scores are all about the same. The questions are too easy or too few. Add more hard questions and increase the count until differences show; see How Many Points Apart Counts as a Real Difference.
Scoring open-ended questions by hand is exhausting. Score a dozen or so by hand first, then try letting AI help with scoring; the next section covers its pitfalls.
Next Section
Manually scoring open-ended questions is exhausting—can AI serve as the judge? Using AI to Score AI.