How to Choose a Large Model
The answer up front: leaderboards are for narrowing the field, your own tasks are for making the decision, and at the end you double-check that what you bought is really that model and that the service is stable.
The top-ranked model on a leaderboard isn't necessarily right for you. Drafting contracts, editing code, handling customer support—each needs different things; a gap of 10 points on the leaderboard may be invisible on your task. Conversely, the model you pick in your evaluation may not be what you actually get when calling through some intermediary API.
This course follows these three steps.
- 1View the leaderboard
- 2Use your own questions
- 3Check if it's the real model
There are many capable large models. Which one to start with? Click "Next".
What You'll Walk Away With
- Know how to read the LMArena leaderboard: what the scores, confidence intervals, and style control each mean
- How to read the Agent leaderboard: pick sub-indexes by use case, then check the Alignment Index
- Know what benchmarks like MMLU-Pro, GPQA, SWE-bench, and HLE each test, and which ones relate to your task
- An evaluation set built from your own tasks, plus a comparison table that shows whether a gap is real
- Several methods for checking whether an API is genuine, and what each method can and cannot detect
- A provider evaluation checklist: for the same model, pick the fastest and most stable one
The Four Stops of This Course
| Stop | Lesson | What Problem It Solves |
|---|---|---|
| Reading Leaderboards | How to Read the LMArena Leaderboard | Where the scores and rankings come from |
| How Long Does the Top Spot Last | How quickly the leaderboard changes | |
| Is the Leaderboard's #1 Really the Best | Known biases of leaderboards | |
| How to Read the Agent Leaderboard | Which index to check when choosing an AI for coding or running tasks | |
| Reading Benchmarks | What Common Benchmarks Test | MMLU-Pro, GPQA, SWE-bench, HLE |
| Evaluating Yourself | Build Your Own Evaluation Set | Write questions from your real tasks |
| Using AI to Grade AI | How reliable are judge models | |
| How Many Points Apart Counts as a Real Difference | Add error bars to your scores | |
| Checking for Genuine APIs | Could Your API Be Serving a Swapped Model | How model substitution happens |
| How to Verify an API Is Genuine | The limits of each detection method | |
| Same Model, Different Providers—What Differs | How to measure latency, throughput, and stability | |
| What's Going On When a Model Gets Dumber | Is the degradation real, and how to check |
What to Do at This Stop
- Think of one concrete thing you want a model to do, such as "turning meeting recordings into minutes"
- Write down two or three models you're currently using or want to try
- Open the next lesson and see where they rank on the leaderboard
What to Prepare Before Starting
No programming background is needed. Stop 3 uses spreadsheet software for the evaluation; if you want to automate it, any scripting language works, but it's not required.
Next Step
First, learn to read the leaderboard: How to Read the LMArena Leaderboard.