Using AI to grade AI
Basically reliable, but it has several fixed biases that must be corrected before use. The 2023 paper "Judging LLM-as-a-Judge" (arXiv 2306.05685) compared AI judges with humans: when strong models like GPT-4 act as judges, their agreement rate with human preferences exceeds 80%, comparable to the agreement rate between humans. The data comes from 3 thousand expert votes and 3 thousand crowdsourced votes.
The same paper also listed its flaws.
What you can do after learning this
- Name the three biases of AI judges
- Add the corresponding fixes to your own evaluation
Bias one: position bias
The judge prefers answers in a certain position. Example from the paper: have GPT-4 compare answers from GPT-3.5 and Vicuna-13B. When GPT-3.5's answer is placed first, GPT-4 considers it more detailed and better; swap the positions of the two answers, and the judgment changes.
Fix: compare each pair of answers twice, swapping the order the second time. Only count it if both conclusions agree; if not, record it as a tie.
Quiz timeYou ask a judge to compare A and B: with A first, it picks A; after swapping, with B first, it picks B. How should you record this?
- My evaluation swaps the order and compares twice
Bias two: length bias
The judge prefers longer answers, even when the extra content is useless. This is the same thing as the stylistic preference in human voting on Arena.
Fix: state explicitly in the scoring criteria that "no extra points for length"; or limit the length of candidate answers in the prompt; when comparing, also record the word counts on both sides and check whether the longer one always wins.
Bias three: self-preference
The judge may favor answers written by itself or by models from the same family.
Fix: do not use any of the candidate models as the judge. If you really must, use two judges from different families and see whether their conclusions agree.
One more issue: limited reasoning ability
The paper points out that the judge can hardly judge correctly on problems it cannot solve itself, such as math and reasoning problems.
Fix: for problems with standard answers, compare directly against the answer instead of using an AI judge. Save the judge for open-ended tasks like writing and summarization.
- I know which problems should not be handed to an AI judge
How to put it into your evaluation
- First, manually score a dozen or so questions
- Have the AI judge score the same questions using the same criteria
- Compare the two sets of results; only let the judge score the remaining questions if the agreement rate is high enough
- When scoring: swap the order and compare twice, do not use candidate models as the judge, and do not hand it problems with standard answers
Checkpoint
- Strong models as judges agree with humans more than 80% of the time
- Three biases: position, length, self-preference
- Fixes: swap the order, no extra points for length, switch to a judge from a different family
- For problems with standard answers, compare directly
Common misconceptions
"A score of 9 from the judge means the answer is very good." Absolute scores are heavily influenced by the prompt. When comparing two models, "pairwise comparison of which is better" is more stable than "giving each a score".
"An agreement rate of 80% is high, so it can fully replace humans." The remaining 20% may be concentrated exactly on the hard problems you care about most. Do a manual spot check before using it.
Next section
After scoring, A averages 82 points and B averages 79 points. Is A really better? How many points apart counts as a real difference.