UUtah CS 6340 NLP | Fall 2026 | LLM-as-a-judge
Overview
LLM-as-a-judge replaces human evaluators with a capable language model to score or compare generated responses, making nuanced evaluation cheaper and scalable without requiring reference texts. The lecture emphasizes that every judge must be validated against independently collected human judgments, checked for biases such as position, verbosity, and self-preference, and tested on held-out data; rubric-based prompts, structured JSON outputs, and correlation measures help make evaluations more reliable.
Key takeaways
- An LLM judge can evaluate nuanced qualities such as tone, relevance, and helpfulness without a reference response, unlike BLEU, ROUGE, or BERTScore.
- Randomize response order in pairwise evaluations and analyze scores by response-length buckets to detect position and verbosity bias.
- Use a judge from a different model family when comparing systems, because a model may favor outputs produced by its own family.
- Validate a judge on the target task using independent human ratings, and compare its agreement with human-to-human agreement when multiple annotators are available.
- Tune prompts on development judgments but preserve held-out human-labeled examples for a final generalization check.
- Human preference judgments on GPT-3 summaries could conflict with BLEU and ROUGE rankings, showing why automatic metrics must be assessed against human judgments.
Chapters
- LLM-as-a-judge uses a language model to perform human-style evaluation, either through pairwise comparisons or ratings of individual responses.
- Unlike BLEU, ROUGE, and BERTScore, an LLM judge can evaluate outputs without a human-authored reference response.
- Validate each judge on a sample of the task data by comparing its decisions with human judgments before using it at scale.
- A typical Chatbot Arena-style prompt assigns an impartial-judge role and asks which response better follows the user's instructions.
- Prompts can specify criteria such as helpfulness, relevance, accuracy, depth, creativity, and level of detail.
- Pairwise prompts should address known position and length biases, then require a machine-readable verdict such as JSON.
- For separate criteria, independent API calls and rubric-based scores are often clearer than asking a model to weigh many factors at once.
- Position bias can make a judge favor the first response, so randomly shuffle which system's answer appears first.
- To test verbosity bias, bucket responses by length, for example in 50-word ranges, and check whether longer answers receive higher scores.
- A model may prefer its own outputs; use a judge from a different model family when comparing systems.
- A judge may lack the capability being assessed, such as spatial reasoning, so prompt adjustments cannot be assumed to fix every evaluation failure.
- Human evaluators can also have biases: a student notes that ACT writing scores favored longer responses regardless of their quality.
- Multiple annotators help estimate human-to-human agreement and reveal whether disagreement comes from unclear instructions or noisy judgments.
- Prompt changes can improve a judge, but compare each iteration against human judgments rather than assuming that an instruction such as 'ignore length' works.
- The lecture discusses a possible generation-versus-judgment capability gap and cautions that model capabilities and known biases change quickly.
- Run human and LLM judgments on a development set, potentially 100 to 200 instances, and compare pairwise choices or ratings.
- Use measures such as Spearman correlation or Cohen's kappa for judgments, and compare LLM-to-human agreement with human-to-human agreement when possible.
- Collect human judgments independently before showing annotators the LLM's decision, which could prime their responses.
- A single human annotator may be a budget constraint, but low agreement is a reason to investigate further or recruit additional annotators.
- Evaluate each rubric criterion independently and define what every point on a 1-to-5 scale means with examples for each level.
- Prefer binary decisions over rating scales when the task can be formulated clearly as a yes-or-no judgment.
- For factuality judgments, ask the model to identify a supporting source and return reasoning and decisions in separate JSON fields.
- Use human judgments to guide prompt iterations, but reserve held-out examples for the final test to check whether the tuned judge generalizes.
- A verified LLM judge enables evaluation across far more examples than human annotators can reasonably assess, increasing speed and statistical power.
- Judges do not eliminate human or model bias; the practical benefit is scaling evaluations after human validation, not replacing all human involvement.
- The lecture compares evaluation benchmarks such as SQuAD, GLUE, SuperGLUE, and MMLU, noting that known dataset shortcuts and flaws have not prevented research progress.
- Simulated users and agent-based judges raise further open questions about whether model-generated interactions support valid claims about people.
- Meta-evaluation tests evaluation metrics themselves by comparing metric scores with human scores on the same responses.
- Pearson correlation can measure how consistently high human ratings align with high metric scores; the lecture describes values above 0.5 as a useful research target.
- For pairwise categorical judgments, measures such as Cohen's kappa can assess agreement, while rank-based comparisons can use Spearman correlation.
- A correlation number is only meaningful in context, including the task, the dataset, and the agreement level among human annotators.
- A 2023 summarization example compared systems including GPT-3 on CNN and BBC articles, with human annotators selecting preferred and least-preferred summaries.
- Humans preferred GPT-3 summaries in the reported comparisons, while BLEU, ROUGE, and other automatic metrics sometimes ranked GPT-3 last.
- The example demonstrates that reference-overlap metrics can disagree with human preferences; it does not demonstrate that an LLM judge succeeded.
- The central recommendation is to verify every judge against human judgments before relying on its scores.
Transcript excerpt
0:00 Sorry about the recording from the or the lack of recording from the last time. It always happens and I was hoping it won't happen and this will be first semester with all recordings there, but seems like it's impossible. So, I'm recording now. I'm going to double-check. Yes, there it is. And yeah, I want to continue continue with our discussion about automated evaluation methods, right? Remember everything we were talking about last time was about how to we have a human evaluation as a one kind of a good way to judge the quality, but it's slow and it is expensive. And we have discussed two types of
0:51 evaluation metrics. One where we measure the overlap between unigrams or bigrams or n-grams. And embedding based approach where we compare the vector representations of human text and generated text. And the last class of automated metrics I will introduce is called LLM as a judge or maybe just LLM judge. Where basically the idea is super super simple. We use an LLM to do a job of a human. Okay. So, remember when I introduced human evaluation, I mentioned three different ways how we can go about human evaluation and I said we have can have for example pairwise comparison where we
1:38 take samples from a system and sample from let's say some human annotators or another system and we are tasking in human evaluation people to say which response is better without disclosing the source of the response. And the idea is I don't know I don't know in a sufficiently big sample, one system is better than the other or one system is on par to the human written response, we can say something about quality of that system and usually that system is ours. So here we basically do the same thing
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.