Save this video free

UUtah CS 6340 NLP | Fall 2026 | LLM-as-a-judge

UofU Data Science · 49:05 · Watch on YouTube

Part of University of Utah CS 6340. See all 19 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

LLM-as-a-judge replaces human evaluators with a capable language model to score or compare generated responses, making nuanced evaluation cheaper and scalable without requiring reference texts. The lecture emphasizes that every judge must be validated against independently collected human judgments, checked for biases such as position, verbosity, and self-preference, and tested on held-out data; rubric-based prompts, structured JSON outputs, and correlation measures help make evaluations more reliable.

Key takeaways

Chapters

0:00 LLM Judges Replace Reference-Based Text Metrics
5:50 Designing Pairwise and Rubric-Based Judge Prompts
10:20 Detecting Position, Verbosity, and Self-Preference Biases
17:45 Human Noise, Model Capabilities, and Prompt Iteration
27:25 Validating Judges Against Independent Human Ratings
30:45 Building Rubrics and Keeping Judge Tuning Honest
35:00 Why LLM Judges Scale Despite Messy Evaluation
42:30 Meta-Evaluation: Measuring Whether Metrics Match Human Judgments
45:35 GPT-3 Summary Preferences Expose Overlap-Metric Limits

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:00 Sorry about the recording from the or the lack of recording from the last time. It always happens and I was hoping it won't happen and this will be first semester with all recordings there, but seems like it's impossible. So, I'm recording now. I'm going to double-check. Yes, there it is. And yeah, I want to continue continue with our discussion about automated evaluation methods, right? Remember everything we were talking about last time was about how to we have a human evaluation as a one kind of a good way to judge the quality, but it's slow and it is expensive. And we have discussed two types of

0:51 evaluation metrics. One where we measure the overlap between unigrams or bigrams or n-grams. And embedding based approach where we compare the vector representations of human text and generated text. And the last class of automated metrics I will introduce is called LLM as a judge or maybe just LLM judge. Where basically the idea is super super simple. We use an LLM to do a job of a human. Okay. So, remember when I introduced human evaluation, I mentioned three different ways how we can go about human evaluation and I said we have can have for example pairwise comparison where we

1:38 take samples from a system and sample from let's say some human annotators or another system and we are tasking in human evaluation people to say which response is better without disclosing the source of the response. And the idea is I don't know I don't know in a sufficiently big sample, one system is better than the other or one system is on par to the human written response, we can say something about quality of that system and usually that system is ours. So here we basically do the same thing

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.