UUtah CS 6340 NLP | Fall 2026 | Intro to QA & Retrieval methods

Overview

The lecture distinguishes information-seeking open-domain question answering from reading-comprehension benchmarks, then compares retrieval-reader, retrieval-generator (RAG), and parametric-only answering. It explains sparse retrieval with TF-IDF, BM25, and inverted indexes, followed by dense retrieval with jointly encoded query-document pairs, dual-encoder DPR, ColBERT, and relevance training using hard negatives and softmax negative log-likelihood.

Key takeaways

Open-domain QA retrieves evidence for a user-supplied question, while reading-comprehension datasets such as DROP provide a passage to test a targeted capability such as numerical reasoning.
BM25 is a strong, efficient sparse-retrieval baseline, but exact lexical matching can miss semantically relevant paraphrases with little token overlap.
Joint BERT cross-encoding can model question-document interactions, but scoring one million documents requires one million forward passes per query.
DPR precomputes document vectors offline and scores them with dot products, trading some joint-interaction accuracy for much faster retrieval.
ColBERT retains token-level representations and sums each query token’s best document-token match, offering greater expressivity than DPR’s single-vector score.
Hard negatives sampled from BM25’s top-ranked nonrelevant results make dense-retriever training more challenging; softmax over document similarities and negative log-likelihood train the relevant document to rank first.

Chapters

0:00 Course Logistics and Discussion of Language-Model Research

The class returned from fall break and discussed grading the PyTorch practicum, which requires reviewing student code.
Bonus-point plans were still undecided, with an update expected the following week.
A conference on language modeling raised a perceived gap between technology-industry discussions and concerns among the general public.
Continual learning was identified as an important challenge for agents that should retain user knowledge across months or years.

5:00 From Tokenization to Transformer-Based Next-Token Prediction

Prompts become token IDs, which retrieve token embeddings from an embedding matrix.
Transformer self-attention uses surrounding tokens to contextualize ambiguous token representations.
Residual connections and layer normalization support stable deep-network training; a vocabulary projection and softmax produce next-token probabilities.
Pretraining learns next-token prediction from large text corpora, while supervised fine-tuning adapts pretrained weights to labeled tasks.

12:00 Why Modern LLMs Require Post-Training and Efficiency Advances

The foundational transformer pipeline resembles 2018 to 2020 models, which had shorter context windows such as 512 tokens.
Post-training adds capabilities for instruction following, multi-turn interaction, reasoning, and tool use beyond next-token pretraining.
Mid-training and efficiency improvements are also part of the broader modern-LLM development picture.
The course plans to cover NLP applications and evaluation before returning to post-training, efficiency, and AI's societal impact.

13:30 Open-Domain QA and Retrieval-Then-Answer Pipelines

Open-domain QA assumes a user provides a question without supplying a passage or answer choices.
A retrieval system can find relevant corpus documents, then pass them with the question to an answerer.
The lecture focuses first on retrieval foundations, with answer generation from retrieved documents scheduled for the next class.

15:00 Information-Seeking QA Versus Reading Comprehension

Information-seeking QA reflects real user questions, which are often underspecified, keyword-like, and unsupported by explicit context.
Reading-comprehension benchmarks provide a passage and ask questions designed to test language or reasoning abilities.
Numerical reasoning, negation, and other targeted capabilities can be isolated through passage-question examples.
Information-seeking QA often aims to find factual information on the web, while reading comprehension often serves NLP research evaluation.

17:30 Natural Questions, DROP, and Negation Benchmarks

Natural Questions includes questions such as when hops are added during brewing, with short and longer answers grounded in sources such as Wikipedia.
DROP tests numerical reasoning: if each of two teams scores one second-half touchdown, the answer is two touchdowns.
A negation-focused example asks whether a car survived a fire; understanding “with the exception of” shows that the car was not destroyed.
A Google search result that once repeated seizure advice users should not follow illustrates the real risks of missing negation.

21:00 Question-Answering Format Is Not the Same as a QA Task

Many tasks can use a question-answering format, including sentiment classification phrased as “Is this review positive or negative?”
A fixed prompt repeated for every example can be replaced by a label or content-free placeholder, so it is not necessarily a true QA task.
Open-ended questions vary with the information sought, as in asking when hops are added or posing different questions about a passage.
A shared QA format can help build multitask systems or prompt language models, even when the underlying tasks are not genuine question answering.

24:00 Question Styles and Answer Types in QA Datasets

Natural grammatical questions and search-engine-style queries are common question forms, while masked cloze prompts reflect older masked-language-model setups.
Cloze questions such as “Einstein was born in [MASK]” and story-completion formats such as SWAG are now less representative of modern generative systems.
Extractive answers are substrings of source documents, enabling precise evaluation but limiting acceptable answers to the chosen span.
Multiple-choice answers simplify scoring but need strong distractors; categorical answers use fixed labels, while free-form answers permit varied wording and longer responses.

33:24 Retrieval-Reader, RAG, and Parametric-Only Answering

A retrieval-reader system finds documents, then predicts the start and end positions of an answer span within them.
A retrieval-generator system, commonly called retrieval-augmented generation (RAG), supplies retrieved documents and a question to an LLM.
Parametric-only answering relies on facts stored in model weights and can fail when the requested knowledge is absent, producing hallucinations.
Extractive answers can be preferable in higher-risk settings where restricting outputs to source text and showing evidence improves accountability.

37:26 Tool Use, External APIs, and the Scope of Retrieval

Live weather or current-information requests may use external APIs rather than knowledge encoded in model weights.
Calling a browser or another API is generally described as tool use, while retrieval usually means selecting relevant items from a data store.
A web agent can use a browser tool to find information and then perform a more classical information-retrieval step.
Agentic tool use is a separate topic from the retrieval foundations covered in this lecture.

43:00 TF-IDF, BM25, and Inverted Indexes for Sparse Retrieval

TF-IDF combines term frequency within a document with inverse document frequency, reducing the influence of common words such as “the.”
A query-document relevance score can sum the query terms’ TF-IDF contributions for each document, then rank documents by score.
BM25 extends TF-IDF with parameters that control term-frequency saturation and document-length normalization; it is a strong retrieval baseline.
An inverted index stores a term dictionary and postings lists containing document IDs and term counts, enabling efficient BM25 scoring without scanning every document.

53:00 Sparse Retrieval’s Exact-Match Limitation

BM25 and other sparse methods depend on lexical overlap between query and document tokens.
A relevant paraphrase with little or no exact token overlap can receive a low BM25 score.
Dense retrieval addresses this limitation by representing queries and documents as vectors that can capture semantic similarity.
After the short break, the lecture shifts from sparse matching to neural retrieval.

57:24 BERT Cross-Encoders Score Query-Document Pairs Jointly

A BERT-style encoder receives a sequence containing [CLS], the question, a separator, the document, and a final separator.
The final [CLS] representation can be projected to relevant and irrelevant logits; the relevant-class score serves as a query-document relevance estimate.
Joint encoding lets attention model interactions between question and document tokens, but scoring a million-document corpus requires a million forward passes per query.
Long inputs can exceed the model context window; sliding windows can score document sections and combine results, for example by taking the maximum.

1:06:40 Dense Passage Retrieval Uses Offline Document Encodings

Dense Passage Retrieval (DPR) uses separate encoders for questions and documents, producing independent vector representations.
A dot product between query and document vectors gives a relevance score without jointly encoding each pair.
Document vectors can be computed offline before deployment, leaving only query encoding and vector comparisons for incoming requests.
DPR gains substantial speed over joint encoding but can lose accuracy because query and document representations are not trained together as a pair; FAISS can accelerate vector search.

1:09:30 ColBERT Matches Each Query Token to Its Best Document Token

ColBERT separately encodes query and document tokens instead of compressing each sequence into a single [CLS] vector.
For every query token, it finds the document token with the highest dot-product similarity.
Summing these per-query-token maxima produces a more expressive relevance score than DPR’s single vector dot product.
Token-level matching increases computation but can recover useful interaction detail lost by separate single-vector encodings.

1:14:00 Training Dense Retrievers with Hard Negatives and NLL

Labeled query-relevant-document pairs can train encoders to bring relevant vectors closer and push irrelevant vectors farther away.
BM25 rankings provide hard negatives: highly ranked documents that are not labeled relevant are more challenging than randomly sampled documents.
For each query, place the relevant document’s similarity first and similarities for negative documents after it, then apply softmax.
Minimizing negative log-likelihood of the first, relevant-document probability trains the retriever; BM25, DPR, and ColBERT results can then feed an answerer.

Transcript excerpt

0:00 Okay, it's 1:25, so we can get started. Um, on Monday I was so focused on getting the practicum set up that I didn't even welcome you back from your fall break. I'm sorry. Hope you had a nice week and did something fun. Um, how did the practicum go? I feel felt like there is a better vibe this time around. I feel I see if you not. Uh, yeah, I guess I guess seeing it once and how it works uh potentially helped or maybe it's easier to work with PyTorch than just Python. I I don't know. I would like to hear it from uh you. We'll try I'm going to try to grade as much as fast as we can. This is a little bit more involved to grade because um we do look into into your code. Uh, so yeah, I will keep you posted um

0:50 when when uh we are close to that uh being done. Um, I did say that around, you know, fall break I will start to think about bonuses and how to integrate them. Um, I have not yet decided, so I don't have information right now, but it is one of my priorities this week to figure that out. So by next week I I'm confident I'll share that information, too. So you can kind of start thinking about where where you are with the course and uh like what else uh where else you could uh earn points. Um, are there any questions before we jump into the content new content? Yes, please. >> Just out of curiosity, how did like the conference you went to go? >> How the conference went? Oh, yeah. Uh,

1:37 so I went to the conference on language modeling column and uh it was an

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.