UUtah CS 6340 NLP | Fall 2026 | Retrieval (cont.) & Answer extraction
Overview
The lecture connects large-scale retrieval to answer generation and extraction, reviewing BM25 and dense retrieval before explaining approximate nearest-neighbor search with Faiss, retrieval metrics such as F1 and mean reciprocal rank, and design choices for retrieval-augmented systems. It then derives BERT-style answer extraction using learned start and end vectors, negative log-likelihood training, no-answer and long-passage handling, and token-level F1, before contrasting extractive answers with retrieval-augmented generation (RAG).
Key takeaways
- Use BM25 as an interpretable lexical-overlap baseline and compare it with a dense retriever; consider ColBERT when token-level matching is needed for stronger expressivity.
- At billion-document scale, Faiss IVF can reduce search work by training 4,096 centroids and scanning only selected nearby clusters, such as 16, before ranking candidate documents.
- Precision, recall, and F1 assess retrieval coverage and relevance, while MRR focuses only on the rank of the first relevant result; no single metric captures every retrieval behavior.
- BERT-style extraction learns separate start and end vectors, applies softmax over passage-token logits, and returns a span trained with negative log-likelihood.
- Answer extraction should explicitly train a CLS-based no-answer option and split passages beyond BERT's 512-token limit into windows that can contain complete answer spans.
- RAG places retrieved evidence in a decoder-only LLM prompt for next-token generation, while extractive QA is preferable when high-risk applications cannot tolerate answers outside the retrieved text.
Chapters
- Quiz bonus rules and formulas are posted in a Canvas announcement, and previous quiz grading is delayed while Jonathan attends EMNLP.
- The next quiz covers lectures after Practical 2, including question answering and text-generation evaluation, while retaining a four-point weight.
- Practical assignments aim to connect practice work with assessments; future tasks may avoid the first practical's unusually creative setup.
- Open-ended question answering retrieves evidence from a data store, while reading comprehension supplies a passage and tests reasoning over it.
- BM25 is an interpretable sparse baseline based on lexical overlap, but it can miss paraphrases that lack shared words.
- Dense Passage Retrieval uses separately parameterized query and document encoders, then scores their vectors with a dot product.
- ColBERT adds token-level similarity and aggregation for more expressive matching; BM25 and one dense retriever are sensible baselines before trying more complex methods.
- Dense retrieval over millions or billions of documents is a maximum inner product search (MIPS) problem.
- Exact search requires comparing each query vector with every stored document vector, creating substantial latency and computation.
- Approximate nearest-neighbor (ANN) search accepts some accuracy loss to achieve large speed gains.
- Faiss, originally developed at Facebook AI and now associated with Meta, is a library implementing multiple ANN algorithms, not the name of the problem itself.
- An illustrative ANN method partitions document vectors into K-means clusters, such as 4,000 clusters for a much larger collection.
- At query time, the system finds nearby cluster centroids and scores documents only in those clusters rather than scanning the whole corpus.
- The dense index can be understood as centroids associated with lists of document vectors, analogous in purpose to BM25's token-to-document inverted index.
- Searching several nearby centroids instead of just one is a tunable trade-off between the number of candidates scored and retrieval quality.
- The Faiss example creates an IVF index for 768-dimensional vectors and configures inner product as the similarity measure.
- Training the index learns 4,096 clusters; adding vectors assigns each document vector and ID to a centroid's list.
- At search time, the example scans the 16 nearest clusters, scores their document vectors, and returns the top 10 results.
- Faiss supports other ANN configurations, including quantization to store vectors at lower precision and reduce memory use.
- Precision measures the fraction of retrieved documents that are relevant; recall measures the fraction of known relevant documents that were retrieved.
- F1 combines precision and recall, but does not indicate whether relevant documents appear in the correct rank order.
- Mean reciprocal rank (MRR) averages the reciprocal rank of the first relevant result, giving 1 when that result is always first and 0 when no relevant result is retrieved.
- MRR ignores the ranks of additional relevant results, so retrieval studies may also report measures such as recall at K.
- Wikipedia articles can be split into roughly 100-word text chunks so each vector represents a focused passage rather than an entire article.
- Retrieval targets can also be entity mentions, tool outputs, or summaries of past agent interactions stored as long-term memories.
- A common RAG design inserts retrieved content directly into an LLM prompt; earlier approaches also connected stored items through encoder-decoder cross-attention.
- Agents can retrieve again after generating a fixed number of tokens or selectively when uncertainty proxies suggest the model needs more evidence.
- When vectors are normalized, dot product and cosine similarity give equivalent rankings; cosine similarity is a useful default when vector magnitudes vary.
- Faiss IVF search checks query similarity against all centroids, then searches documents in the closest selected clusters, such as 16 of 4,096.
- The number of clusters searched and the number of returned results are separate parameters that control candidate coverage and search cost.
- Retrieval evaluation extends beyond MRR and F1, so research comparisons should report the metrics used by relevant prior work.
- The extraction model receives a question and a retrieved passage in a BERT-style sequence with separator tokens.
- Instead of using the CLS representation for the whole input, the model uses the final hidden representation of each passage token.
- Two learned vectors, S and E, score passage-token representations as candidate answer starts and ends.
- The selected start and end positions define a substring copied from the retrieved passage, rather than text generated token by token.
- A dot product between S and each passage-token representation creates one start logit per passage token.
- Softmax converts the M start logits for an M-token passage into a probability distribution; the largest probability identifies the predicted start.
- A separate vector E produces the end-token logits and a second probability distribution.
- The extraction input may include multiple retrieved passages, in which case start and end positions are selected over the combined passage text.
- Training examples provide gold start and end positions, denoted I* and J*, for each question-passage pair.
- Assuming start and end predictions are independent gives a joint probability equal to the product of their probabilities.
- Negative log-likelihood turns maximization of the gold start and end probabilities into a minimization loss, the sum of their negative log probabilities.
- Inference scores valid spans with start and end logits from the trained S and E vectors, then returns the highest-scoring passage substring.
- Because retrieval can return passages without the answer, training data should include unanswerable examples.
- For a passage with no answer, the model can be trained to predict the CLS token, which represents the no-answer outcome.
- BERT's 512-token input limit can be handled by dividing longer passages into overlapping or fixed-size windows, such as 128 tokens.
- If an answer span crosses window boundaries, select a window that contains the complete answer.
- Exact match requires the predicted answer substring to match the gold string exactly, making it sensitive to small wording differences.
- Token F1 measures overlap between predicted and gold answer tokens by calculating token-level precision and recall.
- Research papers commonly report both exact match and token F1 to show strict correctness as well as partial overlap.
- The Hugging Face answer-extraction notebook has substantial data preprocessing, but model training is relatively straightforward once labels and token positions are prepared.
- RAG retrieves relevant documents with a retriever such as BM25 or dense search, then concatenates the query and evidence into one LLM input.
- A pretrained decoder-only transformer generates an answer token by token and can paraphrase evidence rather than copying an exact span.
- RAG is useful when a model's pretraining may not contain reliable knowledge for specialized domains, reducing dependence on parametric memory alone.
- Jointly training retrieval and the answer-generating LLM introduces additional challenges beyond the basic retrieve-then-generate pipeline.
- A generative model can produce unsupported claims when its pretrained knowledge does not cover a specialized question.
- In high-risk settings such as University of Utah Health, even a small hallucination rate may be unacceptable.
- Extractive answering constrains the answer to text present in retrieved passages, providing a more auditable alternative when the evidence contains the answer.
- RAG allows more flexible wording, while span extraction offers a stricter evidence-grounded response.
Transcript excerpt
0:00 Good afternoon, everyone. Um hope you had a nice weekend. Uh I sent an announcement about bonuses and how I will count them for quizzes. So, if you haven't seen that announcement, it's posted on Canvas. Uh so, check that out. Um Uh yeah, I don't have much more to say about that. Are there any questions uh about the announcement or anything else um logistics-wise or not? Yeah, go ahead. >> Have you updated on the previous quizzes yet? >> Uh not not yet. Jonathan is attending EMNLP, one of our conferences, so uh I doubt they're going to be done this week. Uh but, we'll try to get them done soon when he's back. Yeah, sorry about that.
0:48 Yes, please. >> The bonus we added both quizzes for the second quiz. >> Um read the announcement. If it's unclear, ask me. Uh the formula is right there, so I think it's easiest to just read the formula. Yeah. Yeah. Yeah. >> Okay. Well, I'll do fine on the last three. How How well did you do on the first one? >> How How well are they doing? Um well, we graded um you mean the second one? >> Yes, yes, that. >> Uh we don't I don't discuss the intermediate grading with uh yeah, so I wouldn't know that. Yeah, we we discuss them once everything is graded, so I have no information about that. Yeah. But, Lucas, maybe you can >> I thought the grading was pretty good overall. >> Yeah. But, that can be sampling.
1:38 What do you say? With uh teaching experience comes, don't give any high hopes. I I can't even I I I felt the second one, the vibes were in the room generally, I would say better than for on the first one. So, my expectation is
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.