UUtah CS 6340 NLP | Fall 2026 | RAG & ODQA-pipeline training
Overview
The lecture explains how retrieval-augmented generation (RAG) supports open-domain question answering by grounding an LLM’s answer in retrieved documents, then examines practical constraints such as context limits, position bias, and conflicts with parametric knowledge. It compares four ways to train retrieval and answer components, emphasizing in-batch joint training with a relevant document and 10,000 sampled negatives as a scalable compromise, while recommending simple baselines before more complex methods.
Key takeaways
- RAG improves answer grounding by retrieving evidence before generation, which is especially important for niche datasets and high-stakes legal or medical questions.
- Longer prompts are not automatically better: transformer attention costs grow quadratically with sequence length, RoPE may degrade beyond pretraining lengths, and relevant text can be lost to position bias.
- Separate retriever and generator training is computationally straightforward but creates a mismatch when the answer model sees only gold documents during training and noisy retrieval results at inference.
- In-batch joint training uses one relevant document plus 10,000 sampled negatives to connect retrieval and answer losses without evaluating every document in a potentially billion-item store.
- Start with pretrained BM25 or dense-retrieval and answer-generation baselines, then add fine-tuning only if measured performance justifies the added compute and engineering.
- Open-weight, self-hosted models can be preferable when privacy rules prohibit sending sensitive data to proprietary APIs, while proprietary models may be strong off-the-shelf baselines when domain data is not highly specialized.
Chapters
- Open-domain question answering can retrieve relevant documents before either extracting an answer or generating one.
- RAG combines BM25 or dense retrieval with an LLM prompted using the question and retrieved text.
- Retrieved evidence helps with niche domains poorly represented in pretraining and makes answers easier to audit.
- Grounding is especially valuable in legal or medical applications, where fabricated facts such as nonexistent case law can cause serious harm.
- Adding too many retrieved documents can exceed the model’s maximum input length and increase transformer attention memory costs quadratically with sequence length.
- RoPE can encode positions poorly for sequences longer than those seen during pretraining, so extending context indefinitely can reduce performance.
- A practical RAG system can reduce the number of retrieved documents, such as using the top 4 instead of the top 100.
- Long-running agents can compact prior conversation history with an LLM and restart with the summarized context.
- LLMs often use information near the beginning or end of a prompt more reliably than information buried in the middle.
- Ordering retrieved passages by relevance can place the highest-ranked evidence where the model is more likely to use it.
- The lecture’s position-bias example showed answer accuracy varying with where relevant information appeared in the context.
- Context can conflict with parametric knowledge: redefining π in a prompt may confuse a model trained to associate it with approximately 3.14.
- A retriever could learn which evidence the answer module needs, while the answer module could adapt to weaknesses in retrieval.
- Joint optimization can be expressed as answer likelihood summed over documents, combining retrieval probability with the probability of producing the correct answer from each document.
- Computing that objective over a store of billions of documents is impractical because each query would require evaluating an enormous number of candidates.
- Training methods therefore need to connect retrieval and answer quality without backpropagating across the entire document store on every example.
- Separate training fine-tunes dense-retrieval encoders using relevant-document labels, then trains the answer model on question and gold-document pairs.
- The answer model receives clean, relevant evidence during training, so it may be unprepared when the deployed retriever returns irrelevant passages.
- Training on negative documents can help, but the chosen negatives may not match the retrieval errors encountered at inference time.
- Unlike extractive QA, where a CLS prediction can signal that no answer is present, generative RAG must handle the possibility that parametric knowledge supplies an answer despite irrelevant context.
- A pipeline can train the retriever while freezing the answer model, or train the answer model while keeping retrieval fixed.
- In-batch joint training samples 10,000 negative documents and adds one known relevant document, creating a 10,001-document candidate batch.
- A retrieval loss ranks the relevant document above negatives, while a generator negative log-likelihood loss measures whether the selected passage supports the correct answer.
- Combining the losses lets answer failures on poor retrieved passages provide a training signal to retrieval, without updating representations for every document in a billion-item store.
- The four approaches discussed are separate training, training one component against a frozen counterpart, periodic full joint training, and in-batch joint training.
- Periodic full joint training can refresh the model and document index occasionally, but incurs a large compute cost and leaves the index stale between updates.
- Begin with pretrained retriever and answer-model baselines, including BM25 or dense retrieval, before investing in more complicated fine-tuning.
- Measure results at each step: if an off-the-shelf pipeline already meets the target, extra training complexity may not be worthwhile.
- Generative answering is easy to build with current LLM tools, while extractive answering can reduce fabrication risk by selecting an answer span from retrieved text.
- Extractive QA requires preparing data in the format expected by the model, which can be more involved than prompting an LLM.
- More sophisticated dense retrieval or larger models can improve quality but may increase latency and per-token API costs.
- Open-weight models can be self-hosted on GPUs for sensitive data, helping organizations meet privacy obligations such as HIPAA without sending records to a proprietary cloud service.
- Dense retrieval is a maximum inner product search problem, and approximate nearest neighbor methods make search over large stores practical.
- The lecture distinguishes BM25 indexing from dense retrieval and reviews extractive answering versus answer generation.
- When labeled questions, relevant documents, and answers are available, fine-tuning can improve a pretrained retrieval-and-answer pipeline.
- The recommended workflow is to record a simple baseline’s performance, then test increasingly complex methods only when needed.
- Hugging Face notebooks provide practical examples for building answer-extraction systems and preparing their training data.
- An ACL 2023 tutorial offers additional background on retrieval and question answering.
- The legal-domain BriefMe work provides BM25 and dense-retrieval baselines that can serve as implementation starting points.
- The class concludes this RAG and open-domain QA unit before moving on to summarization.
Transcript excerpt
0:00 Sorry I'm late. It's one of those days. I will also need to leave today right after the class. So if you have any questions, I usually stick around, but please come to the office hours. Happy to answer them there. Are there any questions before we continue where we left things off last time? Okay, seems like we are good. Um all right so we are continuing our uh you know learning about question answering and last time we came to about a point where I started to introduce uh retrieval augmented generation. Um and just a kind of reminder we have these three options for how to go about open uh domain uh question answering. Uh two of them uh are pipeliny. They have first
0:51 the retrieval stage and after we retrieve uh certain relevant documents, we answer some question in the context of those documents. And we have seen an approach where we extract the answer. Uh and today we are talking about uh an approach where we generate the uh answer. Um so um why why I I brought up this question a few times. Why wouldn't we just generate the answer immediately without having uh the retrieval stage? And uh the main main uh argument here is that you might have settings where you cannot re not rely on so-called parametric knowledge of the model. Meaning model had acquired a lot of knowledge during pre-training and those are those those facts are stored in the
1:39 weights of the model and this is why models are capable of answering all sorts of uh question without you know retrieving context but if you have a special setup where your data or data domain isn't super well represented on the um internet then you shouldn't expect that the model will
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.