Save this video free

COMP 3200 - Intro to Artificial Intelligence - Lecture 17 - MDP + Dynamic Programming

Dave Churchill · 1:00:30 · Watch on YouTube

Part of Memorial University of Newfoundland COMP 3200. See all 19 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Dave Churchill explains how finite Markov decision processes represent reinforcement-learning tasks using states, actions, transition probabilities, rewards, and a discount factor, then derives how policies determine state and action values. He connects those ideas to Bellman-based dynamic programming for evaluating values and improving policies, while noting that exact dynamic programming requires a complete environment model and motivates temporal-difference learning when that model is unavailable.

Key takeaways

Chapters

0:00 The Markov Property: When Current State Is Enough
5:58 Finite MDPs and the SAPRI Definition
10:18 Discounted Return and the Role of Gamma
17:57 Unifying Episodic and Continuing Tasks
21:21 Why State and Action Values Depend on Policy
30:05 Dynamic Programming Relates Neighboring State Values
32:10 Evaluating a Grid State Under Different Policies
38:46 Bellman Equation: Policy, Transitions, Rewards, and Values
44:27 Bellman Evaluation as a Computation
47:06 Improving a Policy and Handling Tied Actions
55:49 Why Exact Dynamic Programming Is Limited
57:55 Policy Iteration and Exam-Ready Applications

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:00 Lecture number 18. Um just let you know this lecture we're going to be talking about what's here, continuing tasks, MDPs, and dynamic programming. But uh this lecture it's a bit strange because I talk about sort of three independent things, and then at the end we sort of tie them all together. So, if it seems a bit strange, why is he talking about this? It's in the order that the book introduces it. And uh what I've done here, which I probably should have been doing from the beginning, also I'm just linking you to the chapters in the book where this is talked about because if you're studying for the exam, it's nice to be able to just go straight there. Um like I mentioned before, has anyone actually looked at the textbook at all yet? Nobody? Okay. I know that you don't want to, but uh trust me, it's actually like

0:47 a really good read. And if you follow these links, you'll be taken to the HTML version, but if you um if you either go back to the start of the book and the HTML version, or you just Google reinforcement learning textbook, you'll be met with the free downloadable PDF. Um but here are the sections because it does go into more detail, and even though the exam isn't going to talk about that extra detail, knowing that extra detail certainly will make you understand it more when it comes to answering questions on the exam. So, that's why I put it there. So, we'll be talking about uh Markov decision processes, uh continuing tasks, and dynamic programming. So, first uh MDPs, what is an MDP? Well, I'll get to that first, but in order to talk about MDPs, we need to know what uh Markov means. And Markov

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Dave Churchill.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.