Save this video free

RL @ ECE-UofT - Lecture 08: Deep SARSA and DQL

Ali Bereyhi · 2:28:33 · Watch on YouTube

Part of University of Toronto ECE 1508. See all 10 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Ali Bereyhi develops deep control methods by replacing tabular action values with neural networks, first extending SARSA and Q-learning, then addressing DQN’s instability with experience replay and a separate target network. He also explains how the maximum operator can overestimate action values, introduces Double Q-learning’s split between action selection and value evaluation, and motivates policy-gradient and actor-critic methods for continuous actions.

Key takeaways

Chapters

0:00 From Tabular Reinforcement Learning to Function Approximation
3:00 Generalized Policy Iteration and Epsilon-Greedy Control
6:00 Replacing the Q Table with a Neural Network
9:00 Q Networks Output Values for Every Discrete Action
13:00 Game Screens and Convolutional Q-Network Inputs
15:00 Deep SARSA and Deep Q-Learning: On-Policy Versus Off-Policy
18:00 Value Networks and Policy Networks
22:00 Neural TD Evaluation and the Gradient Update
26:00 SARSA Uses the Action Actually Chosen at the Next State
30:00 Deep SARSA Updates Along a Sampled Trajectory
34:00 Eligibility Traces as Gradient Momentum
38:00 Q-Learning Separates the Behavior and Target Policies
42:00 Backpropagation for Deep Q-Learning
46:00 Why Online DQN Updates Can Be Unstable
50:00 Experience Replay Breaks Up Correlated Transitions
54:00 Mini-Batch TD Targets and Replay-Buffer Sampling
58:00 How DQN Builds a Replay Buffer
1:02:00 DQN’s Moving-Target Problem
1:06:00 A Separate Target Network Stabilizes DQN
1:10:00 Putting Experience Replay and Target Networks Together
1:14:00 DeepMind’s 2015 Atari DQN Breakthrough
1:18:00 Q-Value Overestimation from the Maximum Operator
1:22:00 A Statistical Example of Max-Induced Bias
1:26:00 Double Q-Learning Separates Action Selection from Evaluation
1:38:00 Double DQN Results and the Meaning of Value Bias
1:44:00 Prioritized Replay and Distributed DQN Variants
1:54:00 Why DQN Struggles with Continuous Action Spaces
2:10:00 Policy Networks, Policy Gradients, and Actor-Critic Methods
2:20:00 When to Use DQN and What Comes Next

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:00 estimate, right? Monte Carlo, temporal difference, these were the approaches and then we use them basically these two ideas along with the fact that the optimal policy is the one which has maximum action value and we tried to come up with different ideas. Basically the ideas which are more sample efficient, have less bias, and can work fine. And we ended up with a couple of techniques, but then we said this is going to help us up to a level. And that level is when you have the number of estates limited, but if the number of estates now grows significantly large, you cannot do it anymore because you cannot go over every single one, every single state, every single action, do that. Now, the idea was to go for function

0:47 approximation. And we talked last time before the reading week about the function approximation. The idea was there that you model your value function as a neural network. So, you have this neural network which is taking, for example, the state and then giving you the value of the state as the output. Now, you try to look at the environment, you sample from the environment, and you use those samples to train this model, right? That was the idea. We did not talk about the control yet. We just talked about talk about prediction. Evaluation of the policy. So, last time when we talked, we basically wanted to get the idea how we can use function approximation and we thought of a simple example. We have the policy, we want to know how it works. And then we start to

1:34 model the value of that policy using a neural network. Then we sample from the environment, we use those samples to update. All right, cool? You remember what we did last time,

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Ali Bereyhi.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.