Save this video free

RL @ ECE-UofT - Lecture 09: Policy Gradient Theorem and PGMs

Ali Bereyhi · 2:35:10 · Watch on YouTube

Part of University of Toronto ECE 1508. See all 10 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Ali Bereyhi derives the policy gradient theorem for parameterized stochastic policies, showing how the gradient of expected return can be estimated from sampled state-action pairs using action values and the gradient of the policy’s log likelihood. He develops vanilla policy gradient and REINFORCE, explains why reward shifts can destabilize them, then derives action-independent baselines and the advantage function, connecting advantage estimates to TD(0) and n-step returns.

Key takeaways

Chapters

0:00 Why Policy Networks Need a Different Training Objective
5:00 Discrete Policies Use Softmax, Continuous Policies Use Distributions
10:00 Particle Navigation as a Policy-Network Example
15:00 Linear Softmax Policies and Continuous Gaussian Policies
20:00 Expected Value Across States Defines the Policy Objective
25:00 State Visitation Depends on Starts, the Environment, and the Policy
30:00 Episode Returns Express Value as a Trajectory Expectation
35:00 Expanding the Probability of a Sampled Trajectory
40:00 Random Initial States Complete the Expected-Return Objective
45:00 The Log-Derivative Trick Removes the Unknown Dynamics Gradient
50:00 Only Policy Terms Contribute to the Trajectory Log Gradient
55:00 Monte Carlo Estimates Produce Vanilla Policy-Gradient Updates
1:00:00 One-Trajectory Updates and the REINFORCE Algorithm
1:05:00 The Policy Gradient Theorem Uses Action-Value Estimates
1:10:00 Policy-Gradient Methods Include REINFORCE and Actor-Critic
1:15:00 Vanilla PGMs Scale Updates Directly by the Return
1:20:00 Reward Shifts Reveal Vanilla Policy-Gradient Sensitivity
1:25:00 Large Returns Can Cause Abrupt Policy Changes
1:30:00 A State-Dependent Baseline Can Be Subtracted Without Bias
1:35:00 Why the Expected Log-Policy Gradient Is Zero
1:40:00 The State Value Is a Natural Policy-Gradient Baseline
1:45:00 Advantage PGMs Center Action-Value Signals
1:50:00 Why Pure REINFORCE Cannot Bootstrap with TD
1:55:00 Three Ways to Obtain Value Estimates for Policy Updates
2:00:00 Actor-Critic Separates the Policy Actor from the Value Critic
2:05:00 A One-Step Bellman Estimate Produces the TD(0) Advantage
2:10:00 Value Networks Enable Practical Advantage Estimation
2:15:00 N-Step Returns Extend the Advantage Estimate
2:20:00 Advantage PGMs Improve Stability over Raw REINFORCE
2:25:00 Policy-Gradient Sample Efficiency Remains a Challenge
2:30:00 Closing: Reusing Trajectory Samples for Gradient Estimation

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:00 It was quite simple to train it, right? So you had the output of your neural network which was value and then you wanted to make this exactly like the value of this state and then you just instead of the value you sample either using Monte Carlo or TD you just sample that one you take the estimator of the value and you try to match it that was quite simple you just use listed squares and you train it right but in this case it's quite unclear what should be the training approach so I have a policy network I want it to behave like the optimal policy right but what is the optimal policy okay that is the guy which is uh having the maximum value right but how you can interpret this one into training loop that is a bit

0:49 not say straightforward and uh the whole trick of the policy gradient method theorem is that you can show that actually as long as you compute a gradient which is proport proportional to logarithm of the policy network. So log likelihood of the of the policy or the actions that you do then that is a valid gradient to train your neural network. Once you understand that one then the rest of it is again developing tricks and then we're going to see different challenges when we try to come over with with different tricks. Okay. So let's start with this idea of policy network. So the policy network as I told you last time is a neuronal network which is going to take as input your

1:37 state and of course you do know that you're not giving this state itself you're going to give the representation of the state right so in the example of

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Ali Bereyhi.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.